Lesson 03 · Data Science → AI/ML Engineering
Summarise groups
"How many, how good, and how often bad?" That's the first table anyone wants from review data.
A new table: eight product reviews. One review has no star rating. pd and df are already loaded.
| product | stars | |
|---|---|---|
| 0 | AC | 5.0 |
| 1 | Fan | 4.0 |
| 2 | AC | 1.0 |
| 3 | Heater | 5.0 |
| 4 | AC | 2.0 |
| 5 | Fan | NaN |
| 6 | AC | 4.0 |
| 7 | Fan | 3.0 |
Part 1 · Learn
The syntax, and what each piece returns
ACount each value
| Syntax | Returns |
|---|---|
df["product"].value_counts() | A Series: each distinct value is an index label, and the number of times it appears is the value. Sorted most common first. |
….value_counts(normalize=True) | Shares instead of counts (they add up to 1). |
….value_counts(dropna=False) | Also counts NaN. By default NaN is left out, including from the shares' denominator. |
It's a shortcut for "group by this column and count the rows", with the biggest group first.
BSeveral stats at once
| Syntax | Returns |
|---|---|
df.groupby("product")["stars"].agg(["mean", "count"]) | A DataFrame: one row per product, one column per stat, with the columns named mean and count. |
summary["count"] | One stat column, as a Series indexed by product. |
summary.loc["Fan", "count"] | One cell. Here the row label is a product name, not a number. |
Put count next to mean every time. An average of one review is a single opinion, not a trend.
DSort and filter a summary
| Syntax | Returns |
|---|---|
series.sort_values(ascending=False) | A Series sorts by its values, so there's no column name to pass. |
summary.sort_values("mean", ascending=False) | A DataFrame needs the column to sort by, as in Lesson 02. |
summary[summary["count"] >= 2] | A summary is just a DataFrame, so the masks from Lesson 01 work on it too. |
Part 2 · Practice
Now write it yourself
Write a real guess on each # predict: line, replacing the ? or writing after the →. The check no longer accepts the line unchanged.
2 · Share of 5-star reviews
Set five_share to the share of rated reviews that gave 5 stars (a number between 0 and 1).
Hint
A with normalize=True gives a Series indexed by star value. Then look up the label 5.
3 · Average and count per product
Set summary to a table with the mean and count of stars for each product.
Hint
See B.
4 · Share of low reviews per product
A "low" review has 2 stars or fewer. Set low_share to the share of low reviews for each product.
Hint
Two lines, both in C: make the True/False column first, then group and take the mean.
Stretch · Best product with enough reviews
Set ranked to the mean/count summary, keeping only products with at least 2 rated reviews, best average first.
Hint
D: filter with a mask on summary["count"], then sort by "mean".
Part 3 · Remember
Spaced review
Next, do the practice set until this lesson's skills are mastered (3 clean solves each). Then its 5 cards join your Daily review from the next day.
Use it at work tomorrow
Load your review pipeline's output into pandas and write these three lines yourself:
value_counts()on the product column, to see how many reviews each product has.groupby(…)[stars].agg(["mean", "count"]), to see each product's average with its sample size beside it.- The low-review share per product, from C.
Then check one number against your dashboard. If they disagree, find out why: it's usually NaN handling or a filter. This is the start of the Taster 1 test.
Python for Data Analysis ch. 10: Data Aggregation and Group Operations, by Wes McKinney. §10.2 covers agg with several functions. value_counts is in §5.3.