Lesson 03 · Data Science → AI/ML Engineering

Summarise groups

"How many, how good, and how often bad?" That's the first table anyone wants from review data.

≈ 15 minLearn → Practice → ReviewBuilds on: Lesson 02

A new table: eight product reviews. One review has no star rating. pd and df are already loaded.

productstars
0AC5.0
1Fan4.0
2AC1.0
3Heater5.0
4AC2.0
5FanNaN
6AC4.0
7Fan3.0

Part 1 · Learn

The syntax, and what each piece returns

ACount each value

SyntaxReturns
df["product"].value_counts()A Series: each distinct value is an index label, and the number of times it appears is the value. Sorted most common first.
….value_counts(normalize=True)Shares instead of counts (they add up to 1).
….value_counts(dropna=False)Also counts NaN. By default NaN is left out, including from the shares' denominator.

It's a shortcut for "group by this column and count the rows", with the biggest group first.

BSeveral stats at once

SyntaxReturns
df.groupby("product")["stars"].agg(["mean", "count"])A DataFrame: one row per product, one column per stat, with the columns named mean and count.
summary["count"]One stat column, as a Series indexed by product.
summary.loc["Fan", "count"]One cell. Here the row label is a product name, not a number.

Put count next to mean every time. An average of one review is a single opinion, not a trend.

CShare of rows that meet a condition

SyntaxReturns
df["low"] = df["stars"] <= 2Adds a new True/False column called low to df. Assigning to a column name that doesn't exist yet creates it.
df["low"].mean()The share of rows that are True. True counts as 1 and False as 0, so the mean is a proportion.
df.groupby("product")["low"].mean()That share, per product.

Watch out: NaN <= 2 is False, so the review with no stars counts as "not low" and is in the denominator. Decide whether that's what you want.

DSort and filter a summary

SyntaxReturns
series.sort_values(ascending=False)A Series sorts by its values, so there's no column name to pass.
summary.sort_values("mean", ascending=False)A DataFrame needs the column to sort by, as in Lesson 02.
summary[summary["count"] >= 2]A summary is just a DataFrame, so the masks from Lesson 01 work on it too.

Part 2 · Practice

Now write it yourself

Write a real guess on each # predict: line, replacing the ? or writing after the →. The check no longer accepts the line unchanged.

1 · Reviews per product

Set counts to the number of reviews for each product.

Hint

See A.

2 · Share of 5-star reviews

Set five_share to the share of rated reviews that gave 5 stars (a number between 0 and 1).

Hint

A with normalize=True gives a Series indexed by star value. Then look up the label 5.

3 · Average and count per product

Set summary to a table with the mean and count of stars for each product.

Hint

See B.

4 · Share of low reviews per product

A "low" review has 2 stars or fewer. Set low_share to the share of low reviews for each product.

Hint

Two lines, both in C: make the True/False column first, then group and take the mean.

Stretch · Best product with enough reviews

Set ranked to the mean/count summary, keeping only products with at least 2 rated reviews, best average first.

Hint

D: filter with a mask on summary["count"], then sort by "mean".

Part 3 · Remember

Spaced review

Next, do the practice set until this lesson's skills are mastered (3 clean solves each). Then its 5 cards join your Daily review from the next day.

Use it at work tomorrow

Load your review pipeline's output into pandas and write these three lines yourself:

  1. value_counts() on the product column, to see how many reviews each product has.
  2. groupby(…)[stars].agg(["mean", "count"]), to see each product's average with its sample size beside it.
  3. The low-review share per product, from C.

Then check one number against your dashboard. If they disagree, find out why: it's usually NaN handling or a filter. This is the start of the Taster 1 test.

Primary source

Python for Data Analysis ch. 10: Data Aggregation and Group Operations, by Wes McKinney. §10.2 covers agg with several functions. value_counts is in §5.3.

Ask me anything. If it's unclear why the mean of True/False is a share, or when NaN should count as "not low", ask me. Both come up constantly with real review data.