Reference

Pandas gotchas

Defaults that quietly change your answer. Check for these whenever you read AI-written pandas.

1 · Filters silently drop NaN

df[df["sales"] > 100]   # NaN > 100 is False → that row disappears

Ask yourself: should the missing rows have been excluded, or should someone look at them? L01

2 · Aggregations skip NaN

df["sales"].mean()          # skipna=True by default → divides by non-missing count
df["sales"].fillna(0).mean()  # if "missing" really means zero

Whether missing means "unknown" or "zero" is a business decision, not a coding one.

3 · sort_values is ascending, NaN last

df.sort_values("sales")                     # smallest first, NaN at the end
df.sort_values("sales", ascending=False)   # largest first

4 · Parenthesise every condition

df[df["sales"] > 100 & df["region"] == "East"]       # ✗ TypeError (& binds first)
df[(df["sales"] > 100) & (df["region"] == "East")]   # ✓
# use & (and), | (or), ~ (not) — not the words and / or / not

5 · groupby returns keys sorted

df.groupby("region")["sales"].sum()   # East, North, West (alphabetical)

6 · value_counts leaves NaN out

df["stars"].value_counts(normalize=True)   # shares of the rated rows only
df["stars"].value_counts(dropna=False)     # shows how many are missing

L03

7 · A missing value is "not low"

df["low"] = df["stars"] <= 2   # NaN <= 2 is False → unrated reviews count as not low

They stay in the denominator of the share. Filter them out first if "unrated" shouldn't count.

8 · An average without its count

….agg(["mean", "count"])   # Heater: mean 5.0, count 1 → one opinion, not a winner

Rank only groups with enough rows: summary[summary["count"] >= n].

9 · Dates that are secretly text

df["date"].dt.month              # AttributeError: Can only use .dt accessor with datetimelike values
df["date"] = pd.to_datetime(df["date"])   # fix: convert first

CSV files and APIs give you text. Check df["date"].dtype before anything else. L04

10 · value_counts loses time order

df["date"].dt.to_period("M").value_counts()                # sorted by count
df["date"].dt.to_period("M").value_counts().sort_index()   # sorted by month

11 · A rolling average starts with NaN

monthly.rolling(3).mean()   # first 2 values are NaN: not enough months yet

And a month with no rows simply isn't in the grouped result, so "3 rows" may span more than 3 calendar months.

12 · .plot without the brackets draws nothing

monthly.plot     # just names the method: no chart
monthly.plot()   # draws it

13 · A chart keeps the order it's given

df.groupby("product")["stars"].mean().plot.bar()                          # A→Z, not ranked
df.groupby("product")["stars"].mean().sort_values(ascending=False).plot.bar()  # ranked

Same for months after value_counts(): add .sort_index(). And barh puts the first label at the bottom.

14 · An axis that zooms in exaggerates

monthly.plot()               # 4.3 → 4.1 stars fills the whole chart: looks like a crash
monthly.plot(ylim=(1, 5))    # the real scale

Bar charts should always start at 0. And plt.figure() starts a new, empty chart: leave it out when you want two lines on one. L05

15 · A column that's empty, or has gaps, changes type

df["response"].dtype   # float64: every value is NaN, and NaN is a number
df["verified"].dtype   # object: True/False plus NaN (no gaps: bool)

The same file can read differently when a slice happens to be empty. Check df.dtypes after every load. L07

16 · Not every gap is NaN

df["topic"].isna().sum()             # 0: "No topic" is text
(df["topic"] == "No topic").sum()    # 3

Exports write "nothing" as No topic, none, unknown, -. And plain value_counts() drops real NaN: add dropna=False.

17 · drop and dropna go further than you think

df.drop("response")                # KeyError: looks for a ROW labelled response
df.drop(columns=["response"])      # drops the column
df.dropna()                        # drops a row if ANY column is missing: maybe all of them
df.dropna(subset=["verified"])     # only rows missing verified

18 · A notebook remembers, in the order you ran it

[1]  x = 1
[4]  x = x + 1    # ran three times: x is 4, not 2

When a notebook gives odd results, Restart and Run All. If it works top to bottom, trust it. L06

Checked against pandas 3.0.2, the version the sandbox runs. Source: 10 minutes to pandas · Python for Data Analysis, 3e