Reference
Pandas gotchas
Defaults that quietly change your answer. Check for these whenever you read AI-written pandas.
1 · Filters silently drop NaN
df[df["sales"] > 100] # NaN > 100 is False → that row disappears
Ask yourself: should the missing rows have been excluded, or should someone look at them? L01
2 · Aggregations skip NaN
df["sales"].mean() # skipna=True by default → divides by non-missing count
df["sales"].fillna(0).mean() # if "missing" really means zero
Whether missing means "unknown" or "zero" is a business decision, not a coding one.
3 · sort_values is ascending, NaN last
df.sort_values("sales") # smallest first, NaN at the end
df.sort_values("sales", ascending=False) # largest first4 · Parenthesise every condition
df[df["sales"] > 100 & df["region"] == "East"] # ✗ TypeError (& binds first)
df[(df["sales"] > 100) & (df["region"] == "East")] # ✓
# use & (and), | (or), ~ (not) — not the words and / or / not5 · groupby returns keys sorted
df.groupby("region")["sales"].sum() # East, North, West (alphabetical)6 · value_counts leaves NaN out
df["stars"].value_counts(normalize=True) # shares of the rated rows only
df["stars"].value_counts(dropna=False) # shows how many are missing
7 · A missing value is "not low"
df["low"] = df["stars"] <= 2 # NaN <= 2 is False → unrated reviews count as not low
They stay in the denominator of the share. Filter them out first if "unrated" shouldn't count.
8 · An average without its count
….agg(["mean", "count"]) # Heater: mean 5.0, count 1 → one opinion, not a winner
Rank only groups with enough rows: summary[summary["count"] >= n].
9 · Dates that are secretly text
df["date"].dt.month # AttributeError: Can only use .dt accessor with datetimelike values
df["date"] = pd.to_datetime(df["date"]) # fix: convert first
CSV files and APIs give you text. Check df["date"].dtype before anything else. L04
10 · value_counts loses time order
df["date"].dt.to_period("M").value_counts() # sorted by count
df["date"].dt.to_period("M").value_counts().sort_index() # sorted by month11 · A rolling average starts with NaN
monthly.rolling(3).mean() # first 2 values are NaN: not enough months yet
And a month with no rows simply isn't in the grouped result, so "3 rows" may span more than 3 calendar months.
12 · .plot without the brackets draws nothing
monthly.plot # just names the method: no chart
monthly.plot() # draws it13 · A chart keeps the order it's given
df.groupby("product")["stars"].mean().plot.bar() # A→Z, not ranked
df.groupby("product")["stars"].mean().sort_values(ascending=False).plot.bar() # ranked
Same for months after value_counts(): add .sort_index(). And barh puts the first label at the bottom.
14 · An axis that zooms in exaggerates
monthly.plot() # 4.3 → 4.1 stars fills the whole chart: looks like a crash
monthly.plot(ylim=(1, 5)) # the real scale
Bar charts should always start at 0. And plt.figure() starts a new, empty chart: leave it out when you want two lines on one. L05
15 · A column that's empty, or has gaps, changes type
df["response"].dtype # float64: every value is NaN, and NaN is a number
df["verified"].dtype # object: True/False plus NaN (no gaps: bool)
The same file can read differently when a slice happens to be empty. Check df.dtypes after every load. L07
16 · Not every gap is NaN
df["topic"].isna().sum() # 0: "No topic" is text
(df["topic"] == "No topic").sum() # 3
Exports write "nothing" as No topic, none, unknown, -. And plain value_counts() drops real NaN: add dropna=False.
17 · drop and dropna go further than you think
df.drop("response") # KeyError: looks for a ROW labelled response
df.drop(columns=["response"]) # drops the column
df.dropna() # drops a row if ANY column is missing: maybe all of them
df.dropna(subset=["verified"]) # only rows missing verified18 · A notebook remembers, in the order you ran it
[1] x = 1
[4] x = x + 1 # ran three times: x is 4, not 2
When a notebook gives odd results, Restart and Run All. If it works top to bottom, trust it. L06
Checked against pandas 3.0.2, the version the sandbox runs. Source: 10 minutes to pandas · Python for Data Analysis, 3e