Lesson 07 · Data Science → AI/ML Engineering
Check and clean what you loaded
A CSV is only text, so read_csv has to guess what each column holds, and real exports have gaps that don't always look like gaps. This lesson checks the guesses, finds what's missing, and fixes what came in wrong. It's the first ten minutes of any real analysis, right after the load from Lesson 06.
A review export from two shops, saved as reviews.csv. It's made up, but it has the same habits as a real pipeline export: one row per review, ratings written as 5.0, a column nobody filled, a True/False column with gaps, brand names in mixed capitals, and No topic written where a review mentions no topic. This is the file as text, exactly as it sits on disk:
review_id,shop,review_date,rating,verified,response,brand,topic,sentiment
KDM-1041,Kaden Mart,2026-07-03,5.0,True,,Breezo,Noise,POSITIVE
KDM-1057,Kaden Mart,2026-07-09,2.0,,,breezo,Noise,NEGATIVE
SHW-88213,Shopwell,2026-07-14,4.0,False,,Aerora,No topic,
SHW-88240,Shopwell,2026-07-21,1.0,True,,AERORA,Price,NEGATIVE
KDM-1102,Kaden Mart,2026-07-28,3.0,True,,Breezo,No topic,
SHW-88302,Shopwell,2026-08-02,5.0,False,,Aerora,Cooling,POSITIVE
KDM-1139,Kaden Mart,2026-08-06,4.0,,,BREEZO,Cooling,POSITIVE
SHW-88355,Shopwell,2026-08-11,2.0,True,,aerora,Noise,NEGATIVE
KDM-1160,Kaden Mart,2026-08-19,5.0,True,,Breezo,Price,POSITIVE
SHW-88391,Shopwell,2026-08-25,3.0,False,,Aerora,No topic,
Two commas in a row (,,) mean an empty value. pd is loaded and the file is in the page's working folder, but nothing has read it yet: every block below starts with the read_csv line from Lesson 06. Once loaded, it looks like this:
| review_id | shop | review_date | rating | verified | response | brand | topic | sentiment | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | KDM-1041 | Kaden Mart | 2026-07-03 | 5.0 | True | NaN | Breezo | Noise | POSITIVE |
| 1 | KDM-1057 | Kaden Mart | 2026-07-09 | 2.0 | NaN | NaN | breezo | Noise | NEGATIVE |
| 2 | SHW-88213 | Shopwell | 2026-07-14 | 4.0 | False | NaN | Aerora | No topic | NaN |
| 3 | SHW-88240 | Shopwell | 2026-07-21 | 1.0 | True | NaN | AERORA | Price | NEGATIVE |
| 4 | KDM-1102 | Kaden Mart | 2026-07-28 | 3.0 | True | NaN | Breezo | No topic | NaN |
| 5 | SHW-88302 | Shopwell | 2026-08-02 | 5.0 | False | NaN | Aerora | Cooling | POSITIVE |
| 6 | KDM-1139 | Kaden Mart | 2026-08-06 | 4.0 | NaN | NaN | BREEZO | Cooling | POSITIVE |
| 7 | SHW-88355 | Shopwell | 2026-08-11 | 2.0 | True | NaN | aerora | Noise | NEGATIVE |
| 8 | KDM-1160 | Kaden Mart | 2026-08-19 | 5.0 | True | NaN | Breezo | Price | POSITIVE |
| 9 | SHW-88391 | Shopwell | 2026-08-25 | 3.0 | False | NaN | Aerora | No topic | NaN |
Part 1 · Learn
The syntax, and what each piece returns
ACheck what pandas guessed
| Syntax | Returns |
|---|---|
df.dtypes | A Series: one dtype (data type) per column, labelled by column name. |
df["rating"].dtype | The dtype of one column. |
df.info() | Prints, for each column, its dtype and how many values are not missing. The quickest overview of a new file. |
A CSV is only text. pandas reads each column and guesses its type. Here's what it guesses for this file, and why:
| In the file | pandas reads it as | What to do |
|---|---|---|
text: Kaden Mart, No topic | str | Fine. |
dates: 2026-07-03 | str: still text | pd.to_datetime, as in Lesson 04. |
whole numbers with .0: 5.0 | float64: numbers with decimals | Works for maths. Convert to whole numbers if you want them to print as 5 (section C). |
a column that's empty on every row: response | float64 | Surprising. An empty value is NaN, and NaN counts as a number, so a column of nothing looks numeric. Usually you drop it. |
True/False with some gaps: verified | object: a mix | Without gaps it would be bool. One missing value turns the whole column into object, which means "a mix of Python values". |
Never trust the guess: check dtypes first. The same file can read differently another day. If a column happens to be empty in the rows you load (say, the first 5,000 with nrows=), its dtype changes. Code that worked yesterday then breaks.
BFind what's missing
| Syntax | Returns |
|---|---|
df.isna() | A table the same shape as df: True where a value is missing. |
df.isna().sum() | A Series: how many values are missing in each column. (Summing True/False counts the Trues.) |
df.isna().mean() | A Series: the share missing in each column, from 0 to 1. The share-of-True trick from Lesson 03. |
df["verified"].value_counts(dropna=False) | Counts per value, with a NaN row for the gaps. Plain value_counts() leaves them out, so the counts quietly add up to fewer rows than you have. |
df.dropna(subset=["verified"]) | A new DataFrame without the rows where verified is missing. Without subset=, it drops a row if any column is missing. Here every row has a missing response, so that drops all of them. |
Not all missing data is NaN. In topic, a review with no topic says No topic. That's text, so isna() doesn't see it. Exports do this a lot: No topic, none, unknown, -. Count them with a mask, as in Lesson 01: (df["topic"] == "No topic").sum().
Notice that sentiment is missing on exactly those three rows. A pattern like that means the gaps have a reason: no topic, so no sentiment to give. It's not a broken file.
CFix what came in wrong
| Syntax | Returns |
|---|---|
df["rating"].astype(int) | A new Series of whole numbers (int64): 5.0 → 5. Fails with IntCastingNaNError if the column has any NaN, because a gap can't be a whole number. Deal with the gaps first. |
df["brand"].str.lower() | A new Series in lower case: AERORA, aerora and Aerora all become aerora. .str unlocks text methods, the way .dt unlocks date parts. |
df.drop(columns=["response"]) | A new DataFrame without those columns. Without columns=, drop looks for row labels and raises a KeyError. |
df["review_date"] = pd.to_datetime(df["review_date"]) | Real dates, from Lesson 04. |
Each of these returns a new Series or DataFrame and leaves df alone. To keep the fix, assign it back: df["brand"] = df["brand"].str.lower(), or df = df.drop(columns=[…]).
Why fix the casing? groupby and value_counts treat Aerora and AERORA as two different brands, so the counts get split across spellings. Run the example and compare the two counts.
📖Read the docs: read_csv's missing-value markersRequired
Section B counted No topic with a mask, because isna() can't see it. You can also tell read_csv to treat such text as missing while it loads. Then isna(), dropna() and value_counts(dropna=False) all see it. Find the parameter that does that.
- Open pandas.read_csv (pandas 3.0, the version this page runs).
- Read down Parameters, or search the page (⌘F / Ctrl F) for "NA/NaN". Find the one for additional strings to recognize as missing. Its description also lists the strings pandas already treats as missing: notice that
NA,nullandn/aare on that list, butNo topicisn't. - Read the type written after its name. It says what kind of value it takes, which tells you whether to pass one string or a list of them.
Docs · Make the hidden gaps real
Change the starter so that No topic comes in as a missing value. Do it inside the read_csv call, and leave every other value as it is.
Hint
The answer isn't in this lesson. On the read_csv page, the parameter's description starts with "Additional strings to recognize as NA/NaN". Its type allows a list of strings, so pass "No topic" inside a list. It's one more keyword argument inside pd.read_csv(…).
Part 2 · Practice
Now check and clean it yourself
Each run starts fresh, with only the file on disk: the first line of each starter loads it.
1 · Which guesses are odd?
Print the dtypes. Then set floats to a list of the columns pandas read as float64, and mixed to the name of the column it read as object.
Hint
Print df.dtypes and read it. Section A explains why an empty column comes out as a number.
2 · Count the gaps, hidden ones too
Set missing to how many values are missing in each column. Then set no_topic to how many reviews have no topic.
Hint
missing: one method finds the gaps and another counts them per column. no_topic isn't a gap at all, it's the text "No topic": count it with a mask. See B.
3 · Clean it up
Make df ready to analyse: drop the empty response column, make rating whole numbers, put brand in lower case, and turn review_date into real dates.
Hint
Each line is one method from C, assigned back. drop needs columns=, and .str comes before the text method.
Stretch · Which shop has more reviews with no topic?
Set no_topic_share to the share of reviews with no topic, per shop.
Hint
Make a True/False column for "no topic" (a mask, as in exercise 2), then the share of True per group from Lesson 03: groupby → mean().
Part 3 · Remember
Spaced review
Next, do the practice set until this lesson's skills are mastered (3 clean solves each). Then its 3 cards join your Daily review from the next day.
Use it at work tomorrow
Open reviews.ipynb from Lesson 06 and load your local copy, flat-sample.csv. Then:
- Run
df.info()anddf.isna().sum(). Note any column that's entirely empty, anyobjectcolumn, and any text that means "nothing" (aNo topic,noneorunknown). - Load a different slice straight from S3, for example 5,000 rows further in:
pd.read_csv(path, storage_options=…, skiprows=range(1, 50001), nrows=5000). Compare itsdtypeswith the first slice. If a column changes type between slices, set it yourself after loading rather than trusting the guess. - Check whether one row is one review: compare
len(df)withlen(df["review_id"].value_counts()). If there are more rows than reviews, a review spans several rows, and counting rows isn't counting reviews.
Write down what you found. That list is your data-quality note, and it's what you check first before trusting any chart built on this data.
Python for Data Analysis ch. 6: Data Loading, Storage, and File Formats, by Wes McKinney. §6.1 "Reading and Writing Data in Text Format" covers read_csv's missing-value markers (na_values). Ch. 7: Data Cleaning and Preparation §7.1 "Handling Missing Data" covers isna and dropna. §7.4 "String Manipulation" covers .str methods.
df.info() prints.