Lesson 06 · Data Science → AI/ML Engineering · Lab
Get your data: from S3 into a notebook
Your CSV files live in an S3 bucket, so everything this course teaches only helps at work once you can get them into Python. This lesson sets that up once: a Jupyter notebook, your AWS login, and boto3, AWS's own Python library and the one your projects use, to fetch a file from S3 into pandas.
This is a Lab: you do the steps on your Mac, in a real notebook, not on this page. The browser can't log in to AWS or reach your bucket. Each part ends with a Check: answer Yes, No or Not sure. If you answer No or Not sure, you get troubleshooting help, and you can paste the error you saw. The lesson is done when every Check is Yes.
The code boxes in parts D and E are different: they run here in the browser, against a stand-in boto3 with a small made-up bucket called shop-data, so you can try each call before using it on the real bucket.
Part 1 · Learn and do
AOpen a notebook
A Jupyter notebook is a file (ending .ipynb) made of cells. You type Python into a cell, run it, and its output appears right under it, like the code boxes in these lessons. Behind every notebook is a kernel: the Python program that actually runs your cells. Which kernel you pick decides which installed packages your code can use.
| Action | How |
|---|---|
| Run a cell | Shift + Enter: runs it and moves to the next cell |
| Install a package into the kernel's Python | %pip install boto3 in a cell. The % makes it install into the kernel's Python, not some other Python on your Mac. |
| Run a terminal command from a cell | Start the line with !, e.g. !aws --version |
| Start clean | Restart (clears everything the notebook remembers), then Run All |
- In VS Code, install the Python and Jupyter extensions if you don't have them (Extensions panel, search each name). Make a folder for work notebooks outside this course folder, because this folder syncs and real data shouldn't.
- File → New File… → Jupyter Notebook. Save it in that folder as
reviews.ipynb. - Top right, click Select Kernel and pick a Python environment (a Python 3 from Homebrew or a virtual environment is fine).
- In the first cell, type this and press Shift + Enter:
If it saysimport pandas as pd pd.__version__No module named 'pandas', run%pip install pandasin a cell, then run the first cell again. - In the next cell, run
%pip install boto3. It isn't installed on your Mac yet: theawscommand carries its own private copy, which your notebook can't use. A version number such as
2.2.3or3.0.2prints under the cell. (Note it down and tell me: the proof project will follow your version.)
BThe one big difference: a notebook remembers
In these lessons, every Run starts fresh. A notebook doesn't: a variable made in one cell stays until you restart. Cells can run in any order, and the number beside each one ([3]) shows the order they actually ran in. That's handy, because you load a big file once and reuse it. But it's also the most common source of "it worked a minute ago". If results look odd, Restart and Run All: if the notebook still works from top to bottom, you can trust it.
- In a new cell, run
x = 1. In the cell below it, typex = x + 1and thenxon the next line. - Run that second cell three times, then Restart the kernel and Run All.
After three runs the second cell shows
4, because it kept adding to the rememberedx. After Restart and Run All it shows2.
CYour AWS login, briefly
To read from S3, your code has to say who you are. boto3 does that with a profile: a named set of credentials saved in two files in your home folder, ~/.aws/config and ~/.aws/credentials. Your code never holds a password; it only names the profile. There are two kinds of profile, and you use both:
| Kind | How it works | Example |
|---|---|---|
| Key profile | A fixed access key saved in ~/.aws/credentials. It doesn't expire, so there's nothing to refresh. | study-reviews-read: read-only |
| Login profile | No saved key. aws login --profile NAME (or aws sso login --profile NAME, depending on how your account is set up) opens the browser, you sign in, and the CLI saves temporary credentials under that profile name. They expire after some hours, and then you run the login command again. | Your everyday work profile |
So the login command and the profile work together: the command refreshes what the profile holds, and your code just says which profile to use. Two commands help when you're not sure what you have:
| Command | Prints |
|---|---|
aws configure list-profiles | The names of every profile on this Mac |
aws sts get-caller-identity --profile NAME | Who AWS thinks you are with that profile: an Arn ending in a user or role name. If it fails, the profile is wrong or its login has expired. |
- In a notebook cell, run
!aws configure list-profiles. - Then run
!aws sts get-caller-identity --profile study-reviews-read. The list includes
study-reviews-read, and the second command prints anArnending inuser/study-reviews-read.
DReach the bucket
S3 stores files as objects in a bucket. Each object has a key: its full name inside the bucket, such as topic-preview/flat/flat.csv. The slashes make keys look like folders, and the AWS console shows them as folders, but S3 just matches the start of the name: a prefix. boto3 always takes the bucket and the key separately: no s3://, and no bucket name inside the key.
| Syntax | Returns |
|---|---|
import boto3s3 = boto3.Session(profile_name="NAME").client("s3") | A client: the object you call S3 through, logged in with that profile. The Session picks the profile; .client("s3") picks the service. |
r = s3.list_objects_v2(Bucket="shop-data", Prefix="exports/") | A dictionary about the objects whose key starts with the prefix. r["KeyCount"] is how many; r["Contents"] is a list with one small dictionary per object (its Key, Size in bytes, and more). At most 1,000 at a time. |
pd.DataFrame(r["Contents"]) | That list as a table, one row per object. The quickest way to read a listing. |
Careful: when nothing matches the prefix, the dictionary has no "Contents" key at all, so r["Contents"] fails with KeyError: 'Contents'. Prefixes are exact and case-sensitive: export/ doesn't match exports/. Check r["KeyCount"] first.
- Back in your notebook:
import boto3 s3 = boto3.Session(profile_name="study-reviews-read").client("s3") r = s3.list_objects_v2(Bucket="rac-reviews", Prefix="topic-preview/") pd.DataFrame(r["Contents"])[["Key", "Size"]] A table of 4 objects:
flat.csvandpairs.csv, each with a smallquicksight-manifest.json.flat.csv's Size is about 276,600,000 bytes (≈ 277 MB).
ELoad the CSV
| Syntax | Returns |
|---|---|
obj = s3.get_object(Bucket="shop-data", Key="exports/export.csv") | A dictionary about the object. obj["Body"] is the file's contents as a stream: like an open file, read from start to end. |
df = pd.read_csv(obj["Body"]) | A DataFrame. read_csv takes a file name or anything that reads like a file. Its first line (the header) becomes the column names, every other line a row, labelled 0, 1, 2… An empty value becomes NaN. |
pd.read_csv(…, nrows=5000) | Only the first 5,000 rows. Your real file is hundreds of MB, so a first look should read only a slice. |
pd.read_csv(…, usecols=["review_id", "rating"]) | Only those columns, in the file's order (not the list's). |
df.shape | A pair, (rows, columns). No brackets after shape: it's a property, not a method. |
df.head(3) | The first 3 rows (5 if you leave the number out). |
Two steps: boto3 fetches the object, pandas reads it. If something fails, the error's name tells you which step. A stream is read once: to load the file again, call get_object again. The same read_csv also opens a local file by name: pd.read_csv("export.csv").
- In the notebook, load the first 5,000 rows of the real file:
obj = s3.get_object(Bucket="rac-reviews", Key="topic-preview/flat/flat.csv") df = pd.read_csv(obj["Body"], nrows=5000) df.shape - Run
df.head()and scroll sideways through the columns. df.shapeshows(5000, 41): 5,000 rows and 41 columns, in a few seconds.
You'll also see: pd.read_csv("s3://…"). pandas can open an S3 path directly: pd.read_csv("s3://rac-reviews/topic-preview/flat/flat.csv", storage_options={"profile": "study-reviews-read"}). It needs an extra package, s3fs, which makes the same S3 requests behind the scenes and reads the same profiles. Coding agents and the pandas docs use it in notebooks. You don't need to write it; when you see it, read it as "get_object + read_csv in one line".
FKeep a local copy
| Syntax | Does |
|---|---|
df.to_csv("flat-sample.csv", index=False) | Writes df to a CSV file next to the notebook. index=False leaves out the row labels 0, 1, 2…; without it, the file gets an extra unnamed first column. |
Fetching from S3 every time is slow, and it fails when your login has expired. Save the slice once, then load the local copy while you explore. Go back to S3 when you need fresh or more data. Keep these files in your work folder, not in this course folder.
- Run
df.to_csv("flat-sample.csv", index=False). - Restart the kernel (so
dfis gone), then in a cell:import pandas as pd,local = pd.read_csv("flat-sample.csv"),local.shape. flat-sample.csvappears next to the notebook, andlocal.shapeis(5000, 41)again, with no AWS login needed.
GWhen it fails
Read the last line of the error first: it names the problem. These are the ones you'll meet:
| Last line says | Means | Fix |
|---|---|---|
ModuleNotFoundError: No module named 'boto3' | boto3 isn't in the kernel's Python | %pip install boto3 in a cell |
ProfileNotFound: The config profile (…) could not be found | No profile with that name | !aws configure list-profiles and copy the exact name |
NoCredentialsError: Unable to locate credentials | No profile given and no default one | Pass profile_name= to boto3.Session |
ExpiredToken, or "Token has expired and refresh failed" | A login profile's temporary credentials ran out | Run aws login (or aws sso login) for that profile again, then re-create the client |
AccessDenied | The profile isn't allowed to do that, on that path | Check bucket and key; ask whoever manages the profile if you need more access |
NoSuchKey | No object with that key: a typo, or s3:// or the bucket name left inside Key | List the prefix and copy the Key from the table |
KeyError: 'Contents' | The listing matched nothing | Check the prefix: exact and case-sensitive |
Invalid bucket name "…" | A slash in Bucket: part of the key went into the bucket | Bucket is the name alone; the rest goes in Key |
📖Read the docs: boto3's S3 clientRequired
Sometimes you want the whole file on your Mac, not a DataFrame: to open it in another tool, or to keep it as it was. The S3 client has a method that saves an object straight to a file, and it isn't in this lesson. Find it in the boto3 docs. They're laid out differently from pandas': one page per service, then one page per method.
- Open the boto3 S3 reference. Under Client, it lists every method of the client you made.
- Find the method that downloads an object to a file, and open its page. Read its Parameters: which is the bucket, which is the key, and which is the name of the file to write.
- Use it on your
s3client to download the small manifest,topic-preview/flat/quicksight-manifest.json, asmanifest.json. manifest.jsonappears next to your notebook: a small file (a few hundred bytes) that says the CSV is comma-separated with a header row.
Part 2 · Practice
Spaced practice
Practice can't reach S3 either, so it uses the same stand-in boto3 and made-up bucket as part D. Do the practice set until Load from S3 (boto3) and Load a CSV are mastered (3 clean solves each). Then this lesson's 5 cards join your Daily review from the next day.
Use it at work tomorrow
- Keep
reviews.ipynb. Its first cells (imports, the client, thenget_object+read_csv, or your local copy) are the start of every analysis from here on. - Load a file you actually use at work, with your work profile in
boto3.Session(profile_name=…). If it fails with an expired token, that's the login profile timing out: run the login command again. - Before loading something big, check its
Sizein a listing, and start withnrows=. Then open one boto3 script from your project and find itsget_objectorlist_objects_v2call: you can read it now.
Next lesson: the file you just loaded has odd column types and hidden gaps. Lesson 07 finds and fixes them.
boto3 user guide: Amazon S3 and boto3: Credentials (how a profile is found). pandas: Reading/writing remote files shows the s3:// shortcut. AWS CLI: configuration and credential files explains profiles and ~/.aws. Jupyter: getting started covers notebooks and kernels.