Lesson 06 · Data Science → AI/ML Engineering · Lab

Get your data: from S3 into a notebook

Your CSV files live in an S3 bucket, so everything this course teaches only helps at work once you can get them into Python. This lesson sets that up once: a Jupyter notebook, your AWS login, and boto3, AWS's own Python library and the one your projects use, to fetch a file from S3 into pandas.

≈ 15 minDo → Check → PracticeBuilds on: Lesson 05

This is a Lab: you do the steps on your Mac, in a real notebook, not on this page. The browser can't log in to AWS or reach your bucket. Each part ends with a Check: answer Yes, No or Not sure. If you answer No or Not sure, you get troubleshooting help, and you can paste the error you saw. The lesson is done when every Check is Yes.

The code boxes in parts D and E are different: they run here in the browser, against a stand-in boto3 with a small made-up bucket called shop-data, so you can try each call before using it on the real bucket.

Part 1 · Learn and do

AOpen a notebook

A Jupyter notebook is a file (ending .ipynb) made of cells. You type Python into a cell, run it, and its output appears right under it, like the code boxes in these lessons. Behind every notebook is a kernel: the Python program that actually runs your cells. Which kernel you pick decides which installed packages your code can use.

ActionHow
Run a cellShift + Enter: runs it and moves to the next cell
Install a package into the kernel's Python%pip install boto3 in a cell. The % makes it install into the kernel's Python, not some other Python on your Mac.
Run a terminal command from a cellStart the line with !, e.g. !aws --version
Start cleanRestart (clears everything the notebook remembers), then Run All
  1. In VS Code, install the Python and Jupyter extensions if you don't have them (Extensions panel, search each name). Make a folder for work notebooks outside this course folder, because this folder syncs and real data shouldn't.
  2. File → New File… → Jupyter Notebook. Save it in that folder as reviews.ipynb.
  3. Top right, click Select Kernel and pick a Python environment (a Python 3 from Homebrew or a virtual environment is fine).
  4. In the first cell, type this and press Shift + Enter:
    import pandas as pd
    pd.__version__
    If it says No module named 'pandas', run %pip install pandas in a cell, then run the first cell again.
  5. In the next cell, run %pip install boto3. It isn't installed on your Mac yet: the aws command carries its own private copy, which your notebook can't use.
  6. A version number such as 2.2.3 or 3.0.2 prints under the cell. (Note it down and tell me: the proof project will follow your version.)

BThe one big difference: a notebook remembers

In these lessons, every Run starts fresh. A notebook doesn't: a variable made in one cell stays until you restart. Cells can run in any order, and the number beside each one ([3]) shows the order they actually ran in. That's handy, because you load a big file once and reuse it. But it's also the most common source of "it worked a minute ago". If results look odd, Restart and Run All: if the notebook still works from top to bottom, you can trust it.

  1. In a new cell, run x = 1. In the cell below it, type x = x + 1 and then x on the next line.
  2. Run that second cell three times, then Restart the kernel and Run All.
  3. After three runs the second cell shows 4, because it kept adding to the remembered x. After Restart and Run All it shows 2.

CYour AWS login, briefly

To read from S3, your code has to say who you are. boto3 does that with a profile: a named set of credentials saved in two files in your home folder, ~/.aws/config and ~/.aws/credentials. Your code never holds a password; it only names the profile. There are two kinds of profile, and you use both:

KindHow it worksExample
Key profileA fixed access key saved in ~/.aws/credentials. It doesn't expire, so there's nothing to refresh.study-reviews-read: read-only
Login profileNo saved key. aws login --profile NAME (or aws sso login --profile NAME, depending on how your account is set up) opens the browser, you sign in, and the CLI saves temporary credentials under that profile name. They expire after some hours, and then you run the login command again.Your everyday work profile

So the login command and the profile work together: the command refreshes what the profile holds, and your code just says which profile to use. Two commands help when you're not sure what you have:

CommandPrints
aws configure list-profilesThe names of every profile on this Mac
aws sts get-caller-identity --profile NAMEWho AWS thinks you are with that profile: an Arn ending in a user or role name. If it fails, the profile is wrong or its login has expired.
  1. In a notebook cell, run !aws configure list-profiles.
  2. Then run !aws sts get-caller-identity --profile study-reviews-read.
  3. The list includes study-reviews-read, and the second command prints an Arn ending in user/study-reviews-read.

DReach the bucket

S3 stores files as objects in a bucket. Each object has a key: its full name inside the bucket, such as topic-preview/flat/flat.csv. The slashes make keys look like folders, and the AWS console shows them as folders, but S3 just matches the start of the name: a prefix. boto3 always takes the bucket and the key separately: no s3://, and no bucket name inside the key.

SyntaxReturns
import boto3
s3 = boto3.Session(profile_name="NAME").client("s3")
A client: the object you call S3 through, logged in with that profile. The Session picks the profile; .client("s3") picks the service.
r = s3.list_objects_v2(Bucket="shop-data", Prefix="exports/")A dictionary about the objects whose key starts with the prefix. r["KeyCount"] is how many; r["Contents"] is a list with one small dictionary per object (its Key, Size in bytes, and more). At most 1,000 at a time.
pd.DataFrame(r["Contents"])That list as a table, one row per object. The quickest way to read a listing.

Careful: when nothing matches the prefix, the dictionary has no "Contents" key at all, so r["Contents"] fails with KeyError: 'Contents'. Prefixes are exact and case-sensitive: export/ doesn't match exports/. Check r["KeyCount"] first.

  1. Back in your notebook:
    import boto3
    s3 = boto3.Session(profile_name="study-reviews-read").client("s3")
    r = s3.list_objects_v2(Bucket="rac-reviews", Prefix="topic-preview/")
    pd.DataFrame(r["Contents"])[["Key", "Size"]]
  2. A table of 4 objects: flat.csv and pairs.csv, each with a small quicksight-manifest.json. flat.csv's Size is about 276,600,000 bytes (≈ 277 MB).

ELoad the CSV

SyntaxReturns
obj = s3.get_object(Bucket="shop-data", Key="exports/export.csv")A dictionary about the object. obj["Body"] is the file's contents as a stream: like an open file, read from start to end.
df = pd.read_csv(obj["Body"])A DataFrame. read_csv takes a file name or anything that reads like a file. Its first line (the header) becomes the column names, every other line a row, labelled 0, 1, 2… An empty value becomes NaN.
pd.read_csv(…, nrows=5000)Only the first 5,000 rows. Your real file is hundreds of MB, so a first look should read only a slice.
pd.read_csv(…, usecols=["review_id", "rating"])Only those columns, in the file's order (not the list's).
df.shapeA pair, (rows, columns). No brackets after shape: it's a property, not a method.
df.head(3)The first 3 rows (5 if you leave the number out).

Two steps: boto3 fetches the object, pandas reads it. If something fails, the error's name tells you which step. A stream is read once: to load the file again, call get_object again. The same read_csv also opens a local file by name: pd.read_csv("export.csv").

  1. In the notebook, load the first 5,000 rows of the real file:
    obj = s3.get_object(Bucket="rac-reviews", Key="topic-preview/flat/flat.csv")
    df = pd.read_csv(obj["Body"], nrows=5000)
    df.shape
  2. Run df.head() and scroll sideways through the columns.
  3. df.shape shows (5000, 41): 5,000 rows and 41 columns, in a few seconds.

You'll also see: pd.read_csv("s3://…"). pandas can open an S3 path directly: pd.read_csv("s3://rac-reviews/topic-preview/flat/flat.csv", storage_options={"profile": "study-reviews-read"}). It needs an extra package, s3fs, which makes the same S3 requests behind the scenes and reads the same profiles. Coding agents and the pandas docs use it in notebooks. You don't need to write it; when you see it, read it as "get_object + read_csv in one line".

FKeep a local copy

SyntaxDoes
df.to_csv("flat-sample.csv", index=False)Writes df to a CSV file next to the notebook. index=False leaves out the row labels 0, 1, 2…; without it, the file gets an extra unnamed first column.

Fetching from S3 every time is slow, and it fails when your login has expired. Save the slice once, then load the local copy while you explore. Go back to S3 when you need fresh or more data. Keep these files in your work folder, not in this course folder.

  1. Run df.to_csv("flat-sample.csv", index=False).
  2. Restart the kernel (so df is gone), then in a cell: import pandas as pd, local = pd.read_csv("flat-sample.csv"), local.shape.
  3. flat-sample.csv appears next to the notebook, and local.shape is (5000, 41) again, with no AWS login needed.

GWhen it fails

Read the last line of the error first: it names the problem. These are the ones you'll meet:

Last line saysMeansFix
ModuleNotFoundError: No module named 'boto3'boto3 isn't in the kernel's Python%pip install boto3 in a cell
ProfileNotFound: The config profile (…) could not be foundNo profile with that name!aws configure list-profiles and copy the exact name
NoCredentialsError: Unable to locate credentialsNo profile given and no default onePass profile_name= to boto3.Session
ExpiredToken, or "Token has expired and refresh failed"A login profile's temporary credentials ran outRun aws login (or aws sso login) for that profile again, then re-create the client
AccessDeniedThe profile isn't allowed to do that, on that pathCheck bucket and key; ask whoever manages the profile if you need more access
NoSuchKeyNo object with that key: a typo, or s3:// or the bucket name left inside KeyList the prefix and copy the Key from the table
KeyError: 'Contents'The listing matched nothingCheck the prefix: exact and case-sensitive
Invalid bucket name "…"A slash in Bucket: part of the key went into the bucketBucket is the name alone; the rest goes in Key

📖Read the docs: boto3's S3 clientRequired

Sometimes you want the whole file on your Mac, not a DataFrame: to open it in another tool, or to keep it as it was. The S3 client has a method that saves an object straight to a file, and it isn't in this lesson. Find it in the boto3 docs. They're laid out differently from pandas': one page per service, then one page per method.

  1. Open the boto3 S3 reference. Under Client, it lists every method of the client you made.
  2. Find the method that downloads an object to a file, and open its page. Read its Parameters: which is the bucket, which is the key, and which is the name of the file to write.
  3. Use it on your s3 client to download the small manifest, topic-preview/flat/quicksight-manifest.json, as manifest.json.
  4. manifest.json appears next to your notebook: a small file (a few hundred bytes) that says the CSV is comma-separated with a header row.

Part 2 · Practice

Spaced practice

Practice can't reach S3 either, so it uses the same stand-in boto3 and made-up bucket as part D. Do the practice set until Load from S3 (boto3) and Load a CSV are mastered (3 clean solves each). Then this lesson's 5 cards join your Daily review from the next day.

Use it at work tomorrow

  1. Keep reviews.ipynb. Its first cells (imports, the client, then get_object + read_csv, or your local copy) are the start of every analysis from here on.
  2. Load a file you actually use at work, with your work profile in boto3.Session(profile_name=…). If it fails with an expired token, that's the login profile timing out: run the login command again.
  3. Before loading something big, check its Size in a listing, and start with nrows=. Then open one boto3 script from your project and find its get_object or list_objects_v2 call: you can read it now.

Next lesson: the file you just loaded has odd column types and hidden gaps. Lesson 07 finds and fixes them.

Primary sources

boto3 user guide: Amazon S3 and boto3: Credentials (how a profile is found). pandas: Reading/writing remote files shows the s3:// shortcut. AWS CLI: configuration and credential files explains profiles and ~/.aws. Jupyter: getting started covers notebooks and kernels.

Ask me anything. If a step fails, the Check's "Paste what you saw" box takes the whole error. Paste it from the first line to the last.