A practical guide to uploading, downloading, filtering, and managing files in AWS S3 with Python

Amazon S3 is one of the most commonly used AWS services for storing files and datasets.

In the previous tutorial, we learned how to list and print the contents of an S3 bucket using Python and Boto3.

But listing files is usually only the beginning.

In a real application or data pipeline, you will often need to:

• Upload files from Python to S3
• Download files from S3
• Upload files into specific folders
• Download files to a specific location
• Upload file-like objects
• Add metadata to objects
• Handle large file transfers
• Configure concurrent transfers
• Handle AWS errors properly

In this tutorial, we’ll build these operations step by step using Boto3, the AWS SDK for Python.

What Is Boto3?

Boto3 is the AWS SDK for Python.

It allows Python applications to communicate with AWS services programmatically.

For example:

import boto3

You can then create an S3 client:

s3 = boto3.client("s3")

AWS’s current Boto3 documentation supports both client and resource interfaces for S3. For this tutorial, we’ll primarily use the client interface because it makes the underlying S3 operations explicit. (AWS Documentation)

Prerequisites

Before starting, you’ll need:

• Python installed
• An AWS account
• An S3 bucket
• AWS credentials configured
• Boto3 installed

Install Boto3:

pip install boto3

AWS recommends using a virtual environment for Python projects and configuring credentials through supported AWS credential mechanisms rather than hard-coding secrets in your source code. (AWS Documentation)

Configure AWS Credentials

One simple approach for local development is:

aws configure

You’ll be prompted for:

AWS Access Key ID
AWS Secret Access Key
Default region name
Default output format

Boto3 can then automatically discover the configured credentials.

For production applications running on AWS, prefer an appropriate IAM role rather than embedding access keys inside your application.

Create an S3 Client

Let’s start with the basic setup.

import boto3
s3 = boto3.client("s3")

Now the Python application can make requests to Amazon S3.

Suppose our bucket is:

my-data-bucket

We’ll use that bucket throughout the examples.

Uploading a File to S3

The simplest way to upload a local file is upload_file().

import boto3
s3 = boto3.client("s3")
s3.upload_file(
    "customers.csv",
    "my-data-bucket",
    "customers.csv"
)

There are three important arguments here:

Local file
     ↓
S3 bucket
     ↓
S3 object key

In this example:

Local file:
customers.csv
Bucket:
my-data-bucket
S3 key:
customers.csv

The resulting object is:

s3://my-data-bucket/customers.csv

Boto3’s upload_file() method is specifically designed for transferring files to S3 and can handle large files using multipart transfer behavior. (AWS Documentation)

Upload a File Into an S3 Folder

Remember from the previous tutorial that S3 doesn’t fundamentally use traditional folders.

Instead, object keys create the folder-like structure.

So if we upload:

s3.upload_file(
    "customers.csv",
    "my-data-bucket",
    "data/customers.csv"
)

the object key becomes:

data/customers.csv

In the AWS console, this appears as:

my-data-bucket
└── data
    └── customers.csv

You can therefore organize uploaded files using prefixes.

For example:

data/raw/customers.csv
data/raw/orders.csv
data/processed/customers.parquet
data/processed/orders.parquet

Create a Reusable Upload Function

Instead of repeating the same code, let’s create a function.

import boto3
s3 = boto3.client("s3")
def upload_file(
    local_file,
    bucket_name,
    s3_key
):
    s3.upload_file(
        local_file,
        bucket_name,
        s3_key
    )
    print(
        f"Uploaded {local_file} "
        f"to s3://{bucket_name}/{s3_key}"
    )

Now we can call:

upload_file(
    "customers.csv",
    "my-data-bucket",
    "data/customers.csv"
)

Output:

Uploaded customers.csv to s3://my-data-bucket/data/customers.csv

Handling Upload Errors

In production code, you should handle AWS exceptions.

import boto3
from botocore.exceptions import ClientError
s3 = boto3.client("s3")
def upload_file(
    local_file,
    bucket_name,
    s3_key
):
    try:
        s3.upload_file(
            local_file,
            bucket_name,
            s3_key
        )
        print("Upload successful")
    except ClientError as e:
        print(
            f"Upload failed: {e}"
        )

This is particularly useful for handling situations such as:

• Access denied
• Invalid bucket names
• Missing permissions
• Authentication problems
• AWS service errors

Downloading a File From S3

Now let’s reverse the process.

Suppose we have:

s3://my-data-bucket/data/customers.csv

and want to download it to:

customers.csv

Use:

import boto3
s3 = boto3.client("s3")
s3.download_file(
    "my-data-bucket",
    "data/customers.csv",
    "customers.csv"
)

The arguments are:

Bucket
S3 object key
Local destination

AWS documents download_file() as the standard Boto3 method for downloading an S3 object to a local file. (AWS Documentation)

Download to a Specific Directory

You can also specify a complete local path.

s3.download_file(
    "my-data-bucket",
    "data/customers.csv",
    "/tmp/customers.csv"
)

On Windows, for example:

s3.download_file(
    "my-data-bucket",
    "data/customers.csv",
    r"C:\data\customers.csv"
)

Create a Reusable Download Function

Let’s wrap the operation in a function.

import boto3
s3 = boto3.client("s3")
def download_file(
    bucket_name,
    s3_key,
    local_file
):
    s3.download_file(
        bucket_name,
        s3_key,
        local_file
    )
    print(
        f"Downloaded s3://{bucket_name}/{s3_key} "
        f"to {local_file}"
    )

Use it like:

download_file(
    "my-data-bucket",
    "data/customers.csv",
    "customers.csv"
)

Uploading File-Like Objects

Sometimes the file isn’t physically stored on disk.

For example, you may have data in:

• BytesIO
• an in-memory buffer
• a web request
• a generated CSV
• a Pandas DataFrame

In that situation, upload_fileobj() can be useful.

For example:

import boto3
s3 = boto3.client("s3")
with open("customers.csv", "rb") as file:
    s3.upload_fileobj(
        file,
        "my-data-bucket",
        "data/customers.csv"
    )

The file object should be opened in binary mode. AWS documents upload_fileobj() specifically for readable file-like objects. (AWS Documentation)

Downloading Into a File-Like Object

The reverse operation is also possible.

import boto3
s3 = boto3.client("s3")
with open("customers.csv", "wb") as file:
    s3.download_fileobj(
        "my-data-bucket",
        "data/customers.csv",
        file
    )

Notice the difference:

Upload:

"rb"

Download:

"wb"

Because we’re dealing with binary data.

AWS provides download_fileobj() for writing the downloaded object into a writable file-like object. (AWS Documentation)

Uploading a Pandas DataFrame Directly to S3

This becomes particularly useful in data engineering.

Suppose we have:

import pandas as pd
df = pd.DataFrame({
    "name": ["Alice", "Bob"],
    "age": [25, 30]
})

We don’t necessarily need to save the DataFrame to disk first.

We can write it to an in-memory buffer.

import io
import boto3
import pandas as pd
df = pd.DataFrame({
    "name": ["Alice", "Bob"],
    "age": [25, 30]
})
buffer = io.BytesIO()
df.to_csv(
    buffer,
    index=False
)
buffer.seek(0)
s3 = boto3.client("s3")
s3.upload_fileobj(
    buffer,
    "my-data-bucket",
    "data/customers.csv"
)

The flow becomes:

Pandas DataFrame
       ↓
Memory Buffer
       ↓
Boto3
       ↓
Amazon S3

This can be useful in serverless applications and data pipelines where you don’t want to create intermediate local files.

Adding Metadata During Upload

S3 objects can contain metadata.

For example:

s3.upload_file(
    "customers.csv",
    "my-data-bucket",
    "data/customers.csv",
    ExtraArgs={
        "Metadata": {
            "source": "python",
            "dataset": "customers"
        }
    }
)

Now the uploaded object contains custom metadata.

AWS’s Boto3 upload APIs support ExtraArgs for additional upload options, including metadata. (AWS Documentation)

Adding Content Type

You can also specify the content type.

For example, for JSON:

s3.upload_file(
    "data.json",
    "my-data-bucket",
    "data/data.json",
    ExtraArgs={
        "ContentType": "application/json"
    }
)

For CSV:

s3.upload_file(
    "customers.csv",
    "my-data-bucket",
    "data/customers.csv",
    ExtraArgs={
        "ContentType": "text/csv"
    }
)

This becomes useful when the object will later be served through another application or HTTP-based workflow.

What Happens With Large Files?

This is where Boto3 becomes particularly useful.

You might expect:

Large File
   ↓
One HTTP request
   ↓
S3

But Boto3’s managed transfer functionality can automatically handle multipart transfers and concurrency for suitable operations. (AWS Documentation)

Conceptually:

                 ┌→ Part 1 ─┐
Large File ──────┼→ Part 2 ─┼→ S3
                 ├→ Part 3 ─┤
                 └→ Part 4 ─┘

This allows large transfers to be split into multiple parts.

You generally don’t need to implement multipart upload logic yourself for ordinary file-transfer use cases.

Controlling Transfer Configuration

Boto3 provides TransferConfig when you need more control.

For example:

from boto3.s3.transfer import TransferConfig
import boto3
config = TransferConfig(
    max_concurrency=5
)
s3 = boto3.client("s3")
s3.upload_file(
    "large_dataset.csv",
    "my-data-bucket",
    "data/large_dataset.csv",
    Config=config
)

max_concurrency controls the maximum number of concurrent transfer operations used by the managed transfer system. AWS documents a default value of 10 and allows it to be adjusted when necessary. (AWS Documentation)

You shouldn’t blindly increase concurrency.

More concurrency can consume more bandwidth and resources.

The right setting depends on your environment.

Downloading Large Files

The same transfer configuration can be used when downloading.

from boto3.s3.transfer import TransferConfig
import boto3
config = TransferConfig(
    max_concurrency=5
)
s3 = boto3.client("s3")
s3.download_file(
    "my-data-bucket",
    "data/large_dataset.csv",
    "large_dataset.csv",
    Config=config
)

Boto3 manages the transfer details for you. (AWS Documentation)

Adding a Progress Callback

For large files, you may want to show upload progress.

Boto3’s transfer methods support a Callback parameter that receives the number of bytes transferred during the operation. (AWS Documentation)

A simple progress tracker could look like:

import os
import boto3
class ProgressPercentage:
    def __init__(self, filename):
        self.filename = filename
        self.size = os.path.getsize(filename)
        self.transferred = 0
    def __call__(self, bytes_amount):
        self.transferred += bytes_amount
        percentage = (
            self.transferred /
            self.size
        ) * 100
        print(
            f"\rProgress: {percentage:.2f}%",
            end=""
        )

Then:

s3.upload_file(
    "large_dataset.csv",
    "my-data-bucket",
    "data/large_dataset.csv",
    Callback=ProgressPercentage(
        "large_dataset.csv"
    )
)

This can be useful for command-line applications.

Checking Whether an Object Exists

Before downloading a file, you may want to check whether it exists.

One approach is:

import boto3
from botocore.exceptions import ClientError
s3 = boto3.client("s3")
def object_exists(bucket, key):
    try:
        s3.head_object(
            Bucket=bucket,
            Key=key
        )
        return True
    except ClientError:
        return False

Then:

if object_exists(
    "my-data-bucket",
    "data/customers.csv"
):
    print("File exists")
else:
    print("File does not exist")

This can be useful before performing downstream operations.

However, in production code, you should distinguish a genuine “not found” response from other errors such as permission problems rather than treating every ClientError as “missing.”

Upload → List → Download

Let’s put everything together.

Suppose we have:

customers.csv

Step 1 — Upload

s3.upload_file(
    "customers.csv",
    "my-data-bucket",
    "data/customers.csv"
)

Step 2 — List

response = s3.list_objects_v2(
    Bucket="my-data-bucket",
    Prefix="data/"
)
for obj in response.get("Contents", []):
    print(obj["Key"])

Output:

data/customers.csv

Step 3 — Download

s3.download_file(
    "my-data-bucket",
    "data/customers.csv",
    "customers_downloaded.csv"
)

The complete workflow is:

Local File
    ↓
   Boto3
    ↓
Amazon S3
    ↓
   Boto3
    ↓
Local File

A Production-Friendly Utility

We can now create a small reusable S3 utility.

import boto3
from botocore.exceptions import ClientError
class S3Manager:
    def __init__(self):
        self.s3 = boto3.client("s3")
    def upload(
        self,
        local_file,
        bucket,
        key
    ):
        try:
            self.s3.upload_file(
                local_file,
                bucket,
                key
            )
            print(
                f"Uploaded: s3://{bucket}/{key}"
            )
        except ClientError as e:
            print(
                f"Upload failed: {e}"
            )
    def download(
        self,
        bucket,
        key,
        local_file
    ):
        try:
            self.s3.download_file(
                bucket,
                key,
                local_file
            )
            print(
                f"Downloaded: {local_file}"
            )
        except ClientError as e:
            print(
                f"Download failed: {e}"
            )

Now:

s3_manager = S3Manager()

Upload:

s3_manager.upload(
    "customers.csv",
    "my-data-bucket",
    "data/customers.csv"
)

Download:

s3_manager.download(
    "my-data-bucket",
    "data/customers.csv",
    "customers.csv"
)

Client vs Resource

You may encounter another style of Boto3 code:

s3 = boto3.resource("s3")

instead of:

s3 = boto3.client("s3")

Both are supported.

For example, using the resource interface:

s3 = boto3.resource("s3")
bucket = s3.Bucket(
    "my-data-bucket"
)
bucket.upload_file(
    "customers.csv",
    "data/customers.csv"
)

And download:

bucket.download_file(
    "data/customers.csv",
    "customers.csv"
)

AWS’s documentation provides examples using both the client and resource interfaces. (AWS Documentation)

For tutorials and many application-level operations, either can be convenient. The important thing is to understand which abstraction your code is using.

Common Mistakes

1. Hard-coding AWS credentials

Avoid:

aws_access_key_id="..."
aws_secret_access_key="..."

inside application source code.

Use AWS’s credential chain, environment configuration, profiles, or IAM roles instead. (AWS Documentation)

2. Confusing the Local Filename With the S3 Key

This:

s3.upload_file(
    "customers.csv",
    "my-data-bucket",
    "data/customers.csv"
)

means:

Local:
customers.csv
S3:
data/customers.csv

They don’t have to be the same.

3. Forgetting Binary Mode

For file objects:

open(
    "customers.csv",
    "rb"
)

for reading,

and:

open(
    "customers.csv",
    "wb"
)

for writing downloaded binary content.

4. Ignoring IAM Permissions

Your Python code can be completely correct and still return:

AccessDenied

The AWS identity needs appropriate permissions for the operation.

For example, uploading requires appropriate S3 write permissions, while downloading requires read permissions.

5. Assuming Every Transfer Is Small

For large datasets, understand Boto3’s managed transfer functionality and TransferConfig rather than building your own transfer mechanism unnecessarily. (AWS Documentation)

The Bigger Picture

At this point, we have covered two important parts of working with S3 using Python.

Tutorial 1

List and inspect objects

S3
 ↓
List
 ↓
Filter
 ↓
Inspect

Tutorial 2

Upload and download objects

Local
 ↓
Upload
 ↓
S3
 ↓
Download
 ↓
Local

These operations form the foundation for many real-world AWS data workflows.

For example:

Application
     ↓
   Python
     ↓
   Boto3
     ↓
    S3
     ↓
Data Pipeline
     ↓
Processing
     ↓
Analytics / ML

What’s Next?

Now that we know how to list, upload, and download S3 objects, the next interesting problem is:

How do we process S3 files automatically when they are uploaded?

That’s where the architecture becomes much more interesting:

File Upload
     ↓
Amazon S3
     ↓
S3 Event
     ↓
AWS Lambda
     ↓
Python Processing
     ↓
S3 / Database / Queue

Instead of manually running a Python script every time a file arrives, AWS can trigger processing automatically.

That leads naturally to the next tutorial:

“Trigger an AWS Lambda Function Automatically When a File Is Uploaded to S3.”

That tutorial will take us from simple S3 scripting into event-driven AWS architecture.

Final Takeaway

Working with S3 from Python is surprisingly straightforward once you understand the Boto3 transfer APIs.

The core operations are:

s3.upload_file(...)

for uploading files,

s3.download_file(...)

for downloading files,

and:

s3.upload_fileobj(...)
s3.download_fileobj(...)

when working with file-like objects. (AWS Documentation)

For larger workloads, Boto3 also provides managed transfer behavior, multipart transfers, concurrency controls, and callbacks. (AWS Documentation)

The important mental model is:

Local File
    ↓
   Boto3
    ↓
Amazon S3
    ↓
   Boto3
    ↓
Local File

Once you are comfortable with this workflow, you can start building much more powerful systems around S3—especially when you combine it with Lambda, EventBridge, SQS, Step Functions, and data-processing services.

Check out my full tutorial series on Apache Spark
:::