Stop scanning the entire S3 dataset when you only need a handful of Parquet files

When working with large datasets on AWS S3, Parquet is one of the most popular storage formats for data engineering and analytics workloads.

It’s columnar.

It’s compressed.

It supports predicate pushdown.

And it works extremely well with distributed processing frameworks such as Dask, Spark, and Ray.

But there is a common scenario that doesn’t always get enough attention.

Imagine you have a directory in S3 containing thousands of Parquet files:

s3://my-bucket/data/
    part-00001.parquet
    part-00002.parquet
    part-00003.parquet
    ...
    part-10000.parquet

Your application only needs three files:

part-00021.parquet
part-00452.parquet
part-08721.parquet

Do you really want Dask to scan the entire directory?

Probably not.

The question then becomes:

Can we pass a specific list of S3 Parquet files to Dask instead of reading the entire directory?

Yes.

And in many situations, passing an explicit list of paths is the cleanest solution.

The Traditional Approach

A typical Dask Parquet read looks like this:

import dask.dataframe as dd
df = dd.read_parquet(
    "s3://my-bucket/data/*.parquet"
)

This is convenient when you want to process the entire dataset.

You can also specify a single file:

df = dd.read_parquet(
    "s3://my-bucket/data/part-00001.parquet"
)

But what if you have a list?

For example:

files = [
    "s3://my-bucket/data/part-00021.parquet",
    "s3://my-bucket/data/part-00452.parquet",
    "s3://my-bucket/data/part-08721.parquet",
]

You can pass the list directly to dd.read_parquet().

import dask.dataframe as dd
files = [
    "s3://my-bucket/data/part-00021.parquet",
    "s3://my-bucket/data/part-00452.parquet",
    "s3://my-bucket/data/part-08721.parquet",
]
df = dd.read_parquet(files)

That’s it.

Dask will create a DataFrame representing the selected Parquet files.

The important thing is that you’re giving Dask the exact input files you want.

A Complete Example

Let’s say your S3 bucket looks like this:

s3://analytics-bucket/sales/
    sales_2024_01.parquet
    sales_2024_02.parquet
    sales_2024_03.parquet
    sales_2024_04.parquet
    sales_2024_05.parquet
    sales_2024_06.parquet

Your user selects:

January
March
June

Your application can translate that selection into S3 paths:

selected_files = [
    "s3://analytics-bucket/sales/sales_2024_01.parquet",
    "s3://analytics-bucket/sales/sales_2024_03.parquet",
    "s3://analytics-bucket/sales/sales_2024_06.parquet",
]

Then:

import dask.dataframe as dd
df = dd.read_parquet(selected_files)
print(df)

Dask creates a lazy DataFrame.

Remember:

Dask is lazy by default.

Calling:

df = dd.read_parquet(selected_files)

doesn’t necessarily mean all data has already been loaded into memory.

The actual computation generally happens when you perform an operation that requires execution, such as:

result = df.compute()

or when you write the result:

df.to_parquet(
    "s3://analytics-bucket/output/"
)

This lazy execution model is one of the reasons Dask works well for large datasets.

Building the File List Dynamically

In real applications, the file list usually comes from somewhere else.

For example:

User Input
    │
    ▼
Selected File IDs
    │
    ▼
Build S3 Paths
    │
    ▼
Dask read_parquet()

Suppose a user selects these file names:

user_selection = [
    "part-00021.parquet",
    "part-00452.parquet",
    "part-08721.parquet",
]

You can construct the full S3 paths:

bucket = "my-bucket"
prefix = "path/to/data"
files = [
    f"s3://{bucket}/{prefix}/{filename}"
    for filename in user_selection
]

Then:

import dask.dataframe as dd
df = dd.read_parquet(files)

This approach is particularly useful when building:

• Data analytics applications
• Interactive dashboards
• Data processing APIs
• Batch processing pipelines
• User-driven data exploration tools

What If the User Provides File IDs?

Imagine your application stores metadata about files in a database.

For example:

File ID     S3 Path
--------------------------------------------
101         s3://bucket/data/file_101.parquet
102         s3://bucket/data/file_102.parquet
103         s3://bucket/data/file_103.parquet

The user selects:

selected_ids = [101, 103]

Your application first looks up the corresponding S3 paths:

files = [
    "s3://bucket/data/file_101.parquet",
    "s3://bucket/data/file_103.parquet",
]

Then:

df = dd.read_parquet(files)

This gives you a clean separation between:

Application Layer
       │
       ▼
User Selection
       │
       ▼
Metadata / File Catalog
       │
       ▼
S3 Paths
       │
       ▼
      Dask

This pattern is often more maintainable than allowing your application to directly construct arbitrary S3 paths.

Using S3 URLs With Dask

When reading from S3, Dask typically relies on an S3 filesystem implementation such as s3fs.

For example:

import dask.dataframe as dd
files = [
    "s3://my-bucket/data/file1.parquet",
    "s3://my-bucket/data/file2.parquet",
]
df = dd.read_parquet(files)

Your environment needs appropriate AWS credentials and permissions.

The identity used by your application needs access to the relevant S3 objects.

For example, the IAM permissions may need to allow operations such as:

s3:GetObject
s3:ListBucket

depending on how your application accesses the data.

In AWS environments such as EC2, ECS, or EKS, it’s generally preferable to use IAM roles rather than hardcoding AWS access keys.

Method 1: Explicit File List

The simplest solution is:

files = [
    "s3://bucket/data/a.parquet",
    "s3://bucket/data/c.parquet",
    "s3://bucket/data/f.parquet",
]
df = dd.read_parquet(files)

This is ideal when you already know the exact files.

For example:

User selects files
        │
        ▼
Application creates list
        │
        ▼
dd.read_parquet(list)

Advantages

• Simple
• Explicit
• Easy to understand
• Avoids scanning unrelated files

Best for

• Small number of selected files
• User-driven file selection
• Batch processing
• Known file paths

Method 2: Filter Files Before Reading

Sometimes you don’t have the exact paths.

Instead, you know a rule.

For example:

“Read all files for January 2024.”

Your S3 directory might contain:

sales_2023_12.parquet
sales_2024_01.parquet
sales_2024_02.parquet
sales_2024_03.parquet

You can first identify the files you want and then pass them to Dask.

Conceptually:

selected_files = [
    path
    for path in all_files
    if "2024_01" in path
]
df = dd.read_parquet(selected_files)

However, be careful with this approach when the bucket contains a very large number of objects.

Listing thousands or millions of S3 keys just to select a few files may itself become expensive or slow.

In those situations, a metadata catalog or partitioned dataset layout can be a better solution.

Method 3: Use Hive-Style Partitioning

For large datasets, the best solution may not be manually selecting individual files at all.

Instead, organize the S3 data using partitions.

For example:

s3://my-bucket/sales/
    year=2024/
        month=01/
            part-001.parquet
            part-002.parquet
        month=02/
            part-003.parquet
            part-004.parquet
    year=2025/
        month=01/
            part-005.parquet

Now you can read the dataset and filter based on partition columns.

For example:

df = dd.read_parquet(
    "s3://my-bucket/sales/",
    filters=[
        ("year", "==", 2024),
        ("month", "==", 1),
    ],
)

The exact supported behavior can depend on your Dask and Parquet engine versions, so always check the version-specific documentation.

The important idea is:

Partition your data according to the queries you expect to perform.

Instead of:

Find 3 files manually

you can express:

Give me January 2024

This is much more scalable.

Explicit File Selection vs. Partition Filtering

These two approaches solve slightly different problems.

Explicit File List

files = [
    "s3://bucket/data/a.parquet",
    "s3://bucket/data/c.parquet",
]
df = dd.read_parquet(files)

Use this when you know exactly which files to process.

Partition Filtering

df = dd.read_parquet(
    "s3://bucket/data/",
    filters=[
        ("year", "==", 2024),
    ],
)

Use this when your dataset is logically partitioned and you want to select data based on partition values.

The decision looks like:

Do I know the exact files?
        │
        ├── Yes
        │    │
        │    ▼
        │  Explicit File List
        │
        └── No
             │
             ▼
       Can I filter by partition?
             │
             ├── Yes → Partition Filtering
             │
             └── No  → File Discovery + Selection

A Practical Example: User-Driven Data Processing

Imagine you’re building a web application.

A user uploads or selects three datasets:

customers.parquet
orders.parquet
transactions.parquet

These files are stored in S3.

Your backend receives:

{
    "files": [
        "customers.parquet",
        "orders.parquet"
    ]
}

Your Python application converts them to S3 paths:

import dask.dataframe as dd
bucket = "my-data-bucket"
prefix = "uploads"
selected_files = [
    f"s3://{bucket}/{prefix}/{filename}"
    for filename in request_data["files"]
]
df = dd.read_parquet(selected_files)

Now only the selected files are part of the Dask computation.

The architecture looks like:

                  User
                    │
                    ▼
             Web Application
                    │
                    ▼
             Selected Files
                    │
                    ▼
             Backend / Flask
                    │
                    ▼
              S3 File Paths
                    │
                    ▼
            Dask read_parquet()
                    │
                    ▼
             Dask DataFrame
                    │
                    ▼
              Data Processing

This pattern is useful for data-intensive applications where users dynamically control which datasets should be processed.

Don’t Confuse File Selection With Row Filtering

There are two different optimization problems.

Problem 1: Select Specific Files

You want:

file1.parquet
file5.parquet
file9.parquet

Use:

df = dd.read_parquet(files)

Problem 2: Select Specific Rows

You want:

All sales files
but only:
year = 2024
region = "North"

Now you’re talking about filtering rows and potentially using Parquet predicate pushdown.

For example:

df = dd.read_parquet(
    "s3://bucket/sales/",
    filters=[
        ("year", "==", 2024),
        ("region", "==", "North"),
    ],
)

The distinction is important.

File-Level Selection
        │
        ▼
Which files should I read?
        │
        ▼
Explicit paths / partition pruning
Row-Level Selection
        │
        ▼
Which records should I process?
        │
        ▼
Predicate pushdown / filters

A well-designed Parquet dataset can benefit from both.

What About Reading Only Certain Columns?

Parquet is columnar, which means you can also avoid reading columns you don’t need.

For example:

df = dd.read_parquet(
    files,
    columns=[
        "customer_id",
        "amount",
        "transaction_date",
    ],
)

This can significantly reduce I/O when your Parquet files contain many columns.

Imagine a dataset with 100 columns.

Your application needs only three.

Instead of:

100 columns
↓
Read everything

you can request:

customer_id
amount
transaction_date

The Parquet engine can read only the required columns.

This is one of the major advantages of columnar storage.

Combining File Selection and Column Selection

You can combine these techniques.

import dask.dataframe as dd
files = [
    "s3://bucket/data/file1.parquet",
    "s3://bucket/data/file5.parquet",
    "s3://bucket/data/file9.parquet",
]
df = dd.read_parquet(
    files,
    columns=[
        "customer_id",
        "amount",
        "transaction_date",
    ],
)

Now your pipeline is doing two forms of pruning:

S3 Dataset
    │
    ├── File Pruning
    │       ↓
    │   3 selected files
    │
    └── Column Pruning
            ↓
       3 selected columns

This can significantly reduce the amount of data that needs to be read.

A Word About Performance

It’s tempting to think:

“If I give Dask fewer files, everything will automatically become faster.”

Usually, that’s directionally correct—but there are nuances.

If you pass thousands of individual file paths, you may create overhead in:

• S3 metadata operations
• Task graph construction
• File opening
• Network requests

For a handful of files:

files = [
    "s3://bucket/a.parquet",
    "s3://bucket/b.parquet",
]

is perfectly reasonable.

For millions of files, rethink the dataset design.

You may want:

• Partitioned Parquet datasets
• A metadata catalog
• Better file sizing
• Compaction
• Predicate pushdown
• Partition pruning

The architecture matters more than the API call.

The Small Files Problem

One of the biggest problems in cloud data lakes is the small files problem.

Imagine:

1,000,000 Parquet files

Each file is only:

10 KB

That’s generally a poor layout for distributed processing.

The system now has to deal with a huge number of:

• S3 metadata operations
• File opens
• Network requests
• Dask tasks

A better approach may be to compact them into larger files.

For example:

Before:
1,000,000 × 10 KB
After:
1,000 × 10 MB

The exact ideal file size depends on your workload and infrastructure, but the general principle is:

Don’t create one tiny Parquet file for every small unit of data.

This becomes especially important when your application dynamically selects files.

What I Would Recommend

If your requirement is exactly:

“I have a list of specific Parquet files selected by the user, and I want to read only those files.”

Use:

import dask.dataframe as dd
files = [
    "s3://bucket/path/file1.parquet",
    "s3://bucket/path/file5.parquet",
    "s3://bucket/path/file9.parquet",
]
df = dd.read_parquet(files)

If your requirement is:

“I want all files matching a logical condition such as year, month, or region.”

Prefer a partitioned dataset:

df = dd.read_parquet(
    "s3://bucket/path/",
    filters=[
        ("year", "==", 2024),
        ("month", "==", 1),
    ],
)

If your requirement is:

“I have millions of files and need scalable discovery and filtering.”

Consider redesigning the storage layer using:

• Partitioned Parquet
• A metadata catalog
• A lakehouse table format
• Compaction
• Data lineage and metadata management

Final Takeaway

The answer to the original question is simple:

Yes, you can pass a list of Parquet file paths to Dask’s read_parquet() function.

For example:

import dask.dataframe as dd
files = [
    "s3://bucket/path/file1.parquet",
    "s3://bucket/path/file2.parquet",
    "s3://bucket/path/file3.parquet",
]
df = dd.read_parquet(files)

But the bigger lesson is about designing your data lake.

For small, explicit selections:

Explicit File List
        ↓
dd.read_parquet(files)

For logical selections:

Partitioned Dataset
        ↓
Partition Pruning

For row-level filtering:

Predicate Pushdown

For large-scale datasets:

Good Data Layout
        +
Partitioning
        +
Column Pruning
        +
Predicate Pushdown
        +
Efficient File Sizes

The most important optimization isn’t always a clever Python trick.

Sometimes, it’s simply organizing your data so that you don’t have to read data you don’t need in the first place.

That’s the real power of designing an efficient Parquet-based data lake on S3.

Follow me on Medium