Learn how Spark reads data from files and storage systems, how schema inference works, and how to build reliable data ingestion pipelines with PySpark

Almost every Spark application starts with one simple question:

Where is the data, and how do I get it into Spark?

In PySpark, the answer usually starts with:

spark.read

For example:

df = spark.read.parquet("sales/")

or:

df = spark.read.csv(
    "sales.csv",
    header=True,
    inferSchema=True
)

It looks simple.

But reading data efficiently and correctly involves much more than knowing spark.read.csv().

You need to understand:

• Data sources
• File formats
• Schema inference
• Explicit schemas
• Reader options
• Single files vs directories
• Multiple files
• Cloud storage paths
• Partitioned datasets
• Corrupt records
• Performance considerations

This article builds that foundation.

Apache Spark Learning Path

APACHE SPARK LEARNING PATH

✓ 1. What is Apache Spark?
✓ 2. Why Spark is Faster than Hadoop
✓ 3. Spark Architecture
✓ 4. Driver vs Executor
✓ 5. Cluster Managers
✓ 6. RDD vs DataFrame vs Dataset
✓ 7. SparkSession
→ 8. Reading Data
✓ 9. Lazy Evaluation
✓ 10. DAG
✓ 11. Stages and Tasks
✓ 12. Shuffle
✓ 13. Narrow vs Wide Transformations
→ 14. Writing Data
→ 15. Partitioning
→ 16. Data Skew

[View All Apache Spark Tutorials →]

The spark.read API

Once you have a SparkSession:

from pyspark.sql import SparkSession
spark = (
    SparkSession.builder
    .appName("DataReading")
    .getOrCreate()
)

you can use:

spark.read

to read external data.

For example:

df = spark.read.csv("sales.csv")

or:

df = spark.read.json("events.json")

or:

df = spark.read.parquet("sales/")

The general pattern is:

spark
  ↓
spark.read
  ↓
DataFrameReader
  ↓
Data source
  ↓
DataFrame

The result is usually a Spark DataFrame representing the data.

Reading CSV Data

CSV is one of the most common formats encountered in data engineering.

Suppose employees.csv contains:

id,name,department,salary
1,Alice,Engineering,90000
2,Bob,Finance,75000
3,Charlie,Engineering,95000

You can read it using:

df = spark.read.csv("employees.csv")

Then:

df.show()

However, without specifying options, Spark may treat the first row differently than you expect.

A more useful version is:

df = spark.read.csv(
    "employees.csv",
    header=True,
    inferSchema=True
)

Now Spark uses the first row as column names and attempts to infer appropriate data types.

Understanding header=True

Without:

header=True

Spark does not automatically assume that the first row contains column names.

For example:

df = spark.read.csv(
    "employees.csv",
    header=True
)

produces columns such as:

id
name
department
salary

This is different from simply:

spark.read.csv("employees.csv")

where the CSV reader’s default behavior does not treat the first row as a header.

Understanding inferSchema=True

By default, Spark’s CSV reader does not necessarily infer numeric and other types from the input in the way many beginners expect.

For example:

df = spark.read.csv(
    "employees.csv",
    header=True,
    inferSchema=True
)

Spark may infer:

id          → integer
name        → string
department  → string
salary      → integer

You can inspect the result with:

df.printSchema()

For example:

root
 |-- id: integer
 |-- name: string
 |-- department: string
 |-- salary: integer

Schema Inference vs Explicit Schema

Schema inference is convenient.

But it is not always the best choice for production pipelines.

Consider:

df = spark.read.csv(
    "employees.csv",
    header=True,
    inferSchema=True
)

Spark has to inspect the data to determine types.

For a large dataset, you may prefer an explicit schema.

Defining an Explicit Schema

PySpark provides types such as:

from pyspark.sql.types import (
    StructType,
    StructField,
    IntegerType,
    StringType,
    DoubleType
)

You can define:

schema = StructType([
    StructField("id", IntegerType(), True),
    StructField("name", StringType(), True),
    StructField("department", StringType(), True),
    StructField("salary", DoubleType(), True)
])

Then:

df = spark.read.csv(
    "employees.csv",
    header=True,
    schema=schema
)

Now Spark doesn’t need to infer the schema from the data.

Why Explicit Schemas Matter

Consider a production dataset where:

salary

should always be:

double

If the input data changes unexpectedly, schema inference can produce surprising results.

An explicit schema provides:

• Predictability
• Better data contracts
• Easier validation
• More reliable downstream processing
• Better control over data types

For production ingestion pipelines, explicit schemas are often preferable when the expected structure is known.

Reading JSON Data

Spark can read JSON files directly.

Suppose:

{"id":1,"name":"Alice","salary":90000}
{"id":2,"name":"Bob","salary":75000}
{"id":3,"name":"Charlie","salary":95000}

You can use:

df = spark.read.json("employees.json")

Then:

df.show()

and:

df.printSchema()

Spark can infer the structure of JSON data.

Multiline JSON

JSON can appear in different layouts.

For example:

[
    {
        "id": 1,
        "name": "Alice"
    },
    {
        "id": 2,
        "name": "Bob"
    }
]

For JSON documents where records span multiple lines, you may need:

df = spark.read.option(
    "multiLine",
    True
).json("employees.json")

The exact reader option depends on the structure of the JSON being read.

Always understand the input format rather than assuming every .json file has the same layout.

Reading Parquet Data

For analytics workloads, Parquet is one of the most important formats to understand.

You can read it with:

df = spark.read.parquet("sales/")

Then:

df.show()

and:

df.printSchema()

Parquet is a columnar storage format.

That makes it particularly well suited to analytical workloads.

For example, suppose the dataset contains:

customer_id
product_id
country
amount
timestamp

but your query only needs:

customer_id
amount

A columnar format can allow Spark to read only the required columns rather than treating the entire row as the unit of storage.

This is one reason Parquet is commonly used in data lake and analytics architectures.

Parquet and Schema

Unlike a basic CSV file, Parquet stores schema information as part of the file metadata.

Therefore:

df = spark.read.parquet("sales/")

can recover the data types from the Parquet metadata.

You generally don’t need:

inferSchema=True

for Parquet.

CSV vs JSON vs Parquet

For large-scale analytical workloads, Parquet is generally a much better fit than CSV.

Reading Data From a Directory

One important Spark feature is that you don’t have to read only one file.

Suppose:

sales/
├── sales_2026_01.csv
├── sales_2026_02.csv
├── sales_2026_03.csv
└── sales_2026_04.csv

You can read the directory:

df = spark.read.csv(
    "sales/",
    header=True,
    inferSchema=True
)

Spark can process the files as a distributed dataset.

This is a fundamental idea in data engineering:

Directory
   ↓
Multiple files
   ↓
Spark
   ↓
Distributed DataFrame

You don’t need to manually read each file and union them together.

Reading Multiple Paths

You can also provide multiple paths.

For example:

df = spark.read.parquet(
    "sales/january/",
    "sales/february/",
    "sales/march/"
)

This can be useful when data is stored across multiple locations.

Reading From Cloud Storage

In real-world data engineering, your data is often not sitting on the local machine.

It may live in:

Amazon S3
Azure Data Lake Storage
Google Cloud Storage
HDFS

For example:

df = spark.read.parquet(
    "s3://my-bucket/sales/"
)

Or:

df = spark.read.parquet(
    "abfss://container@storageaccount.dfs.core.windows.net/sales/"
)

The exact path and authentication mechanism depend on your Spark environment and cloud platform.

The important idea is that the DataFrameReader API remains conceptually similar:

spark.read.parquet(path)

Only the underlying storage location changes.

Reading Tables

Spark can also read tables rather than directly referencing file paths.

For example:

df = spark.read.table("sales")

or:

df = spark.table("sales")

You can also use SQL:

df = spark.sql("""
    SELECT *
    FROM sales
""")

This is particularly common in environments using managed catalogs and table-based data architectures.

The DataFrameReader API

The object behind:

spark.read

is the DataFrameReader.

It provides methods such as:

csv()
json()
parquet()
orc()
text()
textFile()
table()
load()

The generic form is:

spark.read.format("format").load("path")

For example:

df = (
    spark.read
    .format("parquet")
    .load("sales/")
)

This is equivalent in concept to:

df = spark.read.parquet("sales/")

Using format() and load()

The generic API becomes especially useful when you want to configure a data source explicitly.

For example:

df = (
    spark.read
    .format("csv")
    .option("header", "true")
    .option("inferSchema", "true")
    .load("employees.csv")
)

This style makes the relationship between:

format
options
path

very explicit.

Conceptually:

spark.read
    ↓
format()
    ↓
option()
    ↓
load()
    ↓
DataFrame

Reader Options

Spark readers support many options.

For CSV, you might encounter:

df = (
    spark.read
    .option("header", "true")
    .option("inferSchema", "true")
    .option("delimiter", ",")
    .option("quote", '"')
    .option("escape", '"')
    .csv("employees.csv")
)

Don’t memorize every option.

Instead, learn the pattern:

spark.read.option("key", "value")

and consult the documentation for the specific file format when needed.

Reading TSV Files

CSV doesn’t necessarily mean comma-separated.

Suppose your file uses tabs:

id    name    salary
1     Alice   90000
2     Bob     75000

You can specify the delimiter:

df = (
    spark.read
    .option("header", "true")
    .option("delimiter", "\t")
    .csv("employees.tsv")
)

The same reader can therefore handle many delimiter-separated formats.

Handling Null Values

Real-world datasets contain missing values.

For example:

id,name,salary
1,Alice,90000
2,Bob,
3,Charlie,95000

Spark’s CSV reader provides options for handling aspects of null representation.

For example:

df = (
    spark.read
    .option("header", "true")
    .option("nullValue", "")
    .csv("employees.csv")
)

The correct configuration depends on how the source system represents missing values.

Reading Text Files

Spark can also read text:

df = spark.read.text("logs/")

This typically creates a DataFrame with a column named:

value

For example:

df.show()

might produce:

+----------------------+
|value                 |
+----------------------+
|INFO Application start|
|INFO Reading data     |
|ERROR Connection lost |
+----------------------+

You can then transform the text using Spark functions.

For example:

from pyspark.sql.functions import col
errors = df.filter(
    col("value").contains("ERROR")
)

Reading Data Is Lazy

Here’s an important connection to Lazy Evaluation.

Consider:

df = spark.read.parquet("sales/")

It is tempting to think:

“Spark has now read the entire dataset.”

Not necessarily.

Spark uses lazy evaluation for DataFrame operations.

The DataFrame represents a computation and data source that Spark can plan for execution.

An action such as:

df.count()

or:

df.show()

causes Spark to execute the required work.

Conceptually:

spark.read.parquet()
        ↓
     DataFrame
        ↓
   Execution Plan
        ↓
      Action
        ↓
     Execution

This is one of the most important differences between Spark and traditional local data-processing libraries.

Predicate Pushdown

Columnar formats such as Parquet can work particularly well with Spark’s query optimization capabilities.

Consider:

df = spark.read.parquet("sales/")
result = df.filter(
    "country = 'US'"
)

Spark may be able to push applicable filters closer to the data source.

Conceptually:

Without pushdown
Read everything
      ↓
Filter

versus:

With applicable pushdown
Read only relevant data
      ↓
Filter

The actual behavior depends on the source, file format, predicate, and execution plan.

You can inspect the physical plan with:

result.explain("formatted")

Column Pruning

The same principle applies to columns.

Suppose your dataset contains:

customer_id
name
email
address
country
salary
department
timestamp

but you only need:

df.select(
    "customer_id",
    "salary"
)

Spark’s optimizer may be able to avoid reading unnecessary columns from a columnar source such as Parquet.

Conceptually:

Dataset
 ├── customer_id  ✓
 ├── name         ✗
 ├── email        ✗
 ├── address      ✗
 ├── country      ✗
 ├── salary       ✓
 ├── department   ✗
 └── timestamp    ✗

This is called column pruning.

Reading Partitioned Data

Data lakes often organize files into directory partitions.

For example:

sales/
├── year=2025/
│   ├── month=01/
│   └── month=02/
└── year=2026/
    ├── month=01/
    └── month=02/

You can read the root:

df = spark.read.parquet("sales/")

Spark can infer partition columns from the directory structure.

You may see:

year
month

as DataFrame columns.

Then:

df.filter("year = 2026")

can potentially benefit from partition pruning.

Conceptually:

sales/
├── year=2025/   ← skip
└── year=2026/   ← read

This can dramatically reduce the amount of data that needs to be scanned.

Partition Pruning vs Predicate Pushdown

These concepts are related but different.

Partition pruning

Avoids reading entire directory partitions that cannot satisfy the filter.

year=2025  → skip
year=2026  → read

Predicate pushdown

Pushes applicable predicates closer to the data source so fewer records need to be processed.

Read data
   ↓
Apply filter as early as possible

Both can reduce unnecessary data processing.

Reading Huge Datasets

Suppose you have:

5 TB of Parquet data

You should not think of:

df = spark.read.parquet("sales/")

as:

“Load 5 TB into the driver’s memory.”

That is not how Spark DataFrames work.

Spark creates a distributed representation of the data.

The actual records are processed by tasks running across executors when an action requires computation.

Conceptually:

5 TB
 │
 ├── Partition 1 → Executor 1
 ├── Partition 2 → Executor 2
 ├── Partition 3 → Executor 3
 ├── ...
 └── Partition N → Executor N

This distributed processing model is one of the fundamental reasons Spark can work with datasets much larger than a single machine’s memory.

Common Mistakes

1. Using inferSchema=True Everywhere

This is convenient:

spark.read.csv(
    path,
    header=True,
    inferSchema=True
)

but production pipelines often benefit from explicit schemas when the expected structure is known.

2. Using CSV for Large Analytical Workloads

CSV is convenient for data exchange.

But repeatedly processing large CSV datasets can be inefficient compared with columnar formats such as Parquet.

If you control the data platform, consider converting raw CSV data into an analytics-friendly format.

3. Reading More Columns Than Necessary

Instead of carrying dozens of columns through the pipeline:

df.select(
    "customer_id",
    "amount",
    "timestamp"
)

when those are the only fields required.

This can reduce the amount of data processed and, for supported columnar sources, potentially reduce I/O through column pruning.

4. Reading More Data Than Necessary

If the dataset is partitioned by date:

year=2026/month=01
year=2026/month=02
year=2026/month=03
...

don’t blindly scan every partition when only one month is required.

Use an appropriate filter so Spark can potentially prune irrelevant partitions.

5. Calling collect() After Reading a Large Dataset

This is dangerous:

data = df.collect()

because it brings the result back to the driver.

For a large dataset, this can cause driver memory problems.

Prefer distributed operations:

df.filter(...)
df.groupBy(...)
df.write...

and only collect genuinely small results.

A Production-Oriented CSV Example

Suppose an application receives daily customer files:

s3://company-data/customers/2026/09/07/

A robust ingestion pattern might be:

from pyspark.sql.types import (
    StructType,
    StructField,
    LongType,
    StringType,
    DoubleType
)
schema = StructType([
    StructField("customer_id", LongType(), False),
    StructField("name", StringType(), True),
    StructField("country", StringType(), True),
    StructField("balance", DoubleType(), True)
])
customers = (
    spark.read
    .option("header", "true")
    .schema(schema)
    .csv("s3://company-data/customers/2026/09/07/")
)

Then validate:

customers.printSchema()

and:

customers.show(10)

Then continue with transformations.

This is much more predictable than allowing the schema to change silently based on incoming data.

A Production-Oriented Parquet Example

For a data lake:

sales = spark.read.parquet(
    "s3://company-data/sales/"
)

Then filter early:

us_sales = sales.filter(
    "country = 'US'"
)

Select only required columns:

us_sales = us_sales.select(
    "customer_id",
    "amount",
    "order_date"
)

Then perform the expensive operation:

revenue = (
    us_sales
    .groupBy("customer_id")
    .sum("amount")
)

Conceptually:

Parquet
   ↓
Partition pruning
   ↓
Column pruning
   ↓
Filter
   ↓
Reduced dataset
   ↓
Aggregation

This connects data reading directly to the performance concepts covered earlier in the Spark series.

load() vs Format-Specific Methods

You will often see both:

spark.read.parquet(path)

and:

spark.read.format("parquet").load(path)

Both are valid.

For standard formats, the format-specific API is often simpler:

spark.read.parquet(path)

The generic API is useful when the data source or format is configured dynamically:

df = (
    spark.read
    .format(data_format)
    .options(**options)
    .load(path)
)

This becomes particularly useful in reusable ingestion frameworks.

The Complete Mental Model

The important concepts can be connected like this:

                 SparkSession
                      │
                      ↓
                  spark.read
                      │
          ┌───────────┼───────────┐
          ↓           ↓           ↓
         CSV         JSON       Parquet
          │           │           │
          └───────────┼───────────┘
                      ↓
                  DataFrame
                      │
                      ↓
               Logical Plan
                      │
                      ↓
             Query Optimization
                      │
             ┌────────┴────────┐
             ↓                 ↓
      Partition Pruning   Column Pruning
             │                 │
             └────────┬────────┘
                      ↓
                Physical Plan
                      │
                      ↓
                   Action
                      │
                      ↓
                  Execution
                      │
                ┌─────┴─────┐
                ↓           ↓
           Executors      Tasks

This is the bigger picture.

Reading data is not simply:

“Open a file.”

In Spark, you’re defining a distributed data source that becomes part of an execution plan.

Key Takeaways

Remember these principles:

1. spark.read is the primary DataFrame API for reading external data.
2. Spark supports formats such as CSV, JSON, Parquet, ORC, and text.
3. CSV often requires options such as header and may benefit from an explicit schema.
4. Parquet stores schema information and is designed for efficient analytical workloads.
5. format().load() provides a generic data-reading interface.
6. Spark can read individual files, directories, and multiple paths.
7. Spark can read data from distributed and cloud storage systems when properly configured.
8. Reading a DataFrame is part of Spark’s lazy execution model.
9. Partition pruning can avoid scanning irrelevant directory partitions.
10. Column pruning can reduce unnecessary column reads from supported columnar sources.
11. Explicit schemas provide better control and predictability in production pipelines.
12. Avoid collect() for large datasets because it brings data to the driver.
13. For large analytical workloads, columnar formats such as Parquet are generally preferable to CSV.
14. Always inspect the execution plan when performance matters.

The simplest mental model is:

spark.read
    ↓
Data source
    ↓
DataFrame
    ↓
Transformations
    ↓
Optimization
    ↓
Action
    ↓
Distributed execution

Interview Questions

1. How do you read a CSV file in PySpark?

df = spark.read.csv(
    "data.csv",
    header=True,
    inferSchema=True
)

2. How do you read a Parquet file?

df = spark.read.parquet("sales/")

3. What is the difference between inferSchema=True and an explicit schema?

inferSchema=True asks Spark to infer data types from the input. An explicit schema tells Spark exactly what types and fields are expected.

4. Why is Parquet generally preferred over CSV for analytics?

Parquet is columnar, stores schema metadata, and can support efficient column pruning and predicate pushdown.

5. Can Spark read multiple files at once?

Yes.

df = spark.read.parquet("sales/")

Spark can read files from a directory as a distributed dataset.

6. What is the difference between spark.read.parquet() and spark.read.format("parquet").load()?

They are two interfaces for reading the same format. The format/load pattern is more generic and useful when the source format is determined dynamically.

7. What is partition pruning?

Partition pruning allows Spark to avoid reading entire data partitions that cannot satisfy a filter, when the data layout and query allow it.

8. What is column pruning?

Column pruning allows Spark to avoid reading unnecessary columns from supported data sources, particularly columnar formats.

9. Does spark.read.parquet() immediately load all data into memory?

No. Spark DataFrames use lazy evaluation, and the required computation is executed when an action requires it.

10. Why shouldn’t you use collect() on a huge DataFrame?

Because collect() returns all rows to the driver, potentially exhausting driver memory.

Continue Learning Apache Spark

You now understand how data enters a Spark application.

The next concepts build directly on this:

→ Writing Data in PySpark — how to save DataFrames to CSV, JSON, Parquet, tables, and cloud storage
→ Partitioning in Spark — how Spark distributes data across partitions
→ Lazy Evaluation — why reading and transformations don’t immediately execute
→ DAG — how Spark represents the computation
→ Shuffle — what happens when data must move between partitions

Next Step: Writing Data in PySpark →

Reading data is only half of a data pipeline.

Once Spark has transformed the data, you need to decide:

Where should the result be stored, in what format, and with how many files?

That leads to the next tutorial:

Writing Data in PySpark: CSV, JSON, Parquet, Partitions, Modes, and Best Practices