Shuffle in Apache Spark: How Data Moves Across Partitions

by

If you’ve worked with Apache Spark for even a short time, you’ve probably encountered one of the most important—and potentially expensive—operations in distributed data processing:

Shuffle.

Whenever Spark needs to move data between partitions, a shuffle may occur.

This commonly happens during operations such as:

  • groupBy
  • join
  • distinct
  • orderBy
  • reduceByKey
  • repartition
  • Aggregations

Understanding shuffle is critical if you’re working with PySpark, Databricks, Delta Lake, or large-scale ETL pipelines.

A poorly optimized shuffle can result in:

  • Longer execution times
  • High network I/O
  • Increased disk I/O
  • Large amounts of temporary data
  • Executor memory pressure
  • Spill to disk
  • Straggler tasks
  • Higher Databricks cluster costs

In this tutorial, we’ll understand exactly what shuffle is, why Spark needs it, how it works internally, how to identify it, and how to optimize it in production workloads.

Enjoying this Data Engineering series? Follow me and subscribe by email to get the next PySpark, Spark, Databricks, and Data Engineering tutorials.

📚 Complete Apache Spark Tutorial Progress

APACHE SPARK LEARNING PATH

  1. ✓ What is Apache Spark?
  2. ✓ Why Spark is Faster than Hadoop
  3. ✓ Spark Architecture
  4. ✓ Driver vs Executor
  5. ✓ Cluster Managers
  6. → RDD vs DataFrame vs Dataset
  7. → SparkSession
  8. ✓ Reading Data
  9. ✓ Lazy Evaluation
  10. ✓ DAG
  11. ✓ Stages and Tasks
  12. ✓ Shuffle
  13. ✓ Narrow vs Wide Transformations
  14. → Writing Data
  15. → Data Transformations
  16. →Window Functions
  17. Aggregations
  18. UDFs
  19. Broadcast variables
  20. Accumulators

1. What Is Shuffle in Spark?

Shuffle is the process of redistributing data across partitions so that records with related keys can be processed together.

This movement usually happens across executors.

For example, imagine this dataset:

customer_id    amount
-----------    ------
101            500
102            200
101            300
103            700
102            100

Suppose we want:

df.groupBy("customer_id").sum("amount")

Spark needs all records belonging to the same customer to be available in the same partition.

Therefore:

Partition 1
Customer 101
Customer 102
Partition 2
Customer 101
Customer 103
        ↓
      SHUFFLE
        ↓
Partition 1
Customer 101
Customer 101
Partition 2
Customer 102
Customer 102
Customer 103

Spark has redistributed the records according to the grouping key.

That’s a shuffle.

2. Why Does Spark Need Shuffle?

Spark processes data in parallel.

Each partition can generally be processed independently.

For example:

Partition 1 → Executor 1
Partition 2 → Executor 2
Partition 3 → Executor 3
Partition 4 → Executor 4

But some operations require information from multiple partitions.

Consider:

df.groupBy("department").sum("salary")

Suppose the data initially looks like:

Partition 1
IT
HR
Partition 2
Finance
IT
Partition 3
HR
Finance

To calculate the total salary for each department, Spark needs:

All IT records
→ same partition
All HR records
→ same partition
All Finance records
→ same partition

Therefore, Spark has to redistribute the data.

Initial Partitions
       ↓
Redistribute by Key
       ↓
New Partitions
       ↓
Aggregation

That redistribution is the shuffle.

3. Shuffle and Wide Transformations

One of the most important Spark concepts is the difference between narrow and wide transformations.

A narrow transformation doesn’t require data to move between partitions.

A wide transformation generally requires data to be redistributed.

For example:

df.filter("salary > 100000")

doesn’t normally require records to move between partitions.

But:

df.groupBy("department").sum("salary")

requires data to be grouped by department.

Therefore, it involves a shuffle.

This is why wide transformations are closely associated with shuffle.

Related: [Narrow vs Wide Transformations in Spark →]

4. Simple Example of Shuffle

Let’s consider:

df.groupBy("customer_id").count()

Before shuffle:

Partition 1
101
102
101
Partition 2
103
102
101
Partition 3
102
103

After shuffle:

Partition 1
101
101
101
Partition 2
102
102
102
Partition 3
103
103

Now Spark can perform the aggregation:

101 → 3
102 → 3
103 → 2

The key point is:

Records with the same grouping key need to be brought together.

5. What Happens During a Shuffle?

At a high level, Spark performs several steps.

Input Data
    ↓
Map-side processing
    ↓
Partition data by key
    ↓
Write shuffle data
    ↓
Transfer data across executors
    ↓
Read shuffle data
    ↓
Reduce-side processing

A simplified view:

Executor 1 ─────┐
Executor 2 ─────┼──→ Shuffle ──→ Executors
Executor 3 ─────┤
Executor 4 ─────┘

This is why shuffle can be expensive.

Data may need to:

  • Be serialized
  • Be partitioned
  • Be written
  • Travel over the network
  • Be read by other tasks
  • Be deserialized
  • Potentially spill to disk

6. Shuffle Write and Shuffle Read

When analyzing Spark jobs, you’ll commonly see two important metrics:

Shuffle Write

The amount of data written by upstream tasks for the shuffle.

Shuffle Read

The amount of shuffle data read by downstream tasks.

Conceptually:

Map Tasks
    ↓
Shuffle Write
    ↓
Network / Shuffle Storage
    ↓
Shuffle Read
    ↓
Reduce Tasks

In the Spark UI, these metrics can help you identify stages that are moving significant amounts of data.

Related: [Stages and Tasks in Spark →]

7. Example: groupBy Causes Shuffle

Consider:

result = df.groupBy("department").count()

The operation requires all records for a department to be grouped together.

Therefore:

df
 ↓
Partitioning by department
 ↓
Shuffle
 ↓
Aggregation

The shuffle occurs because records may initially exist in different partitions.

8. Example: Join Can Cause Shuffle

Consider:

orders.join(
    customers,
    orders.customer_id == customers.customer_id
)

Suppose both datasets are large.

Spark may need to redistribute records based on:

customer_id

Conceptually:

Orders
   ↓
Partition by customer_id
   ↓
Shuffle
   ↓
Join
Customers
   ↓
Partition by customer_id
   ↓
Shuffle
   ↓
Join

This is one reason large joins can become expensive.

9. Shuffle in orderBy

Consider:

df.orderBy("salary")

Spark needs to establish a global ordering.

Data from different partitions must therefore be coordinated.

Conceptually:

Partition 1 ──┐
Partition 2 ──┼──→ Shuffle / Global Ordering
Partition 3 ──┤
Partition 4 ──┘

Global sorting can therefore involve significant data movement.

10. Shuffle in distinct()

Consider:

df.select("customer_id").distinct()

Spark needs to identify duplicate values across partitions.

For example:

Partition 1
101
102
Partition 2
101
103

Spark needs to determine that:

101

appears in multiple partitions.

Therefore, data may need to be redistributed based on the distinct key.

11. Repartition vs Coalesce

This is another important interview topic.

repartition()

df.repartition(10)

generally causes a shuffle because Spark needs to redistribute the data into the requested number of partitions.

For example:

Current:
4 partitions
      ↓
  Shuffle
      ↓
10 partitions

coalesce()

df.coalesce(2)

is primarily used to reduce the number of partitions and can avoid a full shuffle in the common case where partitions are being collapsed.

For example:

10 partitions
      ↓
  Coalesce
      ↓
2 partitions

This is why coalesce() is often used after filtering when you want fewer partitions.

Related: [DataFrame Transformations in PySpark →]

12. Why Shuffle Is Expensive

Shuffle isn’t inherently bad.

Spark needs shuffle for many legitimate operations.

The problem is unnecessary or excessive shuffle.

Shuffle can introduce several costs.

1. Network I/O

Data may need to move between executors.

2. Disk I/O

Shuffle data can be written to local disk.

3. Serialization

Objects may need to be serialized before transfer.

4. Memory pressure

Shuffle processing requires memory for intermediate data.

5. Spill

When memory isn’t sufficient, Spark can spill intermediate data to disk.

6. Longer execution time

Large shuffle stages can become bottlenecks.

13. Shuffle Spill

Imagine an executor processing a large partition.

The intermediate shuffle data is too large to fit in memory.

Spark may spill data to disk.

Conceptually:

Data
 ↓
Memory
 ↓
Memory Full
 ↓
Spill
 ↓
Disk

This can significantly increase I/O.

In Spark UI metrics, you may see memory and disk spill metrics that help identify such pressure.

14. Shuffle and Data Skew

Shuffle becomes particularly problematic when the data is skewed.

Suppose:

1 billion transactions

and one customer accounts for:

300 million transactions

while the remaining customers have much smaller volumes.

After partitioning by:

customer_id

one partition could become much larger than the others.

Partition 1 → 20 GB
Partition 2 → 18 GB
Partition 3 → 22 GB
Partition 4 → 500 GB  ← skewed

Most tasks finish quickly.

But one task takes much longer.

This creates a straggler.

Task 1 → 5 min
Task 2 → 6 min
Task 3 → 5 min
Task 4 → 45 min

The entire stage may have to wait for the slow task.

Related: [Data Skew in Spark →]

15. Shuffle and the Small Files Problem

Shuffle itself isn’t the same thing as the small-files problem, but the two can interact in poorly designed pipelines.

For example, excessive partitioning may eventually produce many output files:

Partition 1 → file 1
Partition 2 → file 2
Partition 3 → file 3
...
Partition 10,000 → file 10,000

This can create metadata and file-management overhead.

When writing Delta tables, you therefore need to consider:

  • Partition count
  • File sizes
  • Output partitioning
  • Compaction
  • Data layout

16. How to Identify Shuffle in Spark UI

The Spark UI is one of the most useful tools for troubleshooting shuffle.

A common workflow is:

Spark UI
   ↓
Stages
   ↓
Identify expensive stage
   ↓
Check Shuffle Read
   ↓
Check Shuffle Write
   ↓
Check task duration
   ↓
Look for skew

You can investigate:

  • Shuffle Read
  • Shuffle Write
  • Task duration
  • Spill
  • Input size
  • Output size
  • Failed tasks

The Spark monitoring documentation provides details on the metrics available for Spark applications. Apache Spark Monitoring Documentation

17. How to Reduce Shuffle

You shouldn’t try to eliminate every shuffle.

Instead:

Reduce unnecessary shuffle and make necessary shuffle efficient.

Here are some common strategies.

Strategy 1: Filter Early

Suppose your dataset contains:

1 billion rows

but only:

100 million rows

are relevant.

Instead of:

df.groupBy("customer_id").count()

use:

filtered = df.filter(
    "status = 'ACTIVE'"
)
result = filtered.groupBy(
    "customer_id"
).count()

Now Spark may shuffle significantly less data.

This principle is commonly called:

Filter early, reduce data early.

18. Strategy 2: Select Only Required Columns

Suppose you have:

100 columns

but your join requires only:

customer_id
name
amount

Don’t carry unnecessary columns through the pipeline.

For example:

orders_small = orders.select(
    "customer_id",
    "amount"
)

Reducing the amount of data carried through a shuffle can reduce network and storage overhead.

19. Strategy 3: Use Broadcast Joins

Suppose you have:

Transactions = 5 billion rows
Customers = 10 million rows

If the smaller side can safely fit within the relevant executor memory constraints, a broadcast join may avoid a large shuffle of the small table.

For example:

from pyspark.sql.functions import broadcast
result = transactions.join(
    broadcast(customers),
    "customer_id"
)

Conceptually:

Customers
    ↓
Broadcast
 ┌──┼──┬──┐
 ▼  ▼  ▼  ▼
E1 E2 E3 E4
Transactions
    ↓
Local Join

However, 10 million rows isn’t automatically “small.”

The actual size in bytes, available executor memory, serialization overhead, and workload characteristics matter.

Related: [Broadcast Joins in PySpark →]

20. Strategy 4: Handle Data Skew

If one key dominates the dataset, simply increasing the number of partitions may not solve the problem.

For example:

customer_id = 1001

may represent:

30% of all records

Possible strategies include:

  • Adaptive Query Execution
  • Skew join handling
  • Salting
  • Broadcast joins where appropriate
  • Better data modeling
  • Pre-aggregation

Related: [Data Skew in Spark →]
Related: [Salting in PySpark →]

21. Strategy 5: Avoid Unnecessary repartition()

This code:

df.repartition(1000)

forces redistribution of the data.

If the application doesn’t require 1,000 partitions, this can add unnecessary overhead.

Before calling repartition(), ask:

Why am I repartitioning this dataset?

A partitioning decision should be driven by:

  • Data volume
  • Number of cores
  • File sizes
  • Downstream operations
  • Key distribution
  • Cluster resources

—not simply an arbitrary number.

22. Strategy 6: Use Appropriate Partitioning

Partitioning can reduce future data movement when the same partitioning strategy aligns with downstream operations.

For example, if a workload repeatedly processes data by:

customer_id

you may consider whether partitioning or clustering strategies appropriate to your storage layer can improve access patterns.

However, partitioning is not a universal optimization.

Too many partitions can create:

  • Small files
  • Metadata overhead
  • More task scheduling overhead

Too few partitions can create:

  • Large tasks
  • Poor parallelism
  • Memory pressure

The goal is balanced parallelism.

23. Shuffle and Adaptive Query Execution

Modern Spark includes Adaptive Query Execution (AQE).

AQE allows Spark to optimize parts of the physical execution plan using runtime statistics.

It can help with scenarios such as:

  • Coalescing post-shuffle partitions
  • Handling certain skewed joins
  • Converting join strategies based on runtime information

Conceptually:

Initial Plan
     ↓
Execute
     ↓
Runtime Statistics
     ↓
Adaptive Optimization
     ↓
Improved Execution

This is especially relevant when working with modern Databricks and Spark environments.

Apache Spark SQL Performance Tuning — Adaptive Query Execution

24. Shuffle Partitions

Spark SQL uses a configurable number of shuffle partitions for many shuffle operations.

You may encounter:

spark.conf.get(
    "spark.sql.shuffle.partitions"
)

The configuration controls the number of partitions used for certain shuffle operations.

For example:

spark.conf.set(
    "spark.sql.shuffle.partitions",
    400
)

But don’t blindly increase the number.

Suppose you have:

1 GB of data

and configure:

10,000 partitions

You may create many tiny tasks.

Conversely, too few partitions can create large tasks.

Therefore, partition count should be aligned with:

  • Dataset size
  • Cluster parallelism
  • Workload characteristics
  • Data distribution

Modern Spark environments can also use AQE to dynamically coalesce post-shuffle partitions.

25. Shuffle Optimization Example

Suppose we have:

df = spark.read.parquet("/data/transactions")

and the table contains:

2 billion records

We need active transactions only.

Poor approach

result = df.groupBy(
    "customer_id"
).sum("amount")
result = result.filter(
    "sum(amount) > 10000"
)

The aggregation may process far more records than necessary.

Better approach

filtered = df.filter(
    "status = 'ACTIVE'"
)
result = filtered.groupBy(
    "customer_id"
).sum("amount")
result = result.filter(
    "sum(amount) > 10000"
)

The important principle is:

Reduce data
    ↓
Before expensive operations
    ↓
Less data to shuffle

26. How Shuffle Appears in an Interview

A common interview question is:

Why is shuffle expensive in Spark?

A strong answer would be:

Shuffle is expensive because Spark may need to redistribute data across executors. This introduces network I/O, serialization and deserialization, intermediate storage, memory pressure, and potentially disk spill. Large shuffles can also expose data skew, where one partition becomes much larger than others and creates a slow task.

That’s much stronger than simply saying:

“Shuffle moves data between partitions.”

27. Common Spark Shuffle Interview Questions

Q1. What is shuffle?

Shuffle is the redistribution of data across partitions so that records with related keys can be processed together.

Q2. Which operations can cause shuffle?

Common examples include:

  • groupBy
  • join
  • distinct
  • orderBy
  • repartition
  • Many aggregations

Q3. Why is shuffle expensive?

Because it can involve network transfer, serialization, disk I/O, memory pressure, and spill.

Q4. What is shuffle read?

Data read by downstream tasks from the shuffle output.

Q5. What is shuffle write?

Data written by upstream tasks for consumption by downstream shuffle tasks.

Q6. Does every transformation cause shuffle?

No.

Narrow transformations generally don’t require redistribution across partitions, while wide transformations generally do.

Q7. Does repartition() cause shuffle?

Yes, repartitioning generally requires redistribution of data.

Q8. Does coalesce() cause shuffle?

Reducing partitions with coalesce() can avoid a full shuffle in the common case.

Q9. How can you reduce shuffle?

Common approaches include:

  • Filtering early
  • Selecting only required columns
  • Avoiding unnecessary repartitioning
  • Using broadcast joins where appropriate
  • Handling skew
  • Using AQE
  • Choosing appropriate partitioning

Q10. How do you troubleshoot a large shuffle?

Use Spark UI and inspect:

  • Shuffle Read
  • Shuffle Write
  • Spill
  • Task duration
  • Partition sizes
  • Failed tasks
  • Data skew

28. Shuffle: The Mental Model

Remember this simple diagram:

              INPUT
                │
        ┌───────┼───────┐
        ▼       ▼       ▼
       P1      P2      P3
        │       │       │
        └───────┼───────┘
                │
             SHUFFLE
                │
        Redistribute by Key
                │
        ┌───────┼───────┐
        ▼       ▼       ▼
       P1      P2      P3
        │       │       │
        └───────┼───────┘
                ▼
          Aggregation
             / Join
             / Sort

The most important idea is:

Shuffle is the price Spark pays when data needs to move across partition boundaries.

Good Spark engineering isn’t about avoiding every shuffle.

It’s about understanding when shuffle is necessary and minimizing unnecessary data movement.

29. Key Takeaways

Before moving forward, remember:

  • Shuffle redistributes data across partitions.
  • It commonly occurs during joins, aggregations, grouping, sorting, and repartitioning.
  • Shuffle can involve network I/O, serialization, disk I/O, and memory pressure.
  • Large shuffles can cause spill.
  • Data skew can make one shuffle partition dramatically larger than others.
  • repartition() generally causes a shuffle.
  • coalesce() can reduce partitions without a full shuffle in the common case.
  • Filtering and selecting early can reduce shuffle volume.
  • Broadcast joins can avoid certain large shuffles.
  • AQE can optimize shuffle-related execution at runtime.
  • Spark UI is essential for diagnosing expensive shuffle stages.

Continue Learning PySpark

You’ve now learned one of the most important concepts behind Spark performance: Shuffle.

The next step is to understand Data Skew, because a shuffle can become particularly expensive when data isn’t evenly distributed across partitions.

Recommended next steps:

1. Data Skew

Understand why a single key can dominate a partition and create long-running Spark tasks.

2. Salting

Learn how to distribute heavily skewed keys across multiple partitions.

3. Broadcast Joins

Learn how broadcasting a small dataset can avoid an expensive shuffle.

4. Partitioning

Understand how to choose and manage partitions for large-scale workloads.

[View Complete Apache Spark Learning Path →]

You Might Need This Next

🔥 Seeing one Spark task take dramatically longer than the others?

That is often a sign of data skew.

Learn next:

→ What is Data Skew?
→ How to detect skew in Spark UI
→ Why one partition becomes huge
→ AQE skew join optimization
→ Salting
→ Broadcast Join

[Read Data Skew in Apache Spark →]

Next Step

You’ve learned why Spark moves data between partitions and why shuffle can become expensive.

But what happens when one customer, product, or transaction key represents a huge percentage of the data?

In the next tutorial, we’ll explore:

  • Data skew
  • Skewed partitions
  • Straggler tasks
  • Detecting skew
  • Spark UI analysis
  • AQE skew join optimization
  • Salting
  • Broadcast strategies
  • Real-world PySpark examples

Next: Understand Data Skew in Apache Spark →

Further Reading

Related Tutorials

  1. ✓ What is Apache Spark?
  2. ✓ Why Spark is Faster than Hadoop
  3. ✓ Spark Architecture
  4. ✓ Driver vs Executor
  5. ✓ Cluster Managers
  6. → RDD vs DataFrame vs Dataset
  7. → SparkSession
  8. ✓ Reading Data
  9. ✓ Lazy Evaluation
  10. ✓ DAG
  11. ✓ Stages and Tasks
  12. ✓ Shuffle
  13. ✓ Narrow vs Wide Transformations
  14. → Writing Data
  15. → Data Transformations
  16. →Window Functions
  17. Aggregations
  18. UDFs
  19. Broadcast variables
  20. Accumulators

1 thought on “Shuffle in Apache Spark: How Data Moves Across Partitions”

  1. Pingback: Partitioning in Apache Spark: How to Control Parallelism and Improve Performance - Geeky Codes

Leave a Reply

Discover more from Geeky Codes

Subscribe now to keep reading and get access to the full archive.

Continue reading