Why groupBy(), joins, distinct(), orderBy(), and repartition() can quietly turn a simple Spark job into an expensive workload

There is a Spark operation that can make a perfectly reasonable-looking piece of code suddenly become expensive at scale.

You may not notice it in development.

You may not notice it when testing with a few million rows.

But when the same code runs against hundreds of gigabytes or terabytes of data, your Databricks job suddenly spends most of its time moving data around.

That operation is shuffle.

And one of the most important things a Data Engineer can learn about Spark performance is this:

The expensive part of a distributed computation is often not the calculation itself. It is moving the data required to perform the calculation.

Consider this innocent-looking code:

df.groupBy("customer_id").sum("amount")

The business logic is simple:

“Give me the total amount for every customer.”

But Spark cannot simply calculate each customer’s total independently on whichever executor happens to hold the data.

Rows belonging to the same customer need to meet somewhere.

That often means:

Executor 1 ───┐
Executor 2 ───┤
Executor 3 ───┼──→ SHUFFLE ──→ Reorganized Partitions
Executor 4 ───┤
Executor 5 ───┘

And that is where things can get expensive.

This is Part 2 of the GeekyCodes series:

On Optimizing Your Code to Save Your Bill on Databricks

In Part 1, we established the basic idea:

Better code can reduce unnecessary computation and, consequently, improve cost efficiency.

In this article, we’ll go one level deeper:

Why does Spark shuffle data, which operations trigger it, how do you recognize it, and what can you actually do about it?

A quick note before we start

If you enjoy practical content on PySpark, Databricks, Data Engineering, Machine Learning and GenAI, consider following me and subscribing to GeekyCodes for the rest of this series.

1. First, What Exactly Is a Shuffle?

Let’s start with a simple example.

Suppose your data is distributed across four executors.

Executor 1
Customer A
Customer B
Customer C
Executor 2
Customer A
Customer D
Executor 3
Customer B
Customer C
Customer E
Executor 4
Customer D
Customer E

Now you execute:

df.groupBy("customer_id").sum("amount")

Spark needs all records for Customer A to be processed together.

But currently:

Customer A → Executor 1
Customer A → Executor 2

The records are distributed across different executors.

Spark therefore needs to redistribute the data:

Before Shuffle
Executor 1 ── Customer A
Executor 2 ── Customer A
Executor 3 ── Customer B
Executor 4 ── Customer B
             ↓
          SHUFFLE
             ↓
After Shuffle
Partition 1 → Customer A records
Partition 2 → Customer B records

That redistribution is the shuffle.

In Spark’s physical execution plan, this kind of redistribution commonly appears as an Exchange operator. Spark’s documentation also exposes shuffle-related configuration such as spark.sql.shuffle.partitions, which controls the number of partitions used for joins and aggregations.

2. Why Is Shuffle Expensive?

Shuffle isn’t inherently bad.

It is a fundamental part of distributed computing.

The problem is that moving data between executors can be considerably more expensive than performing a local computation.

A simplified view looks like this:

Local computation
Executor
   │
   ├── Read
   ├── Process
   └── Write
          ↓
       Relatively simple

Whereas a shuffle can involve:

Executor
   ↓
Partition data
   ↓
Serialize
   ↓
Write shuffle data
   ↓
Network transfer
   ↓
Read shuffle data
   ↓
Deserialize
   ↓
Process

Now multiply that by:

Millions / billions of rows
×
Hundreds / thousands of partitions
×
Multiple stages

The amount of work can grow quickly.

And that’s why shuffle is one of the first things I investigate when a large Spark job unexpectedly becomes expensive.

3. The Spark Transformation That Should Make You Pause

Consider:

df.groupBy("customer_id")

This is not automatically bad.

But it should trigger a question:

Does Spark need to redistribute this data?

Often, yes.

For example:

result = (
    df
    .groupBy("customer_id")
    .agg({"amount": "sum"})
)

The execution may conceptually look like:

Scan
  ↓
Partial Aggregate
  ↓
Exchange
  ↓
Final Aggregate

The Exchange is the interesting part.

The first aggregation can reduce data locally.

But Spark still needs records associated with the same grouping key to reach the appropriate partition for the final aggregation.

4. Narrow vs Wide Transformations

This distinction is fundamental to understanding Spark.

Spark transformations can broadly be thought of as narrow or wide dependencies.

Narrow Transformation

A narrow transformation can process data without requiring records to be redistributed across the cluster.

For example:

df.filter("amount > 1000")

Conceptually:

Partition 1 → Filter → Partition 1
Partition 2 → Filter → Partition 2
Partition 3 → Filter → Partition 3
Partition 4 → Filter → Partition 4

There is no requirement for Partition 1 to send data to Partition 3.

Each partition can largely work independently.

Wide Transformation

A wide transformation requires data to be redistributed so that the next stage has the required partitioning.

For example:

df.groupBy("customer_id").sum("amount")

Conceptually:

Partition 1 ──┐
Partition 2 ──┤
Partition 3 ──┼──→ Shuffle → New Partitions
Partition 4 ──┘

This is why wide transformations deserve attention.

Not because they are “bad.”

But because they can introduce expensive data movement.

5. Common Operations That Can Trigger Shuffle

Here are some of the most important ones to recognize.

This does not mean:

“Never use these operations.”

That would be terrible advice.

These operations exist because business requirements require them.

The goal is to understand their cost and avoid unnecessary or poorly designed shuffle.

6. groupBy() — The Classic Example

Suppose we have:

orders = spark.read.table("orders")
customer_sales = (
    orders
    .groupBy("customer_id")
    .sum("amount")
)

At a conceptual level:

Orders
  ↓
Read
  ↓
Partial aggregation
  ↓
Shuffle by customer_id
  ↓
Final aggregation

Why can’t Spark simply aggregate each partition independently?

Because the same customer can appear in multiple partitions.

For example:

Partition 1
Customer 101 → ₹100
Customer 102 → ₹500
Partition 2
Customer 101 → ₹200
Customer 103 → ₹800

The final answer for customer 101 is:

₹100 + ₹200 = ₹300

Spark needs those records to participate in the same logical aggregation.

That’s where redistribution comes in.

7. The First Optimization: Filter Before the Shuffle

Consider:

result = (
    orders
    .groupBy("customer_id")
    .sum("amount")
    .filter("sum(amount) > 10000")
)

The filter here applies to the aggregated result.

But imagine the business requirement was actually:

“Calculate sales for completed orders only.”

Then doing this:

result = (
    orders
    .filter("status = 'COMPLETED'")
    .groupBy("customer_id")
    .sum("amount")
)

can reduce the amount of data entering the aggregation.

Conceptually:

Before

1 Billion Rows
      ↓
GroupBy
      ↓
Shuffle
      ↓
Aggregation
      ↓
Filter

Better

1 Billion Rows
      ↓
Filter
      ↓
200 Million Rows
      ↓
GroupBy
      ↓
Shuffle
      ↓
Aggregation

The exact reduction depends on the data and query plan, but the principle is powerful:

Reduce the data before expensive operations whenever the semantics allow it.

8. Projection Can Help Too

Suppose your DataFrame has 100 columns.

Your aggregation only needs:

customer_id
amount

Instead of carrying everything:

orders.groupBy("customer_id").sum("amount")

you can make the required columns explicit:

orders = orders.select(
    "customer_id",
    "amount"
)
result = (
    orders
    .groupBy("customer_id")
    .sum("amount")
)

Modern Spark optimizers can perform column pruning themselves, so don’t assume manually writing .select() always changes the physical plan.

The important lesson is:

Understand what the optimizer already does before adding manual transformations.

Use explain() to verify.

9. Joins: The Other Major Shuffle Generator

Consider:

orders.join(
    customers,
    orders.customer_id == customers.customer_id
)

If the two datasets are distributed differently, Spark may need to redistribute them so matching keys can meet.

Conceptually:

Orders
   ↓
Shuffle by customer_id
   ↓
          Join
   ↑
Shuffle by customer_id
   ↑
Customers

For large datasets, that can be expensive.

Databricks specifically calls out join performance as an important optimization area and recommends considering join order, statistics, broadcast opportunities, and avoiding unnecessarily expensive join patterns such as cross joins.

10. Can We Avoid a Join Shuffle?

Sometimes.

Suppose:

orders = 500 GB
customers = 20 MB

Broadcasting may be appropriate:

from pyspark.sql.functions import broadcast
result = orders.join(
    broadcast(customers),
    "customer_id"
)

Conceptually:

Small Customers
       ↓
   Broadcast
       ↓
Executors
Large Orders
       ↓
Local Join

Instead of shuffling both sides.

But here’s an important warning:

Don’t turn broadcast joins into another cargo-cult optimization.

Broadcasting a genuinely small table can be useful.

Broadcasting a table that is too large can create memory pressure and potentially make the workload worse.

Spark has automatic broadcast mechanisms, and Databricks’ Adaptive Query Execution can dynamically switch certain sort-merge joins to broadcast hash joins based on runtime information.

So before adding a broadcast hint, understand:

• table size
• join type
• available memory
• statistics
• physical plan
• whether AQE is already making a good decision

11. distinct() Looks Innocent. It Isn’t Always Cheap.

Consider:

unique_customers = df.select("customer_id").distinct()

The business requirement is simple:

Give me unique customer IDs.

But Spark needs to determine which values are duplicates across the distributed dataset.

Conceptually:

Partition 1 ──┐
Partition 2 ──┤
Partition 3 ──┼──→ Shuffle → Deduplicate
Partition 4 ──┘

So:

distinct()

should not automatically be considered “bad.”

But at very large scale, you should ask:

Do I really need a global distinct?

Sometimes the requirement can be redesigned.

Sometimes duplicates can be prevented earlier.

Sometimes filtering can significantly reduce the input first.

Sometimes the operation is unavoidable.

Again, optimization starts with the requirement.

12. orderBy() Can Be Even More Interesting

Now consider:

df.orderBy("transaction_date")

A global sort is fundamentally different from sorting each partition independently.

You are asking Spark for a globally ordered result.

That can require substantial coordination and redistribution.

Conceptually:

Partition 1 ──┐
Partition 2 ──┤
Partition 3 ──┼──→ Global Ordering
Partition 4 ──┘

So before writing:

df.orderBy(...)

ask:

Do I actually need the entire dataset globally sorted?

Or do I only need:

df.orderBy(...).limit(100)

Or perhaps ordering is only required within a downstream operation?

The answer can change the optimal strategy.

13. The repartition() Trap

One of the most common code smells is:

df = df.repartition(100)

followed by:

df = df.repartition(200)

and later:

df = df.repartition("customer_id")

Each repartitioning decision can introduce another redistribution.

You can end up with:

Read
 ↓
Shuffle
 ↓
repartition(100)
 ↓
Shuffle
 ↓
repartition(200)
 ↓
Shuffle
 ↓
repartition(customer_id)
 ↓
Shuffle
 ↓
Join

This is not automatically wrong.

Sometimes explicit repartitioning is exactly what the workload needs.

But if you’re adding repartition() because:

“Someone told me Spark needs more partitions.”

Stop.

Measure first.

14. A Simple Code Smell

If you see code like this:

df = (
    df
    .repartition(1000)
    .filter(...)
    .repartition(500)
    .join(...)
    .repartition(200)
    .groupBy(...)
)

don’t immediately conclude that it is wrong.

Instead ask:

Why is each repartition here?

repartition(1000)
        ↓
Why?
repartition(500)
        ↓
Why?
repartition(200)
        ↓
Why?

If the answer is:

“To make Spark faster.”

That’s not enough.

The better question is:

What execution problem is this repartition solving?

15. Shuffle Partitions Matter

Spark SQL uses a configurable number of partitions for shuffle operations.

The configuration:

spark.sql.shuffle.partitions

controls the number of partitions used for joins and aggregations. Apache Spark documents a default value of 200 for this setting, while Databricks also supports automatic shuffle partition sizing through AQE-related capabilities.

You can inspect the current setting:

spark.conf.get("spark.sql.shuffle.partitions")

And traditionally you could configure it explicitly:

spark.conf.set(
    "spark.sql.shuffle.partitions",
    400
)

But here’s the important part:

More partitions doesn’t automatically mean better performance.

16. Too Few Shuffle Partitions

Suppose:

1 TB data
↓
10 shuffle partitions

Each partition may become enormous.

You could end up with:

Partition 1 → 100 GB
Partition 2 → 100 GB
...

Large partitions can create:

• long-running tasks
• memory pressure
• spill
• poor parallelism

17. Too Many Shuffle Partitions

Now go in the other direction:

1 TB data
↓
100,000 shuffle partitions

You might create a huge number of tiny tasks.

That can increase:

• task scheduling overhead
• task setup overhead
• file/I/O overhead
• coordination overhead

So the goal isn’t:

Maximum number of partitions.

The goal is:

Appropriate parallelism for the workload.

18. This Is Where AQE Becomes Important

Modern Spark and Databricks provide Adaptive Query Execution (AQE).

AQE can use runtime statistics to re-optimize a query while it executes.

Databricks documents several AQE capabilities, including:

• dynamically changing certain sort-merge joins to broadcast hash joins
• dynamically coalescing shuffle partitions
• dynamically handling skewed joins
• propagating empty relations

This is important because the optimizer can learn things at runtime that were difficult to know accurately before execution.

For example:

Initial plan
     ↓
Execute shuffle
     ↓
Observe actual partition sizes
     ↓
AQE
     ↓
Adjust execution strategy

That’s one reason you shouldn’t blindly hard-code every Spark optimization.

19. AQE Can Coalesce Small Shuffle Partitions

Imagine your configuration produces:

Partition 1 → 5 MB
Partition 2 → 7 MB
Partition 3 → 3 MB
Partition 4 → 4 MB
...

You may have too many small tasks.

AQE can dynamically coalesce shuffle partitions into larger, more efficient partitions.

Databricks documents this as one of AQE’s core capabilities.

Conceptually:

Before
5 MB
7 MB
3 MB
4 MB
6 MB
5 MB
        ↓ AQE
25 MB
20 MB
...

The exact behavior depends on the workload and runtime configuration.

20. AQE Can Also Help With Skew

Consider a shuffle where most partitions look like:

100 MB
120 MB
90 MB
110 MB
105 MB

But one is:

4 GB

Now you have a skewed partition.

Without mitigation:

████████████████████  → Finish quickly
████████████████████  → Finish quickly
████████████████████  → Finish quickly
████████████████████████████████████████████████ → Still running

Databricks AQE can detect and handle certain skewed shuffle joins by splitting skewed partitions, subject to its configured conditions and supported join types.

This is one reason the first step should be:

Look at the actual execution behavior before manually rewriting the code.

21. How Do You Know Shuffle Is Your Problem?

Don’t guess.

Use the tools available to you.

Start with the execution plan

df.explain("formatted")

Look for:

Exchange

For example:

== Physical Plan ==
...
Exchange hashpartitioning(customer_id, 200)
...

That is a strong signal that Spark is redistributing data.

But remember:

An Exchange is not automatically a performance bug.

A join or aggregation may legitimately require one.

The question is:

Is the shuffle expensive, unnecessary, excessive, or poorly balanced?

22. Then Go to the Spark UI

Suppose your job takes 40 minutes.

You open the Spark UI.

You discover:

Stage 0 → 2 min
Stage 1 → 3 min
Stage 2 → 4 min
Stage 3 → 29 min
Stage 4 → 2 min

Stage 3 is clearly worth investigating.

Then you see:

Shuffle Read: 700 GB
Shuffle Write: 650 GB

Now you have evidence.

You aren’t saying:

“I think shuffle is expensive.”

You’re saying:

“This stage is spending substantial time processing a large shuffle.”

That’s a much better starting point.

23. Look for Uneven Task Durations

Suppose Stage 3 has 500 tasks.

You inspect the task durations:

Task 1    → 12 sec
Task 2    → 15 sec
Task 3    → 11 sec
...
Task 498  → 13 sec
Task 499  → 14 sec
Task 500  → 18 min

That’s a red flag.

It may indicate:

• skew
• an unusually large partition
• spill
• uneven data distribution
• resource contention

The slowest task can become the bottleneck for the entire stage.

24. The Cost of Shuffle Isn’t Just Network Traffic

It’s tempting to think:

“Shuffle = network transfer.”

But there’s more happening.

A shuffle can involve:

CPU
 ↓
Serialization
 ↓
Memory
 ↓
Network
 ↓
Disk
 ↓
Deserialization
 ↓
Sorting / aggregation

If shuffle data doesn’t fit comfortably in memory, Spark may spill intermediate data to disk.

That can make the workload considerably more expensive.

So when investigating shuffle, don’t look at only:

Shuffle Read
Shuffle Write

Also investigate:

Spill Memory
Spill Disk
Task duration
Input size
Partition distribution

25. The Optimization Ladder

When you discover an expensive shuffle, don’t immediately start changing Spark configurations.

Use this sequence.

Step 1 — Ask whether the operation is necessary

Do you really need:

distinct()

Do you really need:

orderBy()

Do you really need:

repartition()

Do you really need this join?

If the answer is no, removing the operation is the strongest optimization.

Step 2 — Reduce the data before the shuffle

Apply legitimate filters early.

Select required columns.

Avoid carrying unnecessary data.

Conceptually:

Large Dataset
     ↓
Filter
     ↓
Projection
     ↓
Shuffle

is generally preferable to:

Large Dataset
     ↓
Shuffle
     ↓
Filter
     ↓
Projection

when the query semantics and optimizer allow that reduction.

Step 3 — Choose a better join strategy

If one side is genuinely small, consider whether broadcast is appropriate.

If the tables are large, investigate:

• join keys
• statistics
• skew
• join order
• partitioning
• AQE

Databricks recommends maintaining fresh statistics because the optimizer uses them to select join strategies and improve query planning.

Step 4 — Investigate skew

If one partition is dramatically larger than the others, don’t simply increase the number of partitions.

You may need to address the distribution of the key itself.

Step 5 — Tune partitioning

Only after understanding the workload should you consider:

spark.sql.shuffle.partitions

or explicit repartitioning.

Step 6 — Let AQE help

Modern Databricks environments can use AQE to adapt to runtime conditions.

Don’t disable useful adaptive behavior without a specific reason.

Databricks recommends keeping AQE enabled and documents its role in join selection, partition coalescing, and skew handling.

26. A Realistic Example

Let’s imagine an orders pipeline.

orders = spark.read.table("orders")
result = (
    orders
    .groupBy("customer_id")
    .sum("amount")
    .orderBy("sum(amount)", ascending=False)
)

Looks simple.

But now think about what you’re asking Spark to do.

Read orders
    ↓
Group by customer
    ↓
Shuffle
    ↓
Aggregate
    ↓
Global Sort
    ↓
Write

You’ve potentially created two expensive operations:

GROUP BY
   ↓
Shuffle
ORDER BY
   ↓
Global ordering / additional coordination

Now ask:

Do I really need every customer globally sorted?

Suppose the actual requirement is:

“Give me the top 100 customers.”

You could express the requirement as:

result = (
    orders
    .groupBy("customer_id")
    .sum("amount")
    .orderBy("sum(amount)", ascending=False)
    .limit(100)
)

Spark can potentially optimize such a query differently than an unconstrained global result, but the key lesson is bigger than this example:

Always distinguish the business requirement from the most expensive way of implementing it.

27. A More Important Question: Can We Avoid the Shuffle Entirely?

This is the best optimization question.

Not:

“How can I make this shuffle faster?”

But:

“Can I avoid this shuffle?”

Sometimes the answer is no.

For example:

df.groupBy("customer_id").sum("amount")

If the data is not already appropriately organized, some redistribution may be necessary.

But sometimes the answer can be yes.

For example:

• eliminate unnecessary distinct()
• remove unnecessary repartition()
• filter before expensive operations
• use an appropriate broadcast join
• exploit compatible existing data organization
• avoid unnecessary global sorts

Apache Spark also documents techniques that can eliminate shuffle in certain compatible storage-partitioned join scenarios.

That is a much more powerful optimization than simply throwing more compute at the problem.

28. Don’t Over-Partition Just to “Increase Parallelism”

Here’s another common misconception:

“If 200 partitions are good, 2,000 partitions must be better.”

No.

Imagine:

100 GB
↓
2,000 partitions

You might have approximately:

50 MB / partition

That’s not necessarily problematic.

But if you have:

1 GB
↓
100,000 partitions

you are potentially creating enormous task overhead.

More tasks aren’t automatically more parallelism in a useful sense.

Eventually:

More partitions
       ↓
More tasks
       ↓
More scheduling overhead
       ↓
Less efficient execution

The correct number depends on:

• data size
• cluster resources
• workload
• partition sizes
• downstream operations
• runtime behavior

And AQE can dynamically coalesce partitions in supported workloads.

29. What About coalesce()?

Suppose you want fewer partitions.

You might see:

df.coalesce(10)

Unlike repartition(), coalesce() can reduce the number of partitions without performing a full shuffle in the common case.

That can make it useful when reducing partitions before an output operation.

But again:

Don’t use coalesce() simply because it avoids shuffle.

Reducing partitions too aggressively can reduce parallelism and create large tasks.

The question is always:

What partitioning does the next operation actually need?

30. A Simple Before-and-After Investigation

Imagine we have:

Before

Runtime:           42 minutes
Shuffle Read:      800 GB
Shuffle Write:     720 GB
Spill:             180 GB
Longest Task:      17 minutes

We inspect the plan and discover:

Large Join
    ↓
Shuffle
    ↓
GroupBy
    ↓
Shuffle
    ↓
Global Sort

We then:

1. filter unnecessary records earlier
2. select only required columns
3. use a broadcast join where the dimension table is genuinely small
4. remove an unnecessary global sort
5. allow AQE to optimize the remaining shuffle

After benchmarking:

Runtime:           19 minutes
Shuffle Read:      220 GB
Shuffle Write:     190 GB
Spill:              20 GB
Longest Task:       2 minutes

The important result isn’t the exact numbers.

The important result is that we reduced the amount of work Spark needed to perform.

And that’s the philosophy behind this entire series.

31. But Don’t Manufacture Benchmarks

One important rule for real engineering—and technical writing:

Don’t claim a specific percentage of cost savings unless you actually measured it.

You will often see articles saying:

“This optimization reduces costs by 70%.”

That might be true for one workload and completely wrong for another.

Your workload could have:

• different data volume
• different skew
• different cluster size
• different runtime
• different caching
• different storage layout
• different pricing
• different execution plan

Instead, benchmark your own workload.

A useful benchmark table is:

That gives your optimization an engineering foundation.

32. A Practical Shuffle Debugging Checklist

Whenever a Spark job is unexpectedly expensive, ask:

Execution Plan

□ Do I see Exchange?
□ Where does it occur?
□ How many shuffle stages exist?
□ Which operators precede/follow the Exchange?

Data

□ How much data is being shuffled?
□ Can I filter earlier?
□ Can I select fewer columns?
□ Is the data skewed?

Joins

□ Which join strategy is being used?
□ Is one side genuinely small?
□ Are statistics fresh?
□ Is the join key skewed?

Aggregations

□ Do I really need this aggregation?
□ Can the input be reduced first?
□ Is the grouping key highly skewed?

Partitions

□ Are partitions too large?
□ Are there too many tiny partitions?
□ Is shuffle partitioning appropriate?
□ Can AQE help?

Code

□ Am I using unnecessary repartition()?
□ Am I doing unnecessary distinct()?
□ Am I globally sorting unnecessarily?
□ Am I joining more data than necessary?

33. The Biggest Lesson

When you see:

Exchange

in a Spark physical plan, don’t panic.

Don’t immediately remove it.

Don’t immediately increase the number of executors.

Don’t immediately change:

spark.sql.shuffle.partitions

Instead ask:

Why is Spark redistributing this data?

Then:

How much data is being moved?

Then:

Is the redistribution necessary?

And finally:

Can I reduce or eliminate it?

That sequence of questions is far more valuable than memorizing a list of Spark configurations.

34. The Cost Optimization Connection

Remember the theme of this series:

Optimize your code to save your bill.

Shuffle provides a perfect example.

Poor Design
     ↓
More Data
     ↓
More Shuffle
     ↓
More Network / CPU / Disk Work
     ↓
Longer Runtime
     ↓
Higher Resource Consumption

Versus:

Better Design
     ↓
Less Data
     ↓
Less Shuffle
     ↓
Less Unnecessary Work
     ↓
More Efficient Execution
     ↓
Potentially Lower Cost

Again, the final step must be measured, not assumed.

35. The Golden Rule for Shuffle

Here’s the rule I want you to take away from this article:

Don’t ask how to make a shuffle faster until you’ve asked whether you can avoid the shuffle.

That’s the difference between configuration tuning and engineering.

Configuration tuning says:

“Let’s increase the number of executors.”

Engineering says:

“Why are we moving 800 GB of data between executors in the first place?”

That second question can lead to a much bigger improvement.

36. What’s Coming Next?

In this article, we saw why shuffle is expensive and why wide transformations deserve attention.

But there is one particular operation that deserves its own article:

repartition()

It looks innocent.

It’s often recommended as a solution to partitioning problems.

And sometimes it is exactly the right solution.

But used blindly, it can introduce a shuffle that you didn’t need.

That’s why the next article in this series will focus on:

Why repartition() Can Quietly Increase Your Databricks Bill

We’ll break down:

repartition()

vs.

coalesce()

and answer:

• What actually happens during repartition()?
• When does it trigger a shuffle?
• How many partitions should you use?
• Why can too many partitions be harmful?
• How does AQE change the equation?
• How do you decide whether repartitioning is actually helping?
• What does the physical plan look like before and after?

Final Takeaway

Spark is a distributed computing engine.

And distributed computing has a fundamental cost:

Moving data.

A filter can often operate where the data already lives.

A shuffle cannot.

That’s why operations such as:

groupBy()
join()
distinct()
orderBy()
repartition()

deserve careful attention when you’re working with large datasets.

But don’t fall into the opposite trap and treat every wide transformation as a mistake.

Sometimes the business requirement genuinely requires the data to be redistributed.

The goal isn’t:

“Eliminate every shuffle.”

The goal is:

“Eliminate unnecessary shuffle and make the necessary shuffle as efficient as possible.”

So the next time your Databricks job is taking 45 minutes and your bill is climbing, don’t just look at the cluster.

Open the execution plan.

Find the Exchange.

Open the Spark UI.

Look at the shuffle.

Look at the partition distribution.

And ask the most important question:

“Why is Spark moving all this data?”

Because sometimes the biggest cost optimization isn’t buying less compute.

It’s making Spark move less data.

Further Reading

• Apache Spark — Performance Tuning — Official documentation covering partition tuning, join strategies, statistics, and AQE.
• Databricks — Adaptive Query Execution — Details on dynamic partition coalescing, join strategy changes, and skew handling.
• Databricks — Optimize Join Performance — Practical guidance on joins, statistics, and join ordering.

Continue the Series

Part 1: Stop Paying for Bad Code: How Databricks Code Optimization Saves Your Cloud Bill

Part 2: The Databricks Code Smell That Creates Expensive Shuffles

Part 3: Why repartition() Can Quietly Increase Your Databricks Bill