Cost fundamentals + optimization mindset for Data Engineers

Your Databricks job works.

It produces the correct output.

It finishes every morning.

So you assume everything is fine.

Then someone from FinOps asks:

“Why did our Databricks bill increase this month?”

You check the cluster.

Nothing obviously looks wrong.

The code has not changed much.

The data volume has increased only slightly.

But the workload is suddenly consuming significantly more compute.

This is where many data engineering teams make the wrong move.

They increase the cluster size.

More workers.

More memory.

More CPUs.

The job becomes faster.

But the bill gets bigger.

There is another approach:

Before adding more compute, optimize the work that your code is asking Spark to perform.

A poorly designed PySpark transformation can trigger unnecessary shuffles, scan excessive data, create inefficient partitions, cause skew, or repeatedly recompute expensive operations.

And all of that eventually translates into compute consumption.

📚 Learn Apache Spark End-to-End

Databricks performance optimization starts with understanding how Apache Spark actually executes your code.

👉 Read the End-to-End Apache Spark Tutorial Series — from Spark architecture and execution fundamentals to PySpark, shuffles, partitioning, joins, caching, AQE, and performance optimization.

This is the first article in my series:

On Optimizing Your Code to Save Your Bill on Databricks

The goal of this series is simple:

Write better Spark code → do less unnecessary work → use compute more efficiently → control your Databricks cost.

And importantly, this doesn’t mean blindly optimizing everything.

It means measuring first, identifying the real bottleneck, and then optimizing the workload that actually matters.

A quick note before we start

If you enjoy practical articles on Python, PySpark, Databricks, Data Engineering, Machine Learning and GenAI, consider following me and subscribing to GeekyCodes so you don’t miss the upcoming articles in this series.

1. Your Databricks Bill Is Not Just a Cluster Problem

When engineers hear:

“Our Databricks costs are too high.”

The first thought is often:

“We need a smaller cluster.”

But compute sizing is only one part of the problem.

Databricks describes cost optimization around several principles, including choosing appropriate resources, dynamically allocating resources, monitoring consumption, and designing cost-effective workloads.

Think about a simple pipeline:

Source Data
     ↓
Read
     ↓
Filter
     ↓
Join
     ↓
GroupBy
     ↓
Aggregation
     ↓
Write

Suppose the pipeline processes 500 GB of data.

If your code reads all 500 GB even though you only need 50 GB, the cluster has to do unnecessary work.

If your join causes a massive shuffle, the cluster has to move data between executors.

If your data is heavily skewed, a few tasks may process dramatically more data than the others.

If your pipeline creates millions of small files, subsequent jobs may spend significant time dealing with file-level overhead.

The infrastructure is doing exactly what you asked it to do.

The problem is that you asked it to do too much work.

2. The Real Relationship Between Code and Cost

A useful mental model is:

                 Your Code
                    ↓
             Spark Execution Plan
                    ↓
        ┌───────────┼────────────┐
        ↓           ↓            ↓
      I/O         Shuffle      Compute
        ↓           ↓            ↓
        └───────────┼────────────┘
                    ↓
              Job Runtime
                    ↓
             Resource Usage
                    ↓
                 $$$

So:

Code optimization is indirectly cost optimization.

This does not mean that every 10% reduction in runtime automatically produces a 10% reduction in your Databricks bill.

Your actual cost depends on factors such as compute type, pricing model, resource utilization, workload characteristics, and platform configuration.

Databricks usage can involve DBUs as well as cloud infrastructure costs, and Databricks provides billing system tables for analyzing usage and pricing.

But the fundamental principle remains:

If you make Spark perform less unnecessary work, you create an opportunity to reduce resource consumption and/or runtime.

3. Think in Terms of Work, Not Just Machines

Imagine two data pipelines.

Pipeline A

500 GB input
     ↓
500 GB scanned
     ↓
300 GB shuffled
     ↓
2000 tasks
     ↓
45 minutes

Pipeline B

500 GB input
     ↓
80 GB scanned
     ↓
20 GB shuffled
     ↓
500 tasks
     ↓
12 minutes

Both pipelines may produce exactly the same result.

But they are asking the infrastructure to perform very different amounts of work.

This is why I prefer asking:

“How much unnecessary work is my code causing?”

instead of immediately asking:

“How can I get a bigger cluster?”

4. The Five Biggest Sources of Expensive Spark Work

Across many Spark workloads, several patterns repeatedly deserve investigation.

1. Excessive Data Scanning

You only need a subset of the data.

But your query reads everything.

For example:

df = spark.read.format("delta").load("/data/orders")
df.filter("country = 'India'")

If the storage layout and query allow pruning, Spark may avoid reading irrelevant data.

But if your table is poorly organized or the predicate cannot be effectively pushed down, Spark may still perform much more I/O than necessary.

The optimization mindset is:

Can I eliminate unnecessary data before expensive operations begin?

2. Expensive Shuffles

Shuffle is one of the most important concepts to understand when optimizing Spark.

Operations such as:

groupBy()
join()
distinct()
orderBy()
repartition()

can require data to move between executors.

Conceptually:

Executor 1 ─────┐
Executor 2 ─────┤
Executor 3 ─────┼──→ Shuffle ──→ New Distribution
Executor 4 ─────┤
Executor 5 ─────┘

That movement can involve network transfer, serialization, sorting, memory pressure, and disk spill.

This is why a seemingly innocent transformation can become expensive at scale.

We’ll dedicate an entire article in this series to shuffle optimization.

5. Data Skew: When “Parallel” Isn’t Actually Parallel

Suppose you have 1 billion records.

You expect Spark to distribute them evenly.

But your data looks like this:

Partition 1 → 10 million rows
Partition 2 → 11 million rows
Partition 3 → 9 million rows
Partition 4 → 12 million rows
...
Partition 99 → 10 million rows
Partition 100 → 400 million rows

Now you have a problem.

Most executors finish quickly.

One executor is still processing the enormous partition.

Your job effectively waits for the slowest tasks.

This can create:

Fast tasks
████████████████
Slow task
████████████████████████████████████████████████████

This is data skew.

And simply adding more workers doesn’t necessarily solve the underlying problem.

Spark and Databricks provide mechanisms such as Adaptive Query Execution that can help with certain skew and execution-plan problems, but you still need to understand the workload rather than assuming more compute will fix it. Databricks documents AQE and other runtime optimizations as part of its performance recommendations.

6. Small Files: The Problem Nobody Notices Until It Hurts

Consider this:

1 large file

versus:

10,000 tiny files

Both might contain approximately the same amount of data.

But the second layout creates much more file-management overhead.

A pipeline might spend time:

Discover files
     ↓
Open files
     ↓
Read metadata
     ↓
Schedule tasks
     ↓
Process data

instead of efficiently processing larger files.

This is why file layout is an important part of Databricks performance and cost optimization.

Databricks recommends Delta Lake as the storage framework for its platform and provides optimizations designed to improve workload performance and reduce unnecessary compute time.

We’ll explore this in a dedicated article later in the series.

7. The Most Dangerous Optimization Mistake

Here is one of the biggest mistakes I see engineers make:

Optimizing based on intuition instead of evidence.

For example:

"The job is slow."
        ↓
"Let's increase workers."
        ↓
"Still slow."
        ↓
"Let's increase memory."
        ↓
"Still slow."
        ↓
"Let's add more executors."

This can become expensive trial and error.

Instead:

Slow Job
   ↓
Measure
   ↓
Spark UI
   ↓
Execution Plan
   ↓
Identify Bottleneck
   ↓
Optimize
   ↓
Measure Again

That is the mindset this entire series will follow.

8. Start With explain()

One of the simplest tools available to a Spark developer is:

df.explain()

For more detail:

df.explain("formatted")

You don’t need to become a Spark internals expert overnight.

Start by asking:

What is Spark actually doing?

Look for operators such as:

FileScan
Filter
Project
Exchange
Sort
BroadcastHashJoin
SortMergeJoin
HashAggregate

One particularly important operator is:

Exchange

An Exchange often indicates a redistribution of data.

For example:

Scan
  ↓
Filter
  ↓
Exchange
  ↓
Sort
  ↓
Join

That Exchange deserves investigation.

Not every Exchange is bad.

Distributed systems need to redistribute data for many legitimate operations.

The question is:

Is this redistribution necessary, and is it more expensive than it needs to be?

9. Learn to Read the Spark UI

Your code is only the beginning.

The Spark UI shows you what actually happened during execution.

When investigating an expensive job, look at:

Jobs

How many jobs are being executed?

Stages

Which stages consume the most time?

Tasks

Are task durations relatively balanced?

Shuffle

How much data is being read and written?

Spill

Is data spilling to memory/disk?

Input

How much data is actually being read?

Output

How much data is being generated?

A useful investigation might look like:

Job takes 40 minutes
        ↓
Stage 7 takes 31 minutes
        ↓
Stage 7 has huge shuffle
        ↓
One task takes 28 minutes
        ↓
Data distribution is highly skewed
        ↓
Investigate join/groupBy key

Now you have a real optimization target.

10. Before Optimizing Code, Establish a Baseline

Never optimize without measuring the original workload.

Create a simple baseline.

Then change one thing.

Run again.

Now you can answer:

Did my optimization actually improve the workload?

That is engineering.

11. Performance Optimization ≠ Cost Optimization

This distinction is extremely important.

Suppose:

Option A

Runtime: 60 minutes
Compute: 4 workers

Option B

Runtime: 20 minutes
Compute: 12 workers

Option B is faster.

But is it cheaper?

Not necessarily.

You need to evaluate the actual resource consumption and pricing model.

On the other hand, consider:

Option C

Runtime: 25 minutes
Compute: 4 workers

If the same workload can be reduced from 60 minutes to 25 minutes without requiring substantially more resources, you have potentially improved both performance and cost efficiency.

This leads to a very useful principle:

Don’t optimize for runtime alone. Optimize for cost-effective throughput.

Databricks’ performance-efficiency guidance similarly frames optimization around scaling resources appropriately and achieving cost-effective throughput.

12. The Optimization Hierarchy

When I optimize a Databricks pipeline, I like to think about the problem in layers.

Layer 1 — Business Requirement

Ask:

Do we actually need this computation?

For example:

Requirement:
Refresh dashboard every 6 hours.

But the pipeline runs continuously.

That’s not a Spark-code problem.

That’s an architecture problem.

Layer 2 — Data Volume

Ask:

Can I process less data?

Examples:

• incremental processing
• filtering earlier
• partition pruning
• avoiding unnecessary columns
• processing only changed records

Layer 3 — Data Layout

Ask:

Is the data physically organized efficiently?

Examples:

• Delta Lake
• appropriate file sizes
• optimized table layout
• effective clustering/partitioning strategy

Layer 4 — Query Plan

Ask:

Is Spark executing the query efficiently?

Examples:

• join strategy
• predicate pushdown
• column pruning
• broadcast joins where appropriate
• AQE
• avoiding unnecessary shuffles

Layer 5 — Code

Ask:

Is my PySpark/SQL code causing unnecessary computation?

Examples:

collect()
repartition()
distinct()
groupBy()
withColumn()

Used appropriately, these are perfectly valid.

Used incorrectly at scale, they can become expensive.

Layer 6 — Compute

Only after understanding the workload should you ask:

What compute configuration does this workload actually need?

Databricks recommends choosing resources based on workload characteristics, including data volume, computational complexity, data source, partitioning, and required parallelism.

13. A Simple Example: Don’t Process What You Don’t Need

Imagine a table with:

2 billion rows

Your job needs only:

Last 7 days

A naive approach might conceptually process the entire dataset and filter later.

A better approach is to structure the query so the filter can be applied as early and efficiently as possible.

For example:

recent_orders = (
    spark.read.table("orders")
    .filter("order_date >= current_date() - INTERVAL 7 DAYS")
)

Then perform expensive operations on recent_orders.

The principle is more important than the syntax:

Bad:
2 Billion Rows
      ↓
Join
      ↓
Aggregation
      ↓
Filter
Better:
2 Billion Rows
      ↓
Filter
      ↓
Small Dataset
      ↓
Join
      ↓
Aggregation

You want expensive operations to operate on the smallest practical dataset.

14. Another Important Rule: Select Only What You Need

Suppose your table contains 80 columns.

Your pipeline needs 6.

Don’t carry all 80 columns through expensive transformations unnecessarily.

Instead:

orders = (
    spark.read.table("orders")
    .select(
        "order_id",
        "customer_id",
        "order_date",
        "amount",
        "product_id",
        "region"
    )
)

This is an example of column pruning.

Again, the goal is simple:

Move less data through the pipeline.

Less data can mean less I/O, less memory pressure, less network movement, and less work for downstream operators.

15. Don’t Cache Everything

Another common misconception:

“Caching makes Spark faster, so we should cache our DataFrames.”

No.

Caching has a cost.

Consider:

df.cache()

If df is used repeatedly and recomputation is expensive, caching may make sense.

But if:

df.cache()
df.count()
# DataFrame used once

you may be consuming resources without gaining meaningful benefit.

The right question isn’t:

“Can I cache this?”

It is:

“Will the cost of caching be lower than the cost of recomputing this data?”

That’s the optimization mindset.

16. Don’t Use repartition() Just Because You Can

Another common pattern:

df = df.repartition(100)

Looks harmless.

But repartition() generally requires redistribution of data.

That means:

Current Partitions
       ↓
Shuffle
       ↓
100 Partitions

If the existing partitioning is already appropriate, you’ve potentially introduced unnecessary work.

This deserves its own article because partitioning decisions can have a significant effect on Spark performance.

17. The Same Logic Can Have Very Different Execution Costs

Consider two approaches.

Approach A

large_df.join(
    another_large_df,
    "customer_id"
)

Depending on the data and statistics, Spark may need to redistribute substantial amounts of data.

Approach B

If one side is genuinely small enough:

from pyspark.sql.functions import broadcast
large_df.join(
    broadcast(customer_df),
    "customer_id"
)

Spark can use a broadcast-based join strategy.

Conceptually:

Without broadcast
Large DF ──→ Shuffle ──┐
                       ├──→ Join
Large DF ──→ Shuffle ──┘

Versus:

With broadcast
Small DF
   ↓
Broadcast
   ↓
Executors
Large DF ───────────────→ Local Join

But don’t turn this into:

“Always use broadcast.”

Broadcasting is appropriate only when the broadcast side is sufficiently small and the memory implications are acceptable.

Optimization is contextual.

18. Databricks Already Does a Lot of Optimization for You

An important point:

You don’t need to manually optimize every single thing.

Modern Databricks runtimes include numerous automatic performance optimizations. Databricks documents features such as disk caching, dynamic file pruning, low-shuffle merge behavior, cost-based optimization, and Adaptive Query Execution.

That means your job isn’t:

“Override Spark’s optimizer everywhere.”

Your job is:

Understand what Spark and Databricks already do, then intervene only where the workload requires it.

This is a much healthier approach.

19. The “Don’t Tune Randomly” Rule

Here’s a simple checklist I recommend before changing Spark configurations:

❌ Don't start by changing 20 Spark configs.
❌ Don't double the cluster size without understanding the bottleneck.
❌ Don't blindly increase executor memory.
❌ Don't add repartition() everywhere.
❌ Don't cache every DataFrame.
❌ Don't assume faster = cheaper.
❌ Don't optimize a job that runs once a month before optimizing one that runs every hour.

Instead:

✅ Measure
✅ Identify the bottleneck
✅ Understand the execution plan
✅ Change one thing
✅ Benchmark again
✅ Compare resource usage
✅ Validate correctness
✅ Keep the change only if it helps

20. Optimize the Jobs That Actually Matter

Suppose your organization has:

Job A → 5 minutes → runs once/week
Job B → 20 minutes → runs daily
Job C → 45 minutes → runs hourly
Job D → 2 hours → runs once/month

Which one deserves your attention first?

Don’t automatically choose the longest-running job.

Think about:

Impact ≈ Cost per run × Frequency × Business importance

A 20-minute job running 24 times a day may deserve more attention than a 2-hour job running once a month.

This is why cost optimization should be connected to workload economics, not just technical elegance.

21. A Practical Cost Optimization Framework

Here’s the framework I’ll use throughout this series:

                 ┌──────────────────────┐
                 │  1. MEASURE          │
                 │  Runtime              │
                 │  Input                │
                 │  Shuffle              │
                 │  Spill                │
                 │  Compute              │
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │  2. IDENTIFY         │
                 │  Where is time spent?│
                 │  Where is data moving│
                 │  Where is data skewed│
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │  3. OPTIMIZE         │
                 │  Code                 │
                 │  Query                │
                 │  Data layout          │
                 │  Compute              │
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │  4. BENCHMARK        │
                 │  Run again            │
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │  5. COMPARE          │
                 │  Performance          │
                 │  Resource usage       │
                 │  Cost                │
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │  6. PRODUCTIONIZE    │
                 │  Document            │
                 │  Monitor             │
                 │  Revisit             │
                 └──────────────────────┘

This is the core philosophy of the entire series.

22. The Golden Rule

If you remember only one thing from this article, remember this:

Don’t make the cluster work harder. Make the code work smarter.

Before:

More Data
   ↓
More Compute
   ↓
More Workers
   ↓
More $$$

Think instead:

Better Code
   ↓
Less Data Processed
   ↓
Less Shuffle
   ↓
Less Spill
   ↓
Better Parallelism
   ↓
Shorter / More Efficient Workload
   ↓
Potentially Lower Cost

And remember:

Cost optimization is not about making everything slower by using fewer resources.

It is about achieving the required business outcome with an efficient amount of compute.

23. Your Databricks Optimization Checklist

Before declaring a Spark job “optimized”, ask:

Data

• Am I reading more data than necessary?
• Can I use incremental processing?
• Can partition pruning reduce the scan?
• Am I selecting only the required columns?

Transformations

• Am I causing unnecessary shuffles?
• Do I really need distinct()?
• Do I really need repartition()?
• Can an expensive operation happen after filtering?

Joins

• Is the join strategy appropriate?
• Is one side small enough for broadcasting?
• Is the join key skewed?

Partitions

• Are partitions too large?
• Are there too many tiny partitions?
• Is the data distribution balanced?

Storage

• Are there too many small files?
• Is the table layout appropriate?
• Am I using an efficient storage format?

Runtime

• What does explain("formatted") show?
• Which Spark stage consumes the most time?
• Is there significant shuffle?
• Is there spill?
• Are task durations highly uneven?

Cost

• How frequently does this job run?
• What compute is it using?
• What is the actual resource consumption?
• Did the optimization reduce runtime, resource consumption, or both?
• Did I measure the result rather than assume it?

24. What We’ll Optimize in This Series

This article is the foundation.

In the upcoming articles, we’ll go much deeper into individual cost drivers.

Part 2 — Shuffle

The Databricks Code Smell That Creates Expensive Shuffles

We’ll investigate:

groupBy()
join()
distinct()
orderBy()
repartition()

and understand what Spark’s Exchange operation really means.

Part 3 — Partitioning

Why repartition() Can Quietly Increase Your Databricks Bill

We’ll compare:

repartition()

vs.

coalesce()

and understand when each makes sense.

Part 4 — Driver Bottlenecks

Stop Using collect() in Production PySpark

We’ll see why moving a large dataset from distributed executors to the driver can become a serious scalability problem.

Part 5 — Small Files

The Small File Problem: How Millions of Tiny Files Increase Databricks Costs

We’ll look at file layout, Delta tables, and why “more files” doesn’t necessarily mean “more parallelism.”

Part 6 — Caching

Caching in Spark: When It Saves Money and When It Burns Money

We’ll answer the question:

When should you actually use .cache()?

Part 7 — Joins

Broadcast Joins: One of the Easiest Ways to Reduce Spark Compute

We’ll break down broadcast joins, Sort Merge Joins, and the conditions under which broadcasting makes sense.

Part 8 — Data Skew

Data Skew: The Silent Databricks Bill Killer

We’ll investigate skewed keys, long-running tasks, salting, and Adaptive Query Execution.

Part 9 — Data Scanning

Partition Pruning: Stop Making Databricks Read Data You Don’t Need

We’ll explore how to reduce unnecessary data reads.

Part 10 — Delta Lake

Delta Lake Optimization: OPTIMIZE, Clustering and Efficient Data Layout

We’ll move from Spark code optimization toward storage-level optimization.

Part 11 — Execution Plans

Spark SQL vs PySpark: Can Your Coding Style Change Your Databricks Bill?

We’ll compare logical and physical plans rather than relying on assumptions about which API is “faster.”

Part 12 — Real Debugging

How to Read the Databricks Spark UI and Find What’s Increasing Your Bill

We’ll take a slow job and diagnose it step by step.

Final Thoughts

Databricks gives data engineers enormous computational power.

But that power comes with a responsibility:

Just because you can process terabytes of data in minutes doesn’t mean you should make the platform process unnecessary terabytes of data.

The best data engineers don’t simply know how to make a Spark job run.

They understand:

what Spark is doing,

why it is doing it,

how much work it is performing,

and

whether that work is actually necessary.

That’s where performance engineering becomes cost engineering.

The next time someone tells you:

“This Databricks job is expensive.”

Don’t immediately ask:

“Which bigger cluster should we use?”

Ask:

“What is Spark spending all that compute on?”

That question can lead you much closer to the real problem.

Key Takeaways

1. Databricks cost is influenced by both infrastructure choices and workload design.

2. Poorly optimized code can cause unnecessary data scans, shuffles, spills, and computation.

3. Faster does not automatically mean cheaper—you need to measure resource consumption and actual cost.

4. Start with the Spark UI and execution plans before changing configurations.

5. Optimize high-frequency, high-impact workloads first.

6. Don’t blindly cache, repartition, broadcast, or increase cluster size.

7. The best optimization is usually the one that eliminates unnecessary work.

Write less work. Move less data. Shuffle less. Compute smarter. Pay for what you actually need.

References

• Databricks — Cost optimization
• Databricks — Best practices for cost optimization
• Databricks — Optimization recommendations
• Databricks — Performance efficiency best practices
• Databricks — Pricing system table reference