Understanding Directed Acyclic Graphs, transformations, actions, dependencies, stages, shuffles, and how Spark turns your PySpark code into distributed execution.

You write a few lines of PySpark:

df = spark.read.parquet("sales/")
result = (
    df
    .filter("amount > 1000")
    .select("customer_id", "amount")
    .groupBy("customer_id")
    .sum("amount")
)
result.show()

It looks like a simple sequence of operations.

But Spark doesn’t simply execute these lines one after another.

Instead, Spark builds a representation of the computation, analyzes the dependencies between operations, and eventually turns that computation into executable work across the cluster.

One of the most important concepts behind this execution model is the DAG — Directed Acyclic Graph.

If you’re preparing for a Data Engineer or Databricks interview, understanding DAGs is essential because it connects several concepts you’ve probably already encountered:

• Transformations
• Actions
• Lazy evaluation
• Narrow transformations
• Wide transformations
• Shuffle
• Stages
• Tasks
• Executors

In this tutorial, we’ll understand how all of these pieces fit together.

APACHE SPARK LEARNING PATH

┌──────────────────────────────────────────────┐
│ APACHE SPARK LEARNING PATH                   │
│                                              │
│ ✓ 1. What is Apache Spark?                   │
│ ✓ 2. Why Spark is Faster than Hadoop         │
│ ✓ 3. Spark Architecture                      │
│ ✓ 4. Driver vs Executor                      │
│ ✓ 5. Cluster Managers                        │
│ ✓ 6. RDD vs DataFrame vs Dataset              │
│ ✓ 7. Lazy Evaluation                         │
│ → 8. DAG in Apache Spark                     │
│ ○ 9. Stages and Tasks                        │
│                                              │
│ [View All Apache Spark Tutorials →]          │
└──────────────────────────────────────────────┘

1. What Is a DAG?

DAG stands for Directed Acyclic Graph.

Let’s break that down.

Directed

The operations have a defined direction.

For example:

Read
  ↓
Filter
  ↓
Select
  ↓
GroupBy
  ↓
Count

Data flows from one operation to another.

Acyclic

There are no cycles.

The computation cannot go:

A → B → C → A

Instead, it moves forward:

A → B → C → D

Graph

A graph consists of nodes and edges.

In a simplified Spark execution model:

Nodes → Operations
Edges → Dependencies between operations

So we can think of a DAG as a representation of how one operation depends on another.

2. Why Does Spark Need a DAG?

This is where DAGs become important.

Suppose you write:

df = spark.read.parquet("sales/")
df1 = df.filter("amount > 1000")
df2 = df1.select(
    "customer_id",
    "amount"
)
df3 = df2.groupBy(
    "customer_id"
).sum("amount")
df3.show()

A beginner might imagine Spark doing this:

Read data
   ↓
Execute filter
   ↓
Execute select
   ↓
Execute groupBy
   ↓
Execute sum
   ↓
Show result

But Spark’s execution model is more sophisticated.

Because Spark uses lazy evaluation, transformations are recorded rather than immediately executed.

When an action such as:

df3.show()

is called, Spark needs to determine how the entire computation should be executed.

The dependencies can be represented conceptually as:

Read
  ↓
Filter
  ↓
Select
  ↓
GroupBy
  ↓
Sum
  ↓
Show

This computation graph helps Spark determine how the work should be organized.

The Spark documentation describes transformations as lazy operations whose actual computation happens when an action requires the result.

3. DAG and Lazy Evaluation

DAG and lazy evaluation are closely connected, but they are not the same thing.

Lazy evaluation

Lazy evaluation answers:

When should Spark execute the transformations?

Answer:

When an action requires the result.

DAG

The DAG answers:

How are the operations and their dependencies organized for execution?

So the simplified relationship is:

PySpark transformations
        ↓
Lazy evaluation
        ↓
Spark records computation
        ↓
Action
        ↓
Execution planning
        ↓
DAG / execution dependencies
        ↓
Stages
        ↓
Tasks
        ↓
Executors

If you haven’t already, it’s useful to understand lazy evaluation first. It explains why Spark can see more of your computation before execution begins.

4. A Simple DAG Example

Consider:

df = spark.read.parquet("sales/")
result = (
    df
    .filter(df.amount > 1000)
    .select("customer_id", "amount")
)
result.show()

Conceptually, the computation looks like:

        Sales Data
            │
            ▼
         Filter
      amount > 1000
            │
            ▼
         Select
 customer_id, amount
            │
            ▼
          show()

The important point is that Spark doesn’t need to materialize a completely separate dataset after every transformation.

Compatible operations can be executed together as part of the same execution pipeline.

This is one reason Spark can avoid unnecessary intermediate work.

5. DAG With an Aggregation

Now let’s make the example more interesting.

result = (
    df
    .filter(df.amount > 1000)
    .select("customer_id", "amount")
    .groupBy("customer_id")
    .sum("amount")
)
result.show()

Conceptually:

Read
  │
  ▼
Filter
  │
  ▼
Select
  │
  ▼
GroupBy
  │
  ▼
Aggregation
  │
  ▼
Show

But there’s an important detail here.

groupBy() generally requires data to be reorganized by key.

That means a shuffle may occur.

So the execution can conceptually look like:

              Stage 1
       Read → Filter → Select
                    │
                    ▼
                 Shuffle
                    │
                    ▼
              Stage 2
          GroupBy → Aggregate
                    │
                    ▼
                  Result

This is where DAGs become particularly useful for understanding Spark performance.

6. DAG and Shuffle

A shuffle occurs when Spark needs to redistribute data across partitions.

For example:

df.groupBy("customer_id").sum("amount")

Imagine the input data is distributed like this:

Partition 1
Customer A
Customer B
Customer C
Partition 2
Customer A
Customer D
Customer E
Partition 3
Customer B
Customer C
Customer A

Spark needs all records for the same customer to end up together for the aggregation.

It may therefore redistribute the data:

Before Shuffle
Partition 1 ──┐
Partition 2 ──┼──→ Shuffle
Partition 3 ──┘
                    ↓
After Shuffle
Customer A → Partition X
Customer B → Partition Y
Customer C → Partition Z

Shuffle is expensive because it can involve network communication, serialization, and disk I/O.

Spark’s own documentation describes shuffle as the mechanism used to redistribute data across partitions and notes that it can be costly.

7. DAG and Stages

A very important interview concept is:

A Spark job is divided into stages, and shuffle boundaries separate stages.

For example:

df.filter("amount > 1000") \
  .groupBy("customer_id") \
  .count()

Conceptually:

             JOB

              │
              ▼

       ┌──────────────┐
       │   Stage 1    │
       │              │
       │ Read         │
       │ Filter       │
       └──────┬───────┘
              │
           Shuffle
              │
              ▼
       ┌──────────────┐
       │   Stage 2    │
       │              │
       │ GroupBy      │
       │ Count        │
       └──────────────┘

The DAGScheduler divides computation into stages at shuffle boundaries. Spark’s source code describes stages as sets of tasks with the same shuffle dependencies and notes that stages are separated at shuffle boundaries.

This leads to a useful mental model:

DAG
 │
 ├── Stage 1
 │      ├── Task
 │      ├── Task
 │      └── Task
 │
 ├── Stage 2
 │      ├── Task
 │      ├── Task
 │      └── Task
 │
 └── Stage 3
        ├── Task
        ├── Task
        └── Task

We’ll go deeper into Stages and Tasks in the next tutorial.

8. Narrow vs Wide Dependencies

To understand why Spark creates stage boundaries, you need to understand narrow and wide dependencies.

Narrow Dependency

A narrow dependency means that a child partition depends on a relatively small, predictable set of parent partitions—typically one parent partition.

Examples include:

filter()
map()
select()

Conceptually:

Partition 1 ──→ Partition 1
Partition 2 ──→ Partition 2
Partition 3 ──→ Partition 3

There is no need to redistribute the entire dataset across the cluster.

This makes these operations good candidates for pipelining within a stage.

Wide Dependency

A wide dependency occurs when output partitions depend on multiple input partitions.

Common examples include:

groupBy()
join()
distinct()
orderBy()

These operations can require a shuffle.

Conceptually:

Partition 1 ──┐
Partition 2 ──┼──→ Shuffle → New partitions
Partition 3 ──┘

This creates a natural stage boundary.

9. How Spark Pipelines Transformations

Consider:

df.filter("age > 30") \
  .select("name", "age") \
  .withColumn("age_plus_10", df.age + 10)

These operations can often be pipelined together.

Conceptually:

Partition
   │
   ▼
Filter
   │
   ▼
Select
   │
   ▼
withColumn
   │
   ▼
Output

Spark doesn’t necessarily create a separate stage for every transformation.

This is an important distinction.

Incorrect mental model

filter → Stage 1
select → Stage 2
withColumn → Stage 3

Better mental model

filter
  ↓
select
  ↓
withColumn
  ↓
same execution pipeline

A stage boundary generally appears when a dependency requires a shuffle.

10. DAG vs Stage vs Task

These three concepts are frequently confused in interviews.

Here’s the simplest way to remember them.

DAG

Represents the overall computation and its dependencies.

Read
 ↓
Filter
 ↓
Shuffle
 ↓
Aggregate

Stage

A portion of the computation between shuffle boundaries.

Stage 1:
Read → Filter
Stage 2:
Aggregate

Task

A unit of work executed for a particular partition within a stage.

For example:

Stage 1
Partition 1 → Task 1
Partition 2 → Task 2
Partition 3 → Task 3
Partition 4 → Task 4

Spark’s scheduler source describes a task as an individual unit of work sent to a machine.

So remember:

DAG
 ↓
Stages
 ↓
Tasks
 ↓
Executors

11. A Complete Example

Let’s take a realistic Data Engineering example.

Suppose you have a transaction dataset:

transactions = spark.read.parquet(
    "/data/transactions"
)

You want to:

1. Keep US transactions
2. Keep transactions above $1,000
3. Select relevant columns
4. Aggregate by customer

Code:

result = (
    transactions
    .filter("country = 'US'")
    .filter("amount > 1000")
    .select(
        "customer_id",
        "amount"
    )
    .groupBy("customer_id")
    .sum("amount")
)
result.show()

The logical flow is:

Read Transactions
        │
        ▼
Filter Country
        │
        ▼
Filter Amount
        │
        ▼
Select Columns
        │
        ▼
Shuffle
        │
        ▼
Group By Customer
        │
        ▼
Sum Amount
        │
        ▼
Result

The execution can be viewed conceptually as:

┌──────────────────────────────┐
│           Stage 1            │
│                              │
│ Read                         │
│   ↓                          │
│ Filter country = US          │
│   ↓                          │
│ Filter amount > 1000         │
│   ↓                          │
│ Select                       │
└──────────────┬───────────────┘
               │
               ▼
            SHUFFLE
               │
               ▼
┌──────────────────────────────┐
│           Stage 2            │
│                              │
│ GroupBy customer_id          │
│   ↓                          │
│ Sum amount                   │
└──────────────────────────────┘

This is the mental model you should have when looking at a Spark pipeline.

12. What Happens When You Call an Action?

Until now, we’ve talked about the DAG conceptually.

Let’s connect it to execution.

Suppose you have:

result = (
    df
    .filter("amount > 1000")
    .groupBy("customer_id")
    .sum("amount")
)

At this point, Spark hasn’t necessarily performed the complete distributed computation.

Then you call:

result.count()

count() is an action.

The action causes Spark to execute the computation required to produce the result.

A simplified flow is:

PySpark Code
     │
     ▼
Transformations
     │
     ▼
Execution Plan
     │
     ▼
DAG / Dependencies
     │
     ▼
Stages
     │
     ▼
Tasks
     │
     ▼
Executors
     │
     ▼
Result

Spark’s RDD programming guide confirms that actions trigger computation and that Spark breaks the required computation into tasks that run across the cluster.

13. Does Every Action Create a DAG?

A common interview question is:

Does every action create a new job?

For practical Spark reasoning, an action on an RDD/DataFrame generally triggers a job for the required computation.

For example:

df.count()

can trigger one job.

Then:

df.show()

can trigger another job if the required data isn’t already available through persistence or another mechanism.

This is why repeatedly doing:

df.show()
df.show()
df.show()

on an expensive lineage can result in repeated computation.

Caching can change this behavior by allowing Spark to reuse materialized data.

14. DAG and Caching

Consider:

processed = (
    df
    .filter("amount > 1000")
    .groupBy("customer_id")
    .sum("amount")
)

Suppose you use processed several times:

processed.count()
processed.show()
processed.write.parquet("/output")

Without persistence, Spark may need to recompute the required lineage for separate actions.

You can consider:

processed.cache()

Then the first action can populate the cache, and subsequent actions may reuse the cached data.

But don’t blindly cache every DataFrame.

Caching consumes resources, and Spark’s documentation notes that transformed RDDs may otherwise be recomputed for each action while persistence allows reuse.

A good production mindset is:

Cache data when the cost of recomputation is greater than the cost of storing and maintaining the cached data.

15. DAG and Fault Tolerance

DAGs are also connected to Spark’s fault-tolerance model.

Suppose an executor fails while processing a partition.

Spark can use the lineage/dependency information to determine how the lost computation can be recomputed.

Conceptually:

Source
  ↓
Filter
  ↓
Transformation
  ↓
Partition Lost
  ↓
Recompute

This is one reason Spark’s distributed abstractions can recover from certain failures without restarting the entire application.

RDDs maintain lineage information that describes how derived datasets were created.

16. DAG Is Not the Same as a DataFrame Logical Plan

This is an important distinction for experienced Data Engineers.

You’ll often hear people say:

“Spark creates a DAG from your DataFrame operations.”

That’s useful as a high-level explanation, but technically it can be too simplistic.

For DataFrame and Spark SQL workloads, Spark goes through planning and optimization stages before physical execution.

A simplified view is:

PySpark / SQL
      ↓
Logical Plan
      ↓
Optimized Logical Plan
      ↓
Physical Plan
      ↓
Execution

The scheduler then works with the resulting execution dependencies to create stages and tasks.

So don’t think of DAG as simply:

DataFrame operations = DAG

A better mental model is:

Your code
   ↓
Logical / physical planning
   ↓
Execution dependencies
   ↓
DAG scheduling
   ↓
Stages
   ↓
Tasks

This distinction becomes increasingly important when you start learning Spark SQL and Databricks performance optimization.

17. How to Visualize the DAG in Databricks

One of the most useful skills for a Data Engineer is being able to move from theory to debugging.

Suppose a Spark job is taking two hours.

You shouldn’t immediately say:

“Increase the cluster size.”

Instead, inspect the execution.

Databricks provides Spark monitoring information that can help you understand:

• Jobs
• Stages
• Tasks
• Shuffle
• Input/output
• Task duration
• Failed tasks
• Skew
• Executor behavior

You can then ask:

Which stage is slow?
        ↓
Is there a shuffle?
        ↓
Is one task much slower?
        ↓
Is there data skew?
        ↓
Are partitions too large?
        ↓
Is the join expensive?
        ↓
Is the cluster underutilized?

This is where DAG knowledge becomes practical rather than theoretical.

18. DAG and Performance Optimization

Understanding the DAG helps you identify performance problems.

Problem 1: Excessive Shuffle

Stage 1
   ↓
Shuffle
   ↓
Stage 2
   ↓
Shuffle
   ↓
Stage 3

Multiple expensive shuffles can significantly increase execution time.

Problem 2: Data Skew

Suppose one customer has 40% of all transactions.

During a join or aggregation:

Partition 1 → 10 GB
Partition 2 → 2 GB
Partition 3 → 2 GB
Partition 4 → 2 GB

One task may take significantly longer than the others.

This can cause the entire stage to wait for the slow task.

Techniques such as better partitioning, broadcast joins where appropriate, salting, and Adaptive Query Execution can help depending on the workload.

Problem 3: Too Many Partitions

More partitions don’t automatically mean better performance.

Too many tiny tasks can introduce scheduling overhead.

Problem 4: Too Few Partitions

Too few partitions can leave cluster resources underutilized.

Problem 5: Unnecessary Actions

Code such as:

df.show()
df.count()
df.collect()

can trigger multiple computations.

During development, repeatedly triggering actions on large datasets can make notebooks unnecessarily slow.

19. A Real-World Data Engineering Example

Imagine an ETL pipeline processing 5 TB of transaction data.

The pipeline looks like:

Raw Transactions
       ↓
Filter Invalid Records
       ↓
Join Customer Data
       ↓
Aggregate Transactions
       ↓
Calculate Metrics
       ↓
Write Delta Table

Suppose the pipeline takes four hours.

Instead of immediately increasing the cluster size, you inspect the execution.

You discover:

Stage 1 → 10 minutes
Stage 2 → 15 minutes
Stage 3 → 3 hours 20 minutes
Stage 4 → 10 minutes

Stage 3 is clearly the bottleneck.

You inspect it further.

You discover a huge shuffle caused by a join.

Then you discover:

Customer ID = 10001

contains 35% of all records.

Now you have identified a data-skew problem.

Possible optimization strategies could include:

• Broadcast join if the other dataset is sufficiently small
• Better partitioning
• Salting for severe skew
• AQE skew join optimization
• Reducing unnecessary columns before the join
• Filtering data earlier

The key point is that understanding the execution graph helps you find the actual bottleneck instead of guessing.

20. Common DAG Interview Questions

Let’s turn the concept into interview-ready questions.

Question 1: What is DAG in Spark?

A strong answer:

DAG stands for Directed Acyclic Graph. It represents the dependencies between operations in a Spark computation. Spark uses these dependencies to organize execution into stages and tasks. Shuffle boundaries typically separate stages.

Question 2: Why is DAG important?

A good answer:

DAG allows Spark to understand the dependency structure of a computation and organize execution efficiently. Compatible transformations can be pipelined, while operations requiring shuffle create stage boundaries.

Question 3: Does every transformation create a new stage?

No.

For example:

df.filter(...)
  .select(...)
  .withColumn(...)

These operations can often be pipelined within the same stage.

A shuffle-producing operation can introduce a stage boundary.

Question 4: What creates a stage boundary?

Typically, a shuffle boundary creates a stage boundary.

For example:

df.filter(...)
  .groupBy(...)
  .count()

Conceptually:

Filter
  ↓
Stage 1
  ↓
Shuffle
  ↓
Stage 2
  ↓
GroupBy
  ↓
Count

Question 5: What is the difference between DAG and lineage?

Lineage describes how a dataset was derived through a sequence of transformations.

A DAG represents the directed dependency structure used to organize computation.

They are closely related concepts, but you shouldn’t treat them as exact synonyms.

Question 6: Does filter() create a new stage?

Not necessarily.

A filter() is typically a narrow operation and can often be pipelined with other compatible operations in the same stage.

Question 7: Does groupBy() create a stage?

groupBy() itself is a transformation, but it generally requires a shuffle for the aggregation, so the shuffle creates a stage boundary in the execution.

Question 8: How does DAG help with performance?

It allows Spark to organize compatible operations into pipelines and identify shuffle boundaries.

This helps reduce unnecessary intermediate processing and provides a useful way to reason about where expensive data movement occurs.

21. The DAG Mental Model You Should Remember

If you’re a beginner, don’t try to memorize every Spark scheduler detail.

Start with this:

                 PySpark Code
                      │
                      ▼
              Transformations
                      │
                      ▼
                 Lazy Plan
                      │
                      ▼
                   Action
                      │
                      ▼
             Execution Planning
                      │
                      ▼
                     DAG
                      │
          ┌───────────┴───────────┐
          ▼                       ▼
       Stage 1                 Stage 2
          │                       │
       Tasks                   Tasks
          │                       │
          ▼                       ▼
      Executors               Executors

And remember:

Narrow operation
      ↓
Can often be pipelined
Wide operation
      ↓
Shuffle
      ↓
Stage boundary

That mental model will take you a long way.

22. DAG, Stages and Tasks: The Complete Picture

Let’s put everything together.

Suppose we have:

df.filter("amount > 1000") \
  .groupBy("customer_id") \
  .count()

The conceptual execution is:

                 Spark Job
                    │
                    ▼
                  DAG
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
       Stage 1             Stage 2
          │                   │
   Read → Filter         GroupBy → Count
          │                   │
          ▼                   ▼
       Tasks               Tasks
          │                   │
          ▼                   ▼
     Executors           Executors

Between the stages:

Stage 1
   ↓
Shuffle
   ↓
Stage 2

This is the foundation for understanding Spark’s execution model.

23. Why DAG Matters for Data Engineers

You don’t need to understand Spark’s DAG merely to pass an interview.

It helps you answer real production questions.

For example:

“Why is my Spark job slow?”

Look at the stages.

“Why is one stage taking much longer?”

Look for:

• Data skew
• Large partitions
• Uneven data distribution
• Expensive joins
• Shuffle

“Why is my cluster underutilized?”

Look at:

• Number of tasks
• Number of partitions
• Stage parallelism
• Data size

“Why did adding more workers not help?”

The bottleneck may be:

• Shuffle
• Skew
• Driver-side work
• I/O
• Poor partitioning
• Insufficient parallelism

Understanding the DAG gives you a framework for answering these questions.

24. Key Takeaways

Let’s summarize the most important concepts.

1. DAG means Directed Acyclic Graph

It represents directed dependencies between operations.

2. Spark uses lazy evaluation

Transformations are not immediately executed.

3. An action triggers execution

Examples include:

count()
show()
collect()
write()

4. Transformations can be pipelined

Compatible narrow transformations can often run within the same stage.

5. Shuffle creates a major execution boundary

Operations such as aggregations and joins may require data redistribution.

6. Stages are separated by shuffle boundaries

A job can contain multiple stages.

7. Tasks execute work for partitions

Each stage contains tasks that operate on partitions.

8. Executors execute the tasks

The Driver coordinates the application, while executors perform the distributed computation.

9. DAG knowledge helps with optimization

It helps you reason about:

• Shuffle
• Data skew
• Partitioning
• Joins
• Parallelism
• Caching
• Stage bottlenecks

Final Thought

A Spark pipeline may look like a simple sequence of Python statements:

read()
filter()
select()
join()
groupBy()
write()

But underneath, Spark is building and organizing a distributed computation.

The important mental model is:

Your Code
   ↓
Transformations
   ↓
Lazy Evaluation
   ↓
Execution Plan
   ↓
DAG
   ↓
Stages
   ↓
Tasks
   ↓
Executors
   ↓
Result

Once you understand this flow, Spark performance optimization becomes much easier to reason about.

You stop asking:

“Why is Spark slow?”

and start asking:

“Which stage is slow?”

“Where is the shuffle?”

“Are the partitions balanced?”

“Is there data skew?”

“Can these operations be pipelined?”

“Is the join causing excessive data movement?”

That’s the mindset you need when working with Spark in production.

Continue Learning Apache Spark

You’ve now learned how Spark represents computation as a Directed Acyclic Graph and how transformations, dependencies, shuffles, and stages are connected.

But there is still one important question:

How does Spark turn a DAG into actual work running across executors?

That’s where Stages and Tasks come in.

Recommended next step

1. Stages and Tasks

Learn how Spark divides a DAG into stages, creates tasks for individual partitions, and schedules those tasks across executors.

2. Narrow vs Wide Transformations

Understand why some transformations can be pipelined while others trigger shuffle.

3. Spark Shuffle

Learn what actually happens when data moves between partitions and why shuffle is often one of the biggest performance bottlenecks.

4. Spark UI

Learn how to use the Spark UI to identify slow stages, skewed tasks, shuffle problems, and execution bottlenecks.

[View Complete Apache Spark Learning Path →]

Next Step: Stages and Tasks →

You’ve seen the DAG.

Now let’s go one level deeper.

In the next tutorial, we’ll break the DAG into:

DAG
 ↓
Stages
 ↓
Tasks
 ↓
Partitions
 ↓
Executors

You’ll learn exactly how Spark converts a high-level computation into units of work that can run across a distributed cluster.

Continue to Stages and Tasks →