Apache Spark looks simple from the outside: write a few DataFrame operations and run your pipeline. Under the hood, however, Spark builds an execution plan, breaks the work into stages and tasks, distributes those tasks across executors, and coordinates the entire computation through the Driver.

Introduction

In the previous article, “What is Apache Spark?”, we introduced Spark as a distributed data processing engine.

But knowing what Spark does is only the beginning.

If you want to work professionally with PySpark, Databricks, or large-scale data pipelines, you need to understand what happens after you execute code such as:

df.filter(df.salary > 50000) \
  .groupBy("department") \
  .count()

Does the Driver process the data?

Where does the actual computation happen?

What is an Executor?

What is a Job?

What is a Stage?

What is a Task?

And perhaps most importantly:

How does Spark turn a few lines of PySpark code into distributed computation across a cluster?

This article answers those questions step by step.

1. The Big Picture

At a high level, a Spark application consists of three major components:

                  Spark Application
                         │
                         ▼
                  ┌──────────────┐
                  │    Driver    │
                  │              │
                  │ SparkSession │
                  │ Scheduler    │
                  └──────┬───────┘
                         │
                Requests Resources
                         │
                         ▼
                ┌─────────────────┐
                │ Cluster Manager │
                └────────┬────────┘
                         │
              Allocates Executors
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
    ┌──────────┐   ┌──────────┐   ┌──────────┐
    │Executor 1│   │Executor 2│   │Executor 3│
    │          │   │          │   │          │
    │ Tasks    │   │ Tasks    │   │ Tasks    │
    │ Memory   │   │ Memory   │   │ Memory   │
    └──────────┘   └──────────┘   └──────────┘

The basic responsibilities are:

Driver → coordinates the application.

Cluster Manager → provides computing resources.

Executors → perform the actual data processing.

However, this is only the infrastructure view.

To understand Spark execution, we also need to understand:

Application
     ↓
Job
     ↓
Stage
     ↓
Task

We’ll come back to this hierarchy shortly.

2. What Is a Spark Application?

A Spark application is a program written using Spark APIs.

For example:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .appName("EmployeeAnalysis") \
    .getOrCreate()
df = spark.read.parquet("/data/employees")
result = df.filter(
    df.salary > 100000
).groupBy(
    "department"
).count()
result.show()

When this program starts, Spark creates a Driver for the application.

The Driver then coordinates the distributed execution of the program.

3. The Driver

The Driver is the central coordinator of a Spark application.

You can think of it as the brain of the application.

It is responsible for:

• Creating the Spark context
• Converting user code into an execution plan
• Creating jobs
• Breaking jobs into stages
• Scheduling tasks
• Communicating with executors
• Tracking execution progress

A common misconception is:

“The Driver processes all the data.”

That’s incorrect.

The Driver primarily coordinates the computation.

The actual distributed data processing happens on the Executors.

4. SparkSession

When working with modern PySpark, you’ll typically create a SparkSession.

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .appName("MyApplication") \
    .getOrCreate()

The SparkSession is the main entry point for working with Spark’s DataFrame and SQL APIs.

For example:

df = spark.read.parquet("/data/sales")

The Driver uses the SparkSession to interact with the Spark execution environment.

5. Cluster Manager

The Driver needs computing resources to execute the application.

That’s where the Cluster Manager comes in.

The Cluster Manager is responsible for allocating resources such as:

• CPU
• Memory
• Executors

Depending on the environment, Spark can run with different cluster managers.

Common examples include:

• Spark Standalone
• Apache YARN
• Kubernetes

Managed platforms such as Databricks provide their own infrastructure and cluster management experience on top of Spark.

Conceptually:

Driver
   │
   │ Request resources
   ▼
Cluster Manager
   │
   │ Allocate resources
   ▼
Executors

6. Executors

Executors are the processes that perform the actual computation.

Each executor runs on a worker machine and provides resources for executing tasks.

An executor is responsible for:

• Running tasks
• Processing partitions
• Storing cached data
• Returning results to the Driver

For example:

Executor 1
├── Task 1
├── Task 2
├── Task 3
└── Task 4
Executor 2
├── Task 5
├── Task 6
├── Task 7
└── Task 8

The number of tasks an executor can run concurrently depends largely on the executor’s available CPU cores.

7. Worker Node vs Executor

These two concepts are often confused.

A worker node is a machine in the cluster.

An executor is a process running on that machine.

For example:

Worker Node 1
└── Executor 1
Worker Node 2
└── Executor 2
Worker Node 3
└── Executor 3

Depending on the cluster configuration, a worker node may run one or more executors.

8. What Happens When You Run Spark Code?

Consider:

df = spark.read.parquet("/data/sales")
result = df.filter(
    df.amount > 1000
).groupBy(
    "customer_id"
).sum("amount")
result.show()

Let’s follow the execution.

Step 1 — Application Starts

The Spark application starts and the Driver is created.

Step 2 — DataFrame Operations Are Defined

Spark receives:

filter()
groupBy()
sum()

These operations describe what we want Spark to do.

Step 3 — Spark Builds an Execution Plan

Spark analyzes the operations and builds an execution plan.

Step 4 — An Action Is Called

The following statement is an action:

result.show()

This triggers execution.

Step 5 — Job Is Created

Spark creates a Job corresponding to the action.

Step 6 — Job Is Divided into Stages

Spark divides the job into stages based on dependencies between operations.

Step 7 — Stages Become Tasks

Each stage is divided into tasks.

Step 8 — Tasks Are Sent to Executors

Executors process the data partitions.

This gives us:

PySpark Code
     ↓
Execution Plan
     ↓
Action
     ↓
Job
     ↓
Stages
     ↓
Tasks
     ↓
Executors
     ↓
Result

9. What Is a Job?

A Job represents a computation triggered by an action.

Examples of actions include:

df.show()

df.count()

df.collect()

df.write.parquet(...)

When an action requires Spark to actually compute the result, Spark creates a job.

For example:

df.filter(df.age > 30).count()

The count() action triggers a Spark Job.

10. What Is a Stage?

A Job is divided into one or more Stages.

A stage is a set of tasks that can execute together without requiring a shuffle boundary between them.

The important concept here is shuffle.

Some operations can be performed independently on each partition.

Others require data to move between partitions.

For example:

df.filter(...)

usually doesn’t require a shuffle.

But:

df.groupBy("customer_id").count()

may require data with the same customer_id to be brought together.

That data movement creates a shuffle boundary and can result in multiple stages.

11. What Is a Task?

A Task is the smallest unit of work that Spark schedules for execution.

A task typically processes one partition of data for a particular stage.

For example, suppose a stage contains:

100 partitions

Spark can create approximately:

100 tasks

for that stage.

Conceptually:

Stage
│
├── Task 1 → Partition 1
├── Task 2 → Partition 2
├── Task 3 → Partition 3
├── ...
└── Task 100 → Partition 100

Those tasks can run in parallel, subject to available cluster resources.

12. Application → Job → Stage → Task

This hierarchy is one of the most important concepts in Spark.

Application
    │
    ├── Job 1
    │     ├── Stage 1
    │     │     ├── Task
    │     │     ├── Task
    │     │     └── Task
    │     │
    │     └── Stage 2
    │           ├── Task
    │           ├── Task
    │           └── Task
    │
    └── Job 2
          └── ...

Remember:

An application can contain multiple jobs, a job can contain multiple stages, and a stage can contain multiple tasks.

13. Partitions and Tasks

Partitions are fundamental to Spark’s distributed execution model.

Suppose your dataset contains:

1 billion records

Spark does not necessarily process all 1 billion records as one giant unit.

Instead, the data is divided into partitions:

Partition 1
Partition 2
Partition 3
...
Partition N

Tasks then process these partitions.

Partition → Task

This is why partitioning has such a significant impact on Spark performance.

Too few partitions can limit parallelism.

Too many partitions can introduce unnecessary scheduling and processing overhead.

We’ll explore partitioning in detail in a dedicated tutorial.

14. Narrow and Wide Dependencies

Another important concept in Spark architecture is the relationship between transformations.

Narrow Dependency

A narrow dependency occurs when a child partition depends on a small number of parent partitions, typically without requiring data to be redistributed across the cluster.

Examples include:

filter()
select()
withColumn()

Conceptually:

Partition 1 ─────→ Partition 1
Partition 2 ─────→ Partition 2
Partition 3 ─────→ Partition 3

Wide Dependency

A wide dependency occurs when data must be redistributed between partitions.

Common examples include:

groupBy()
join()
distinct()
orderBy()

Conceptually:

Partition 1 ──┐
Partition 2 ──┼──→ Shuffle ──→ New Partitions
Partition 3 ──┘

Wide dependencies generally introduce shuffle operations and can create stage boundaries.

15. The Role of the DAG

Spark internally represents computation as a Directed Acyclic Graph (DAG).

The DAG represents dependencies between operations.

For example:

df.filter(...)
  .select(...)
  .groupBy(...)
  .count()

can be conceptually represented as:

Read Data
    ↓
Filter
    ↓
Select
    ↓
GroupBy
    ↓
Count

Spark’s scheduler analyzes this dependency graph and determines how the computation should be divided into stages and tasks.

The DAG is therefore a key part of Spark’s execution model.

16. Lazy Evaluation

Spark uses lazy evaluation for transformations.

Consider:

df2 = df.filter(df.salary > 50000)

Spark doesn’t necessarily execute the filtering immediately.

Instead, it records the transformation as part of the computation plan.

Execution is triggered when an action is called.

For example:

df2.show()

or:

df2.count()

This separation between defining transformations and executing them allows Spark to analyze the computation before actually running it.

17. Transformations vs Actions

Spark operations are broadly divided into two categories.

Transformations

Transformations define how data should be transformed.

Examples:

filter()
select()
join()
groupBy()
withColumn()

They generally build the execution plan rather than immediately executing the computation.

Actions

Actions request a result.

Examples:

count()
show()
collect()
first()

Writing data is also an execution-triggering operation:

df.write.parquet("/output")

A useful mental model is:

Transformations
      ↓
Execution Plan
      ↓
Action
      ↓
Execution

18. A Complete Example

Consider:

df = spark.read.parquet("/data/orders")
result = (
    df.filter(df.amount > 1000)
      .groupBy("customer_id")
      .sum("amount")
)
result.show()

Let’s break it down.

Step 1: Read

spark.read.parquet(...)

Spark creates a logical representation of the input.

Step 2: Filter

filter(...)

Spark records the filtering operation.

Step 3: Group

groupBy(...)

Spark identifies that the grouping may require data redistribution.

Step 4: Aggregation

sum(...)

Spark adds the aggregation to the execution plan.

Step 5: Action

show()

The action triggers execution.

Step 6: Job

Spark creates a job.

Step 7: Stages

Spark divides the job around dependency boundaries.

Step 8: Tasks

Each stage generates tasks based on its partitions.

Step 9: Executors

Executors execute those tasks.

The simplified execution flow becomes:

User Code
    │
    ▼
Driver
    │
    ▼
Execution Plan
    │
    ▼
Job
    │
    ▼
Stages
    │
    ▼
Tasks
    │
    ▼
Executors
    │
    ▼
Result

19. What Happens During a Shuffle?

Shuffle is one of the most important concepts to understand when working with Spark.

Suppose we execute:

df.groupBy("customer_id").count()

Imagine the data is distributed like this:

Partition 1
Customer A
Customer B
Partition 2
Customer A
Customer C
Partition 3
Customer B
Customer C

To calculate a complete count for each customer, Spark needs all records for the same customer to be brought together.

Conceptually:

Partition 1 ──┐
Partition 2 ──┼──→ Shuffle ──→ Customer-based partitions
Partition 3 ──┘

This redistribution of data is expensive because it involves network and disk I/O.

We’ll explore shuffle in much greater detail later in this series.

20. Driver Memory vs Executor Memory

Another important distinction is memory.

Driver Memory

Used primarily for:

• Application coordination
• Execution planning
• Metadata
• Small result collections

Executor Memory

Used for:

• Processing data
• Intermediate computations
• Cached data
• Shuffle-related operations

This distinction explains why the following can be dangerous:

df.collect()

collect() brings the entire result back to the Driver.

If the result is extremely large, the Driver can run out of memory.

For large datasets, operations should generally remain distributed rather than unnecessarily collecting data to the Driver.

21. What Happens When an Executor Fails?

Distributed systems must assume that machines can fail.

Spark is designed to recover from certain failures by recomputing lost work using lineage and the execution plan.

For example:

Executor 1
    ↓
Task 1
Task 2
Task 3
Executor fails
    ↓
Task 2 needs to be recomputed

The Driver can reschedule the failed task on available resources.

This is an important part of Spark’s distributed execution model.

22. Spark Architecture in Databricks

If you’re learning Databricks, the architecture becomes even more relevant.

A simplified Databricks environment looks like:

              Databricks Workspace
                      │
                      ▼
                 Spark Driver
                      │
             ┌────────┴────────┐
             │                 │
             ▼                 ▼
        Executor 1         Executor 2
             │                 │
             └────────┬────────┘
                      ▼
                  Data Lake
               Delta / Parquet

Databricks manages much of the underlying infrastructure, but the fundamental Spark concepts remain:

• Driver
• Executors
• Tasks
• Stages
• Jobs
• Partitions
• Shuffle

Understanding these concepts makes debugging Databricks workloads significantly easier.

23. Spark UI

One of the most useful tools for understanding Spark execution is the Spark UI.

It provides information about:

• Jobs
• Stages
• Tasks
• Execution time
• Shuffle read
• Shuffle write
• Input data
• Output data
• Executor activity
• Failed tasks

A simplified workflow when debugging a slow pipeline is:

Slow Pipeline
     ↓
Spark UI
     ↓
Identify Slow Job
     ↓
Identify Slow Stage
     ↓
Identify Slow Task
     ↓
Analyze Shuffle / Skew / Partitions
     ↓
Optimize

This is one of the most important skills for a production Data Engineer.

24. Common Spark Architecture Misconceptions

Misconception 1: Driver processes all the data

Incorrect.

Executors perform distributed data processing.

Misconception 2: Every transformation immediately executes

Incorrect.

Spark transformations are generally evaluated lazily.

Misconception 3: Every transformation creates a new stage

Incorrect.

Stages are determined by dependencies, particularly shuffle boundaries.

Misconception 4: One task processes the entire dataset

Incorrect.

Tasks generally operate on individual partitions.

Misconception 5: More partitions always mean better performance

Incorrect.

Too few partitions can reduce parallelism, while excessive partitioning can create overhead.

25. The Mental Model You Should Remember

If you’re preparing for a Data Engineering interview, remember this flow:

                 Spark Application
                        │
                        ▼
                     Driver
                        │
                Creates Execution Plan
                        │
                        ▼
                      Job
                        │
                 ┌──────┴──────┐
                 ▼             ▼
              Stage 1       Stage 2
                 │             │
                 ▼             ▼
               Tasks         Tasks
                 │             │
                 └──────┬──────┘
                        ▼
                    Executors
                        │
                        ▼
                     Result

And remember the hierarchy:

Application → Job → Stage → Task

This single sequence answers a surprisingly large number of Spark interview questions.

26. Spark Architecture Interview Questions

Once you understand the architecture, you should be able to answer questions such as:

Beginner

What is the Driver?

The Driver coordinates the Spark application and schedules work.

What is an Executor?

An Executor is a process that runs tasks and performs distributed computation.

What is a Cluster Manager?

It allocates resources required by Spark applications.

Intermediate

What is a Spark Job?

A computation triggered by an action.

What is a Stage?

A set of tasks that can execute together without a shuffle boundary between them.

What is a Task?

The smallest unit of work scheduled by Spark, typically operating on a partition.

Advanced

What causes a stage boundary?

Typically a wide dependency requiring shuffle.

Why is shuffle expensive?

Because data may need to be redistributed across executors, involving network and potentially disk I/O.

How would you investigate a slow Spark job?

Start with the Spark UI, identify the slow stage and tasks, then investigate shuffle, skew, partition sizes, input/output, and resource utilization.

Conclusion

Spark’s architecture is built around a simple but powerful idea:

The Driver coordinates distributed computation while Executors perform the actual work on data partitions.

A Spark application is broken down into:

Application
     ↓
Jobs
     ↓
Stages
     ↓
Tasks

The Driver creates and coordinates this execution, the Cluster Manager provides resources, and Executors execute tasks.

Once you understand this model, many seemingly complicated Spark concepts become much easier:

• Lazy evaluation
• DAGs
• Transformations
• Actions
• Partitions
• Shuffle
• Narrow and wide dependencies
• Data skew
• Caching
• Performance tuning
• Spark UI

And this is exactly why understanding architecture should come before trying to optimize Spark jobs.

What’s Next?

Now that we understand the architecture, the next step is to understand how Spark actually represents and manipulates data.

In the next tutorial, we’ll explore:

RDD vs DataFrame vs Dataset — What’s the Difference?

We’ll understand how these three abstractions evolved, when each should be used, and why modern Spark applications primarily rely on DataFrames and Spark SQL.