When you run a PySpark or Spark application, your code does not execute on a single machine. Spark divides the work across a cluster—but two components play very different roles: the Driver coordinates everything, while Executors perform the actual computation.
If you are learning Apache Spark, understanding the difference between the Driver and Executors is one of the most important concepts.
Consider this simple PySpark code:
df = spark.read.parquet("/data/transactions")result = ( df.filter("amount > 1000") .groupBy("customer_id") .sum("amount"))result.show()
It looks like normal Python code.
But internally, Spark may process billions of records across multiple machines.
So what happens behind the scenes?
Who decides how the work should be executed?
Who processes the actual data?
The answer involves two key components:
- Driver
- Executors
In This Series: Apache Spark Learning Path
┌──────────────────────────────────────────────────────────┐│ APACHE SPARK LEARNING PATH ││ ││ ✓ 1. What is Apache Spark? ││ ✓ 2. Why Spark is Faster than Hadoop ││ ✓ 3. Spark Architecture ││ → 4. Driver vs Executor ││ ○ 5. Cluster Managers ││ ○ 6. RDD vs DataFrame vs Dataset ││ ○ 7. SparkSession Explained ││ ○ 8. Lazy Evaluation ││ ○ 9. DAG in Spark ││ ○ 10. Stages and Tasks ││ ││ [View All Apache Spark Tutorials →] │└──────────────────────────────────────────────────────────┘
← Back to Apache Spark Tutorials
What Is the Spark Driver?
The Driver is the central coordinator of a Spark application.
When you submit a Spark application, the Driver is responsible for understanding your code, creating an execution plan, and coordinating the work across the cluster.
Think of the Driver as the brain of the Spark application.
Its major responsibilities include:
- Creating the Spark application
- Maintaining application metadata
- Building the logical execution plan
- Creating the DAG
- Dividing work into jobs, stages, and tasks
- Scheduling tasks
- Sending tasks to Executors
- Tracking task execution
- Collecting results when required
Conceptually:
Spark Application
Driver
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
Executor 1 Executor 2 Executor 3
│ │ │
▼ ▼ ▼
Tasks Tasks Tasks
The Driver generally does not process the entire dataset itself.
Instead, it coordinates distributed computation.
What Are Spark Executors?
Executors are worker processes responsible for actually executing Spark tasks.
They process the partitions of your data.
For example, imagine you have a dataset containing:
1 billion transaction records
Spark may divide the dataset into multiple partitions:
1 Billion Records │ ▼ ┌──────┼──────┬──────┬──────┐ ▼ ▼ ▼ ▼ ▼Part 1 Part 2 Part 3 Part 4 ...
These partitions can then be processed by different Executors.
Driver
│
┌────────────┼────────────┐
▼ ▼ ▼
Executor 1 Executor 2 Executor 3
│ │ │
▼ ▼ ▼
Partitions Partitions Partitions
1, 2, 3 4, 5, 6 7, 8, 9
The Executors are responsible for:
- Running Spark tasks
- Processing data partitions
- Performing transformations
- Participating in shuffle operations
- Storing cached or persisted data
- Returning task results or status information to the Driver
In simple terms:
The Driver decides what needs to be done. Executors do the actual work.
Driver vs Executor: The Core Difference
| Component | Driver | Executor |
|---|---|---|
| Primary role | Coordinates the application | Executes the computation |
| Runs tasks? | Schedules tasks | Yes |
| Processes partitions? | Usually coordinates processing | Yes |
| Creates execution plan? | Yes | No |
| Builds DAG? | Yes | No |
| Stores cached data? | No, not as distributed executor storage | Yes |
| Communicates with | Cluster Manager and Executors | Driver |
| Number per application | Usually one Driver | Usually multiple Executors |
The easiest way to remember this is:
Driver ↓Plans the workExecutors ↓Perform the work
How Driver and Executors Work Together
Let’s understand this using a simple example.
transactions = spark.read.parquet("/data/transactions")result = ( transactions .filter("amount > 1000") .groupBy("customer_id") .sum("amount"))result.show()
At first glance, this looks like a normal sequence of Python operations.
Internally, something different happens.
Step 1: Your Application Starts
When you submit your Spark application:
spark-submit │ ▼ Driver Starts
The Driver creates the Spark application context and begins coordinating the application.
Step 2: Spark Understands Your Transformations
When Spark sees:
transactions.filter("amount > 1000")
it does not necessarily process the data immediately.
Instead, Spark records the transformation.
Then:
.groupBy("customer_id")
Spark adds another transformation to the execution plan.
Then:
.sum("amount")
adds additional processing requirements.
Spark builds a logical representation of what needs to happen.
This is part of Spark’s lazy evaluation model.
Read Data ↓Filter ↓Group By ↓Aggregation
The Driver is responsible for coordinating this execution planning process.
You Might Need This Next: Lazy Evaluation
Spark transformations such as:
filter()select()withColumn()groupBy()
do not always execute immediately.
Spark first builds an execution plan and waits until an action is triggered.
Learn more about:
→ Lazy Evaluation
→ Transformations vs Actions
→ DAG
→ Query Optimization
Read: Lazy Evaluation in Apache Spark →
Step 3: An Action Triggers Execution
Now consider:
result.show()
This is an action.
At this point, Spark needs to actually process the data.
Conceptually:
TransformationsRead ↓Filter ↓GroupBy ↓Aggregation │ │ No execution yet ▼ show() │ ▼ Execution Starts
The Driver now coordinates the execution process.
Step 4: The Driver Creates Tasks
The Driver breaks the work into smaller units.
A simplified example:
Spark Job │ ├── Stage 1 │ ├── Task 1 │ ├── Task 2 │ └── Task 3 │ └── Stage 2 ├── Task 4 ├── Task 5 └── Task 6
These tasks are distributed to Executors.
Driver
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
Executor 1 Executor 2 Executor 3
│ │ │
Tasks 1,2 Tasks 3,4 Tasks 5,6
The Executors then process their assigned partitions.
Step 5: Executors Process the Data
Suppose your dataset is divided into 100 partitions.
The Driver can schedule tasks for those partitions across available Executors.
For example:
100 Data PartitionsExecutor 1 → Partitions 1–25Executor 2 → Partitions 26–50Executor 3 → Partitions 51–75Executor 4 → Partitions 76–100
The actual distribution depends on:
- Number of partitions
- Available Executors
- Available CPU cores
- Task scheduling
- Cluster configuration
Each Executor processes tasks in parallel.
This distributed execution is one of the main reasons Spark can handle very large datasets.
What Happens During a Shuffle?
Let’s consider this code:
df.groupBy("customer_id").count()
A groupBy often requires Spark to redistribute data.
Suppose the data initially looks like this:
Executor 1Customer ACustomer BCustomer CExecutor 2Customer ACustomer DCustomer E
To calculate the total count for each customer, all records for the same customer may need to be brought together.
Conceptually:
Before ShuffleExecutor 1 Executor 2A, B, C A, D, EAfter ShuffleExecutor 1 Executor 2A, A, B C, D, E
The Executors exchange data across the network.
The Driver coordinates the execution plan, while Executors perform the actual shuffle and computation.
This distinction is important:
The Driver coordinates the work. Executors process and exchange the distributed data.
💡 Going Deeper: Why Data Skew Can Affect Executors
Imagine you have transaction data where one customer accounts for 30% of all records.
Customer A → 30% of dataCustomer B → 1%Customer C → 1%Customer D → 1%...
After a shuffle operation such as:
df.groupBy("customer_id").count()
one Executor may receive significantly more data than the others.
Executor 1 → 10 GBExecutor 2 → 12 GBExecutor 3 → 11 GBExecutor 4 → 500 GB ← Skewed partition
Executor 4 becomes the bottleneck.
The other Executors may finish quickly while one Executor continues processing.
This is known as data skew.
Later in this series, you’ll learn about:
- Shuffle
- Data skew
- Salting
- Adaptive Query Execution
- Broadcast joins
Executor Memory
Executors require memory to process data.
For example:
Executor Memory = 16 GBExecutor Cores = 4
This means an Executor can use its allocated resources to process multiple tasks.
However, allocating extremely large memory to an Executor is not always the best configuration.
Problems can include:
- Long garbage collection pauses
- Memory pressure
- Out-of-memory errors
- Inefficient resource usage
Similarly, using too many small Executors can create unnecessary scheduling overhead.
The goal is to balance:
- Number of Executors
- Number of cores
- Memory per Executor
- Number of partitions
⚡ Performance Tip: Driver Memory Also Matters
Many developers focus only on Executor memory.
But the Driver can also become a bottleneck.
For example:
df.collect()
This brings data from the distributed Executors back to the Driver.
If the dataset is large:
Executors │ │ 500 GB of data ▼Driver
The Driver may run out of memory.
This can result in errors such as:
OutOfMemoryError
The same issue can occur with operations like:
df.toPandas()
if the resulting dataset is too large for the Driver’s memory.
A safer approach is often to inspect only a small sample:
df.show(20)
or:
df.limit(100).toPandas()
You Might Need This Next: Spark Memory and Performance
When Spark jobs become slow or fail, simply increasing cluster size is not always the answer.
The issue could be related to:
→ Executor memory
→ Number of partitions
→ Shuffle
→ Data skew
→ Large joins
→ Driver memory
Explore the Apache Spark Learning Path →
Can One Spark Application Have Multiple Drivers?
Normally, a Spark application has one Driver.
Spark Application │ ▼ Driver │ ┌─────┼─────┐ ▼ ▼ ▼E1 E2 E3
The Driver acts as the central coordinator.
If the Driver fails, the application is generally affected or terminated unless the environment and deployment architecture provide mechanisms to restart or recover the application.
Executors, on the other hand, may be replaced depending on the cluster manager and application configuration.
Can Executors Be Added or Removed?
Yes, depending on the cluster configuration.
With dynamic resource allocation or autoscaling environments, the number of Executors can change based on workload requirements.
For example:
Low WorkloadDriver │ ├── Executor 1 └── Executor 2
During a larger workload:
Driver │ ├── Executor 1 ├── Executor 2 ├── Executor 3 ├── Executor 4 ├── Executor 5 └── Executor 6
When the workload decreases, resources may be reduced again.
This can help optimize infrastructure utilization.
Driver vs Executor in Databricks
When using Databricks, you typically see a cluster consisting of:
Databricks Cluster
Driver Node
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
Worker 1 Worker 2 Worker 3
│ │ │
Executor Executor Executor
The Driver node coordinates the Spark application.
The Worker nodes provide resources where Spark Executors run and execute distributed tasks.
When you run a notebook command, Spark can translate the relevant distributed operation into work that is executed across the available cluster.
Common Driver vs Executor Interview Questions
1. What is the role of the Driver in Spark?
The Driver is responsible for coordinating the Spark application. It creates execution plans, schedules tasks, communicates with Executors, and tracks application execution.
2. What is the role of an Executor?
An Executor is a process responsible for executing Spark tasks, processing data partitions, storing cached data, and participating in distributed operations such as shuffles.
3. Does the Driver process all the data?
No.
The Driver primarily coordinates execution. Executors process the distributed data.
However, operations such as collect() can bring results back to the Driver, potentially causing memory issues if the result is too large.
4. What happens if an Executor fails?
The Driver and underlying cluster environment can detect task or executor failures, and Spark may retry failed tasks according to its fault-tolerance mechanisms and configuration.
The exact behavior can depend on the deployment environment and cluster manager.
5. What happens if the Driver fails?
The Spark application is generally interrupted because the Driver is responsible for coordinating its execution. Recovery behavior depends on the deployment mode and platform configuration.
6. Can multiple tasks run on an Executor?
Yes.
The number of tasks that can run concurrently depends primarily on the resources allocated to the Executor, including available CPU cores.
Common Mistakes
Mistake 1: Thinking the Driver processes the entire dataset
The Driver coordinates the application.
Executors process distributed data.
Mistake 2: Using collect() on a massive DataFrame
This can move a large amount of data to the Driver and cause memory problems.
Avoid:
df.collect()
unless you are certain that the result is small enough.
Mistake 3: Assuming More Executors Always Means Better Performance
More Executors can help, but performance can still suffer because of:
- Data skew
- Large shuffle operations
- Poor partitioning
- Inefficient joins
- Too many small files
Cluster resources and application design both matter.
Mistake 4: Confusing Worker Nodes with Executors
A Worker Node is the machine or compute resource.
An Executor is the process that runs Spark tasks using resources allocated on that worker.
The exact relationship can vary based on deployment and configuration.
Driver vs Executor: A Complete Example
Let’s combine everything.
Suppose you run:
df = spark.read.parquet("/data/sales")result = ( df.filter("amount > 100") .groupBy("country") .agg({"amount": "sum"}))result.show()
A simplified execution flow looks like this:
1. Application Submitted │ ▼2. Driver Starts │ ▼3. Spark Builds Execution Plan │ ▼4. Action: show() │ ▼5. Driver Creates Job │ ▼6. Work Is Divided Into Stages │ ▼7. Stages Are Divided Into Tasks │ ▼8. Tasks Sent to Executors │ ▼9. Executors Process Data │ ▼10. Results Returned
The key relationship is:
Driver │ │ Plans and coordinates ▼Executors │ │ Execute tasks ▼Distributed Data
Key Takeaways
The Driver and Executors have very different responsibilities.
Driver
- Coordinates the Spark application
- Creates execution plans
- Builds the DAG
- Creates jobs, stages, and tasks
- Schedules tasks
- Tracks execution
Executors
- Execute tasks
- Process data partitions
- Perform transformations
- Participate in shuffles
- Store cached or persisted data
The simplest way to remember the difference is:
The Driver is the brain. Executors are the workers.
Continue Learning Apache Spark
You now understand how Spark divides responsibility between the Driver and Executors.
But there is another important question:
Who provides the machines and resources needed to launch those Executors?
The answer is the Cluster Manager.
In the next tutorial, you’ll learn:
- What a Cluster Manager is
- How Spark requests resources
- Spark Standalone
- Hadoop YARN
- Kubernetes
- How managed platforms such as Databricks handle cluster resources
Next Step
You’ve learned how the Driver coordinates Spark applications and how Executors process data across the cluster.
Next, let’s understand who manages the resources required to run those Executors.
→ Continue to Cluster Managers in Apache Spark: How Spark Gets and Manages Computing Resources
You can also return to the Complete Apache Spark Tutorials Learning Path to explore the full logical sequence.
2 thoughts on “Driver vs Executor in Apache Spark: Understanding How Spark Applications Actually Run”