Data Engineering interviews have evolved significantly. Knowing SQL and writing basic PySpark transformations is no longer enough for many modern Data Engineer roles.

If you’re interviewing for a Databricks, Spark, PySpark, or Azure Data Engineer position, interviewers often want to know whether you understand what happens under the hood and whether you can troubleshoot real production problems.

Based on common interview discussions, here are 20 important questions covering Spark, PySpark, Databricks, Delta Lake, performance optimization, and migration.

1. Explain the Architecture of Apache Spark

A typical Spark application consists of:

• Driver
• Cluster Manager
• Executors
• Tasks

The high-level flow is:

    User Application
           |
           v
         Driver
           |
           v
     Cluster Manager
           |
           v
  +---------+---------+
  |                   |
Executor 1         Executor 2
  |                   |
Tasks               Tasks

The Driver runs the main application and coordinates execution.

The Cluster Manager allocates resources. Depending on the environment, this could be Spark Standalone, YARN, Kubernetes, or a cloud-managed environment.

The Executors run tasks and store cached data.

The Driver creates the execution plan and schedules work across executors.

Interview Tip

Don’t stop at:

“The Driver sends tasks to Executors.”

A stronger answer should also explain:

Application
    ↓
  Job
    ↓
  Stages
    ↓
  Tasks
    ↓
Executors

Understanding this hierarchy is essential for debugging Spark jobs.

2. What Is Lazy Evaluation in Spark?

Spark uses lazy evaluation.

Transformations such as:

df.filter(...)
df.select(...)
df.withColumn(...)

do not immediately execute the computation.

Instead, Spark builds an execution plan.

Execution starts when an action is called.

Examples of actions include:

df.show()
df.count()
df.collect()
df.write.parquet(...)

For example:

result = (
    df
    .filter("age > 30")
    .select("name", "age")
)
result.show()

The filter() and select() operations are transformations.

The actual computation is triggered by:

show()

Why is lazy evaluation useful?

It allows Spark to optimize the execution plan before running the computation.

3. What Is a DAG in Spark?

DAG stands for Directed Acyclic Graph.

Spark creates a logical representation of transformations and their dependencies.

For example:

df.filter(...)
  .select(...)
  .groupBy(...)
  .count()

Spark doesn’t simply execute each statement independently.

It builds an execution plan and optimizes it before execution.

A simplified flow is:

Transformations
      ↓
Logical Plan
      ↓
Optimization
      ↓
Physical Plan
      ↓
    Stages
      ↓
    Tasks
      ↓
  Executors

A common mistake is saying:

“DAG combines multiple filters.”

That’s incomplete.

The important idea is that Spark uses the DAG to understand dependencies and construct an efficient execution plan.

4. Are Multiple Transformations Executed as Separate Stages?

Consider:

df.filter(...)
  .filter(...)
  .select(...)

These are generally narrow transformations.

Spark can pipeline compatible narrow transformations within the same stage.

Conceptually:

Data Source
    ↓
Filter 1
    ↓
Filter 2
    ↓
Select
    ↓
Same Stage

However, an operation such as:

df.groupBy("customer_id").count()

typically introduces a shuffle.

This can create a stage boundary.

Stage 1
   ↓
Shuffle
   ↓
Stage 2

Key takeaway

Not every transformation creates a new stage.

Shuffle boundaries are a major factor in stage creation.

5. What Is the Difference Between Narrow and Wide Transformations?

Narrow Transformation

A narrow transformation does not require data to be redistributed across partitions.

Examples:

filter()
select()
withColumn()

Data can be processed largely within the existing partitions.

Wide Transformation

A wide transformation requires data movement between partitions.

Examples:

groupBy()
join()
distinct()
orderBy()

These operations can cause a shuffle.

A simplified example:

Narrow:
Partition 1 → Filter → Select
Wide:
Partition 1 ─┐
Partition 2 ─┼── Shuffle ──> New Partitions
Partition 3 ─┘

Understanding narrow vs. wide transformations is critical for Spark performance tuning.

6. What Is a Shuffle?

A shuffle occurs when Spark needs to redistribute data across partitions.

For example:

df.groupBy("customer_id").count()

If records belonging to the same customer exist in different partitions, Spark must move them so that the same keys can be processed together.

Shuffle is expensive because it can involve:

• Network I/O
• Disk I/O
• Serialization
• Memory usage

Therefore, excessive shuffling can significantly slow down Spark jobs.

7. What Is Data Skew?

Data skew occurs when data is distributed unevenly across partitions.

Suppose:

Customer 100 → 500 million transactions
Other customers → 100 transactions each

During a join or aggregation, one partition may receive a disproportionate amount of data.

You might see:

Task 1 → 1 minute
Task 2 → 1 minute
Task 3 → 1 minute
Task 4 → 90 minutes

This is a classic symptom of data skew.

The entire stage must wait for the slow task.

8. What Is Salting and When Do We Use It?

Salting is a technique commonly used to mitigate data skew.

Suppose:

transactions.join(customers, "customer_id")

and customer_id = 100 is extremely frequent.

We can add an artificial salt key to distribute the skewed records.

Conceptually:

customer_id = 100
100_0 → Partition 1
100_1 → Partition 2
100_2 → Partition 3
100_3 → Partition 4

The corresponding records in the smaller table must also be replicated across the salt values.

The join then becomes:

customer_id + salt

Important

Salting is not the same as repartitioning.

• Repartitioning redistributes data across partitions.
• Salting changes the join key to distribute skewed values.
• Bucketing organizes data into predefined buckets based on a key.

9. Can AQE Solve Data Skew?

AQE stands for Adaptive Query Execution.

AQE allows Spark to adapt the physical execution plan based on runtime statistics.

It can help with:

• Skewed joins
• Coalescing small shuffle partitions
• Dynamically changing join strategies

For example:

spark.conf.set(
    "spark.sql.adaptive.enabled",
    "true"
)

AQE can reduce the need for manual optimization in some scenarios.

However, AQE and salting are not identical.

Salting
→ Manual technique
→ Modify the join key
→ Distribute skewed records
AQE
→ Runtime optimization
→ Spark adapts the execution plan

A strong candidate should know when to use each approach.

10. What Is the Difference Between Repartition and Coalesce?

Repartition

df.repartition(100)

Redistributes data across partitions and generally causes a shuffle.

You can also repartition by a key:

df.repartition("customer_id")

Coalesce

df.coalesce(10)

is commonly used to reduce the number of partitions with less data movement than a full repartition.

Interview Tip

Never say:

“Use repartition to optimize every Spark job.”

Repartition itself can be expensive.

The correct approach is to understand the problem first.

11. What Is the Difference Between Cache and Persist?

Spark supports caching intermediate results.

For example:

df.cache()

Caching is useful when the same DataFrame is reused multiple times.

However, caching is lazy.

Calling:

df.cache()

doesn’t immediately load the data into memory.

An action is required:

df.cache()
df.count()

After materialization, subsequent operations may reuse the cached data.

persist() allows you to specify a storage level.

For example:

df.persist()

Important

Don’t cache everything.

Caching can consume memory and may actually hurt performance if the data is used only once.

12. How Would You Troubleshoot a 1 TB Spark Job Taking 3 Hours?

A weak answer is:

“Restart the cluster.”

Restarting may temporarily remove a resource problem, but it doesn’t identify the root cause.

A better troubleshooting process is:

Step 1: Check Spark UI

Look for:

• Long-running stages
• Slow tasks
• Shuffle read/write
• Task distribution
• Spill to disk
• Executor failures

Step 2: Check for Data Skew

If one task is much slower than others, investigate skew.

Step 3: Check Shuffle

Look for expensive:

groupBy()
join()
distinct()
orderBy()

Step 4: Check Join Strategy

Consider:

• Broadcast join
• Sort-merge join
• AQE

Step 5: Check Partitioning

Look for:

• Too many partitions
• Too few partitions
• Uneven partition sizes

Step 6: Check Data Access

Use:

• Predicate pushdown
• Column pruning
• Partition pruning

Step 7: Check Delta Optimization

For Delta tables, investigate:

• Small files
• File compaction
• Data layout
• Appropriate clustering strategy

The key is:

Measure first. Optimize second.

13. What Is a Broadcast Join?

A broadcast join is useful when one dataset is sufficiently small.

For example:

from pyspark.sql.functions import broadcast
result = large_df.join(
    broadcast(small_df),
    "customer_id"
)

Spark can distribute the smaller dataset to executors instead of performing a large shuffle join.

However, don’t blindly broadcast tables.

If the table is too large, it can cause memory pressure or executor failures.

14. How Do You Find the Top 3 Highest-Paid Employees in Each Department?

Suppose the data contains:

employee_id
department
salary
joining_date

Use a Window function:

from pyspark.sql import functions as F
from pyspark.sql.window import Window
window_spec = (
    Window
    .partitionBy("department")
    .orderBy(
        F.col("salary").desc(),
        F.col("joining_date").asc()
    )
)
result = (
    df
    .withColumn(
        "rn",
        F.row_number().over(window_spec)
    )
    .filter(F.col("rn") <= 3)
)

Here:

• Highest salary gets priority.
• If salaries are equal, the employee who joined earlier gets priority.
• Only three employees are returned per department.

15. What Is the Difference Between row_number, rank, and dense_rank?

Consider:

Salary
100
100
90
80

row_number()

1
2
3
4

Every row receives a unique number.

rank()

1
1
3
4

Ranks are skipped after ties.

dense_rank()

1
1
2
3

Ranks are not skipped.

Interview Tip

If the requirement says:

“Return exactly 3 employees per department, with earlier joining date breaking salary ties.”

Use:

row_number()

with:

salary DESC,
joining_date ASC

16. How Do You Validate a Databricks Migration?

Suppose you migrate a pipeline from SQL/SAS to Databricks.

A robust migration validation strategy can involve:

Legacy Pipeline
      |
      |----> Output A
      |
      |
New Databricks Pipeline
      |
      |----> Output B

Compare:

• Row counts
• Null counts
• Duplicate counts
• Aggregations
• Min/max values
• Business metrics
• Record-level differences
• Hash/checksum comparisons

For continuously changing data, run the legacy and new pipelines in parallel for a defined validation period.

For example:

Day 1 → Compare
Day 2 → Compare
...
Day 10 → Compare
        ↓
   Cutover

This helps ensure that the new pipeline consistently produces the expected results before decommissioning the legacy system.

17. How Do You Handle Incremental Loads?

A common approach is to identify records that have changed since the previous run.

For example:

last_updated > previous_watermark

Other approaches include:

• Change Data Capture (CDC)
• Change Tracking
• Timestamp-based incremental loading
• Version columns

In Delta Lake, incremental updates can be handled using MERGE.

Conceptually:

MERGE INTO target
USING source
ON target.customer_id = source.customer_id
WHEN MATCHED THEN
  UPDATE SET *
WHEN NOT MATCHED THEN
  INSERT *

The exact implementation depends on the source system and business requirements.

18. How Would You Design a Databricks ETL Pipeline?

A typical architecture might look like:

Source Systems
     |
     v
Azure Data Factory
     |
     v
ADLS
     |
     v
Bronze / Raw
     |
     v
Silver / Trusted
     |
     v
Gold / Unified
     |
     +--------> Snowflake
     |
     +--------> BI
     |
     +--------> APIs

ADF can be used for orchestration.

Databricks handles:

• Data transformations
• Spark processing
• Delta Lake
• Data quality
• Business logic

ADLS provides cloud storage.

The exact architecture depends on organizational requirements.

19. How Do You Monitor Production Pipelines?

A production data platform should have:

• Pipeline monitoring
• Failure alerts
• Retry mechanisms
• Logging
• Data quality checks
• SLA monitoring

In Azure environments, tools such as Azure Monitor and Log Analytics can help centralize monitoring and diagnostics.

For example:

ADF Pipeline
      |
      v
Failure
      |
      v
Monitoring
      |
      v
Alert
      |
      v
Email / Teams / Incident System

Monitoring successful pipeline completion can also be useful when downstream consumers depend on the pipeline’s SLA.

20. What Makes a Strong Databricks Data Engineer?

A strong Data Engineer should be able to connect all these concepts:

Data Sources
     ↓
Ingestion
     ↓
ADF / Orchestration
     ↓
ADLS
     ↓
Databricks
     ↓
PySpark
     ↓
Delta Lake
     ↓
Transformations
     ↓
Optimization
     ↓
Data Quality
     ↓
Monitoring
     ↓
Consumption

But that’s not enough.

A strong candidate should also understand why a particular design is chosen.

For example:

• Why use a broadcast join?
• When does a shuffle happen?
• Why is a task slow?
• How do you detect data skew?
• When is salting appropriate?
• Can AQE solve the problem?
• When should you cache?
• How do you validate a migration?
• How do you make a pipeline incremental?
• How do you troubleshoot a production failure?

Final Thoughts

The biggest lesson from Databricks and PySpark interviews is that knowing terminology is not the same as knowing Spark.

A candidate might say:

“I know partitioning.”

The interviewer should ask:

“How would you troubleshoot a skewed partition?”

A candidate might say:

“I know salting.”

Ask:

“Write the code.”

A candidate might say:

“I know Databricks.”

Ask:

“Explain one pipeline you personally built from source to consumption.”

The strongest candidates can move between concepts, code, architecture, and production troubleshooting.

If you’re preparing for a Data Engineering interview, focus on these four levels:

Level 1 → Concepts
Level 2 → Coding
Level 3 → Architecture
Level 4 → Production Troubleshooting

Mastering all four is what separates someone who has read about Spark from someone who can actually build and operate Spark-based data platforms.

This is part 1 of our series, Subscribe to email notification for other parts as well and follow me on medium