Part 5 moves beyond Spark API knowledge into the decisions that actually matter when a PySpark pipeline is slow, unstable, expensive, or producing incorrect results.

If you’re preparing for PySpark interviews or working with Spark in production, follow me for practical data engineering and AI/ML content.

📩 Subscribe by email to get new tutorials and interview questions delivered straight to your inbox.

Now, let’s get into 20 PySpark questions that test what you actually know about Spark in production.

Most PySpark candidates can answer:

“What is a DataFrame?”

Far fewer can answer:

“Your Spark job normally finishes in 20 minutes, but once a week it takes 3 hours. The cluster has plenty of resources. How would you investigate?”

That difference matters.

Production Spark engineering is rarely about remembering functions. It is about understanding execution, data distribution, failure modes, and trade-offs.

Here are 20 questions designed to test exactly that.

1. Your Spark job suddenly became 5× slower. What would you check first?

Don’t immediately increase the cluster size.

Start by determining where the time is being spent.

I would investigate:

1. Spark UI → Jobs
2. Identify the slow stage
3. Check task duration distribution
4. Look at shuffle read/write
5. Check for data skew
6. Check input size changes
7. Check number of partitions
8. Inspect the physical plan
9. Look for spills
10. Check executor failures or GC time

A useful mental model is:

Job Slow
   ↓
Which Stage?
   ↓
Which Tasks?
   ↓
Why are those Tasks Slow?
   ├── Skew
   ├── Shuffle
   ├── Spill
   ├── Too much data
   ├── Bad partitioning
   ├── GC
   └── Resource contention

Production insight:
“Add more executors” is an infrastructure response, not a diagnosis.

2. One Spark task takes 40 minutes while all other tasks finish in 2 minutes. What is happening?

This is a classic sign of data skew or an unusually expensive partition.

Imagine:

Partition 1 → 2 min
Partition 2 → 2 min
Partition 3 → 2 min
Partition 4 → 40 min  ← Problem
Partition 5 → 2 min

The entire stage may effectively wait for the slow task.

I would check:

• partition sizes
• key distribution
• shuffle statistics
• whether a particular join/grouping key dominates the data

For example:

customer_id = 1001 → 50 million records
customer_id = 1002 → 10,000 records
customer_id = 1003 → 8,000 records

The 1001 partition can become a straggler.

Possible solutions include:

• salting
• better partitioning
• broadcasting the small side of a join
• filtering unnecessary data earlier
• Spark’s adaptive skew-join handling where applicable

3. What is the difference between repartition() and coalesce()?

This is a very common production question.

repartition()

Generally causes a shuffle.

df = df.repartition(100)

It can increase or decrease the number of partitions.

coalesce()

Usually reduces partitions without a full shuffle.

df = df.coalesce(10)

It’s particularly useful when you want to reduce the number of output partitions.

Think:

repartition()
      ↓
Redistribute data
      ↓
Shuffle

versus:

coalesce()
      ↓
Combine existing partitions
      ↓
Usually avoids full shuffle

Production rule: Don’t use repartition() casually because it can introduce a large shuffle.

4. Why can having too many partitions hurt performance?

More partitions don’t automatically mean better performance.

Suppose you have:

10 GB data

and create:

100,000 partitions

Spark may spend significant time managing a huge number of tiny tasks.

You can get:

Too many partitions
       ↓
Many tiny tasks
       ↓
Scheduling overhead
       ↓
Poor performance

On the other hand:

Too few partitions
       ↓
Large partitions
       ↓
Less parallelism
       ↓
Long-running tasks
       ↓
Possible memory pressure

The goal is appropriate partition sizing and parallelism, not simply maximizing the partition count.

5. How would you detect data skew in a Spark job?

I would start with the Spark UI.

Look for:

• one or a few tasks significantly slower than others
• large differences in task input size
• large differences in shuffle read
• unusually large shuffle partitions

I might also inspect the key distribution directly.

For example:

df.groupBy("customer_id").count().orderBy("count", ascending=False).show(20)

If the distribution looks like:

customer_id    count
---------------------
1001           50,000,000
1002               20,000
1003               18,000
1004               15,000

you have a strong skew candidate.

6. When would you use a broadcast join?

Suppose you have:

Large transactions → 500 GB
Small customer table → 20 MB

A normal join may require significant data movement.

A broadcast join can distribute the smaller dataset to executors:

               Small Table
                   ↓
        ┌──────────┼──────────┐
        ↓          ↓          ↓
    Executor 1 Executor 2 Executor 3
        +          +          +
    Large Data  Large Data  Large Data

In PySpark:

from pyspark.sql.functions import broadcast
result = transactions.join(
    broadcast(customers),
    "customer_id"
)

The key idea is:

Replicate the small dataset instead of shuffling the large dataset unnecessarily.

But don’t blindly broadcast large datasets.

Broadcasting something that doesn’t fit comfortably within executor memory can create serious problems.

7. Why is collect() dangerous in production?

Consider:

rows = df.collect()

Spark is distributed:

Executor 1 ─┐
Executor 2 ─┤
Executor 3 ─┼──→ Driver
Executor 4 ─┤
Executor 5 ─┘

collect() brings the complete result to the driver.

If the result is large:

Huge Dataset
     ↓
Network transfer
     ↓
Driver memory
     ↓
OOM

Instead of:

df.collect()

consider whether you really need:

df.show(20)

or:

df.take(20)

or whether the computation should remain distributed.

Interview answer:
collect() is not inherently wrong. It is dangerous when the result isn’t known to be small.

8. Why does groupBy() often cause a shuffle?

Consider:

df.groupBy("department").count()

Suppose data is distributed like this:

Executor 1:
Engineering
Finance
Executor 2:
HR
Engineering
Executor 3:
Finance
HR

To calculate a global count for each department, records with the same key generally need to be brought together.

Conceptually:

Engineering ────────┐
Engineering ────────┼──→ Partition A
                    │
Finance ────────────┼──→ Partition B
Finance ────────────┘

That redistribution is a shuffle.

Shuffle can involve:

• network I/O
• serialization
• disk I/O
• memory pressure
• spill

That’s why operations involving data redistribution deserve particular attention during performance tuning.

9. What is the difference between cache() and persist()?

cache() is essentially a convenient way to request the default persistence behavior.

df.cache()

persist() lets you explicitly choose a storage level.

df.persist()

The important production concept isn’t memorizing the API difference.

It’s understanding why you persist something.

Suppose:

expensive = (
    df
    .join(...)
    .groupBy(...)
    .agg(...)
)

and then:

expensive.count()
expensive.show()
expensive.write.parquet(...)

Without persistence, Spark may need to recompute parts of the lineage for different actions.

Persistence can allow reuse of the computed data.

But:

Don’t cache everything.

Persistence consumes resources and can make a job worse if used unnecessarily.

10. Does calling cache() immediately load the DataFrame into memory?

No.

This:

df.cache()

marks the DataFrame for persistence.

The data is materialized when an action executes.

For example:

df.cache()
df.count()

Conceptually:

cache()
   ↓
Mark for persistence
   ↓
count()
   ↓
Execute
   ↓
Populate cache

This is another example of Spark’s lazy execution model.

11. Why might withColumn() repeatedly be a problem?

Consider:

df = df.withColumn("a", ...)
df = df.withColumn("b", ...)
df = df.withColumn("c", ...)
df = df.withColumn("d", ...)

Each individual withColumn() is not necessarily expensive because it immediately executes the dataset.

The concern is the size and complexity of the resulting logical plan, particularly when hundreds or thousands of sequential projections are created.

A better approach can be to add multiple columns together:

from pyspark.sql import functions as F
df = df.select(
    "*",
    (F.col("x") + 1).alias("a"),
    (F.col("y") * 2).alias("b"),
    F.upper("name").alias("c")
)

The broader production lesson:

Don’t confuse lazy transformations with zero cost. Very large plans can themselves become a problem.

12. Why should you prefer built-in Spark functions over Python UDFs?

Suppose you need to transform a column.

A Python UDF can look like:

from pyspark.sql.functions import udf
@udf
def clean_name(name):
    return name.strip().lower()

But when Spark has an equivalent built-in function:

from pyspark.sql import functions as F
df = df.withColumn(
    "clean_name",
    F.lower(F.trim("name"))
)

the built-in expression is usually preferable.

Why?

Spark understands built-in expressions and can optimize them as part of its query plan.

Python UDFs can introduce additional serialization and execution overhead and can limit some optimizer opportunities.

Production rule:

Built-in Spark function
        ↓
Preferred
Python UDF
        ↓
Use when necessary

13. How would you investigate a slow join?

I would not start by changing the join syntax.

First I would ask:

1. How large are both datasets?

Left → 500 GB
Right → 20 MB

This may suggest a broadcast strategy.

2. Is the join key skewed?

customer_id = 1 → 100M records

3. Is Spark shuffling both sides?

Inspect:

df.explain("formatted")

and the Spark UI.

4. Are unnecessary columns being carried?

Select only what is needed.

5. Can filtering happen earlier?

filtered = df.filter(...)

before the join when logically valid.

6. Is the data already partitioned appropriately?

The answer depends on the workload and physical plan.

The important point is:

Join optimization starts with understanding the data distribution, not memorizing one magic technique.

14. What is predicate pushdown?

Suppose you write:

df.filter("country = 'US'")

If the underlying data source supports predicate pushdown, Spark may be able to push the filter closer to the data source.

Instead of:

Read everything
     ↓
Filter US

the effective work can become closer to:

Read relevant data
     ↓
Return US records

This can reduce data read from the source.

This is especially useful with columnar formats and data sources that support filtering.

Production insight:
Reducing data at the earliest possible stage can reduce downstream CPU, network traffic, memory usage, and shuffle volume.

15. What is column pruning?

Suppose your table contains:

100 columns

but your query needs:

customer_id
salary
department

A good execution plan may avoid reading unnecessary columns from a columnar data source.

Conceptually:

100 columns
     ↓
Only 3 required
     ↓
Less I/O
     ↓
Less processing

This is one reason selecting only required columns can be valuable in large pipelines.

Instead of carrying:

df.select("*")

through a complex pipeline, consider selecting the required columns when appropriate.

16. What is Adaptive Query Execution (AQE)?

Adaptive Query Execution allows Spark SQL to use runtime statistics to improve certain execution decisions.

Instead of making every decision entirely from estimates before execution, Spark can adapt parts of the physical plan based on observed runtime information.

AQE can help with scenarios such as:

• coalescing post-shuffle partitions
• handling skewed joins
• changing certain join strategies

Conceptually:

Initial Plan
     ↓
Execute
     ↓
Observe Runtime Statistics
     ↓
Adapt Plan
     ↓
Continue Execution

This is particularly important in modern Spark because runtime data can differ significantly from what was estimated beforehand.

17. Your output directory contains 10,000 tiny Parquet files. What’s wrong?

This is the small files problem.

Suppose:

Input
 ↓
Spark
 ↓
10,000 output partitions
 ↓
10,000 tiny files

This can create overhead for:

• metadata management
• file listing
• subsequent reads
• scheduling
• storage systems

One possible solution is to control the number of output partitions.

For example:

df.coalesce(100).write.parquet("output/")

But don’t blindly choose 100.

The appropriate number depends on:

• data volume
• downstream workload
• storage system
• parallelism requirements
• file size targets

Production lesson:
Partition count affects not only compute performance but also the physical layout of your data.

18. What happens if one executor repeatedly crashes?

Don’t simply restart it repeatedly.

Investigate the reason.

Potential causes include:

Executor failure
     ├── OutOfMemoryError
     ├── Excessive GC
     ├── Large broadcast
     ├── Data skew
     ├── Container/resource limits
     ├── Bad UDF behavior
     └── Infrastructure failure

For memory-related problems, inspect:

• executor memory
• memory overhead
• task sizes
• broadcast variables
• cached datasets
• skewed partitions
• spill behavior

The important distinction is:

Increasing memory can hide the symptom without fixing the underlying problem.

19. How would you make a PySpark pipeline idempotent?

Suppose a daily pipeline processes:

2026-09-11

and fails halfway through.

You rerun it.

You don’t want:

duplicate records

or:

partially written output

An idempotent pipeline produces the same correct result when safely rerun for the same input.

Typical strategies depend on the storage architecture:

• partition outputs by processing date
• overwrite the affected partition
• use deterministic transformations
• use merge/upsert semantics where supported
• maintain checkpoints or processing metadata
• avoid blindly appending duplicate results

For example:

/raw/date=2026-09-11/
             ↓
     transformation
             ↓
/processed/date=2026-09-11/

If processing fails, the pipeline can safely reprocess the affected partition according to its write strategy.

Production insight:
A pipeline isn’t production-ready merely because it works on the first successful run. It must also behave correctly when it fails and is retried.

20. A Spark job works on 10 GB but fails on 2 TB. What would you change?

This is the ultimate production-thinking question.

I wouldn’t assume the algorithm is scalable just because it works on small data.

I would examine:

2 TB Dataset
    ↓
Input size
    ↓
Partitioning
    ↓
Shuffle volume
    ↓
Data skew
    ↓
Join strategy
    ↓
Memory / spill
    ↓
Serialization
    ↓
Output file count
    ↓
Cluster utilization

Then I would ask:

Can I reduce the data earlier?

df.filter(...)

Can I read fewer columns?

df.select(...)

Can I avoid a shuffle?

Maybe through:

• appropriate join strategy
• partitioning
• broadcast where suitable

Is there skew?

Inspect task-level metrics.

Am I collecting data?

Remove unnecessary:

collect()

Am I using Python UDFs unnecessarily?

Replace them with built-in functions where possible.

Is the partition count appropriate?

Tune based on workload and observed execution.

Is the output creating thousands of tiny files?

Control output partitioning appropriately.

The production answer is not:

“Use a bigger cluster.”

It is:

“First identify the bottleneck, then reduce unnecessary computation and data movement, and only then scale resources when justified.”

The Production Spark Debugging Framework

When a Spark job is slow, use this sequence:

                Spark Job Slow
                      │
                      ↓
                Spark UI
                      │
          ┌───────────┴───────────┐
          ↓                       ↓
      Slow Stage?            Failed Tasks?
          │                       │
          ↓                       ↓
      Task Skew?              Memory?
          │                       │
          ↓                       ↓
    Shuffle Heavy?             GC/Spill?
          │
          ↓
     Data Distribution
          │
          ↓
     Query / Physical Plan
          │
          ↓
   Optimize Data Movement
          │
          ↓
      Re-test

This is a much stronger approach than randomly changing Spark configurations.

20 Questions — Quick Revision

The Real Interview Pattern

Notice something about these questions.

Very few are asking:

“What is the syntax?”

Instead, they ask:

“What would you do when the system behaves badly?”

That’s what separates a Spark developer who knows the API from an engineer who has operated Spark workloads in production.

A strong answer usually follows this pattern:

Observe
  ↓
Measure
  ↓
Identify bottleneck
  ↓
Understand data distribution
  ↓
Change one thing
  ↓
Measure again

Not:

Job slow
  ↓
Increase executors
  ↓
Hope

Final Takeaway

Production PySpark knowledge is fundamentally about data movement.

When a Spark pipeline becomes slow or unstable, ask:

How much data am I reading?
          ↓
How much data am I processing?
          ↓
How much data am I shuffling?
          ↓
How evenly is that data distributed?
          ↓
How much data am I sending to the driver?
          ↓
How many files am I producing?

Once you learn to reason about those questions, Spark optimization becomes much less mysterious.

You stop memorizing configuration parameters.

You start understanding why the cluster is doing what it’s doing.

And that’s exactly what production-level PySpark interviews are trying to uncover.

Continue the Series

Part 1 → PySpark fundamentals that interviewers expect
Part 2 → Spark execution and architecture
Part 3 → Transformations, actions, and optimization
Part 4 → Production Spark scenarios
Part 5 → Production debugging, scalability, joins, skew, and reliability — You are here

Next: Part 6 will go deeper into Spark performance tuning scenarios, including shuffle partitions, AQE, executor sizing, serialization, spills, GC, join strategies, and real-world debugging decisions.

Further Reading

For the technical details behind these topics, the best references are the official Apache Spark documentation, particularly the Spark SQL/DataFrame programming guide, PySpark API documentation, and Spark tuning documentation.

These primary sources are especially useful when validating behavior that can vary across Spark versions.