Learn how Spark DataFrames transform data lazily, how transformations build execution plans, and which operations can trigger expensive shuffles.

Apache Spark Learning Path

APACHE SPARK LEARNING PATH

• ✓ 1. What is Apache Spark?
• ✓ 2. Why Spark is Faster than Hadoop
• ✓ 3. Spark Architecture
• ✓ 4. Driver vs Executor
• ✓ 5. Cluster Managers
• → 6. RDD vs DataFrame vs Dataset
• → 7. SparkSession
• ✓ 8. Reading Data
• ✓ 9. Lazy Evaluation
• ✓ 10. DAG
• ✓ 11. Stages and Tasks
• ✓ 12. Shuffle
• ✓ 13. Narrow vs Wide Transformations
• → 14. Writing Data
• → 15-Data Transformations
• → 16. Partitioning
• → 17. Data Skew

Where Are We Now?

We’ve already learned how Spark reads data and how its execution engine works.

Now we need to understand the most common thing we actually do with a DataFrame:

Transform it.

A typical Spark pipeline looks like this:

Read
 │
 ▼
DataFrame
 │
 ├── filter()
 ├── select()
 ├── withColumn()
 ├── groupBy()
 ├── join()
 ├── orderBy()
 └── drop()
 │
 ▼
Transformed DataFrame
 │
 ▼
Action
 │
 ▼
Execution
 │
 ▼
Result

The important part is that most of these transformations do not immediately execute the computation.

Spark builds a plan first.

Execution happens later when an action is called.

What You’ll Learn

In this tutorial, we’ll cover:

• What a DataFrame transformation is
• Why transformations are lazy
• select()
• filter() / where()
• withColumn()
• drop()
• withColumnRenamed()
• distinct()
• dropDuplicates()
• groupBy()
• agg()
• orderBy() / sort()
• join()
• union()
• limit()
• repartition()
• coalesce()
• Narrow vs wide transformations
• Transformations that cause shuffles
• How Catalyst optimizes DataFrame transformations
• Common mistakes
• Interview questions

1. What Is a DataFrame Transformation?

A transformation is an operation that creates a new DataFrame from an existing DataFrame.

For example:

filtered_df = df.filter(df.salary > 80000)

The original df isn’t modified.

Instead:

df
 │
 │ filter()
 ▼
filtered_df

This is an important property of Spark DataFrames:

DataFrames are immutable.

You don’t modify an existing DataFrame in place.

Instead, you create another DataFrame representing the transformed result.

2. Transformations Are Lazy

Consider:

df = spark.read.parquet("/data/employees")
filtered_df = df.filter(df.salary > 80000)
selected_df = filtered_df.select(
    "name",
    "salary"
)

At this point, Spark has not necessarily scanned the entire dataset and performed the filter.

Instead, Spark has essentially built a computation plan:

Read employees
      │
      ▼
Filter salary > 80000
      │
      ▼
Select name, salary

Nothing needs to happen until an action requires the result.

For example:

selected_df.show()

Now Spark needs to execute the plan.

Transformations
      │
      ▼
Execution Plan
      │
      ▼
Action
      │
      ▼
Spark Job
      │
      ▼
Tasks
      │
      ▼
Result

This is the foundation of Spark’s lazy evaluation model.

3. select()

select() chooses columns from a DataFrame.

Suppose:

df.show()

+---+-------+----------+------+
|id |name   |department|salary|
+---+-------+----------+------+
|1  |Alice  |IT        |80000 |
|2  |Bob    |HR        |70000 |
|3  |Charlie|IT        |90000 |
|4  |David  |Finance   |85000 |
+---+-------+----------+------+

Select specific columns:

result = df.select("name", "salary")

Result:

+-------+------+
|name   |salary|
+-------+------+
|Alice  |80000 |
|Bob    |70000 |
|Charlie|90000 |
|David  |85000 |
+-------+------+

You can also use column expressions:

result = df.select(
    "name",
    "salary",
    (df.salary * 1.10).alias("new_salary")
)

4. Why select() Matters for Performance

Suppose your source has:

100 columns
1 TB of data

But your application needs only:

name
salary
department

Selecting only those columns can reduce unnecessary data processing.

df.select(
    "name",
    "salary",
    "department"
)

With columnar formats such as Parquet, Spark can often take advantage of column pruning.

100 columns
     │
     ▼
select 3 columns
     │
     ▼
Read only relevant columns

⚡ Performance Tip

Don’t carry dozens of unused columns through a long Spark pipeline.

Project only the columns you actually need when practical.

5. filter()

filter() keeps rows satisfying a condition.

high_salary = df.filter(df.salary > 80000)

Equivalent SQL-style expression:

high_salary = df.filter("salary > 80000")

You can combine conditions:

result = df.filter(
    (df.salary > 80000) &
    (df.department == "IT")
)

Result:

+---+-------+----------+------+
|id |name   |department|salary|
+---+-------+----------+------+
|3  |Charlie|IT        |90000 |
+---+-------+----------+------+

6. where() Is an Alias for filter()

You can also write:

df.where(df.salary > 80000)

For DataFrames, where() and filter() are equivalent APIs.

df.filter(...)

and:

df.where(...)

are commonly interchangeable.

7. withColumn()

withColumn() is used to add a new column or replace an existing column.

For example:

result = df.withColumn(
    "bonus",
    df.salary * 0.10
)

Now:

+---+-------+------+------+
|id |name   |salary|bonus |
+---+-------+------+------+
|1  |Alice  |80000 |8000  |
|2  |Bob    |70000 |7000  |
|3  |Charlie|90000 |9000  |
+---+-------+------+------+

8. Creating Conditional Columns

You can use when() and otherwise().

from pyspark.sql.functions import when
result = df.withColumn(
    "salary_level",
    when(df.salary >= 90000, "High")
    .when(df.salary >= 75000, "Medium")
    .otherwise("Low")
)

Result:

Alice   → Medium
Bob     → Low
Charlie → High
David   → Medium

This is similar to a SQL CASE WHEN.

9. withColumnRenamed()

To rename a column:

result = df.withColumnRenamed(
    "salary",
    "annual_salary"
)

You can then use:

result.select("name", "annual_salary")

This is useful for cleaning inconsistent source schemas.

10. drop()

Remove a column:

result = df.drop("salary")

Multiple columns:

result = df.drop(
    "salary",
    "department"
)

Again, the original DataFrame remains unchanged.

11. distinct()

distinct() removes duplicate rows.

Suppose:

Alice
Alice
Bob
Bob
Charlie

Then:

result = df.distinct()

returns unique rows.

However, distinct() can require a shuffle.

Conceptually:

Partitions
   │
   ├── Alice
   ├── Bob
   │
   ├── Alice
   └── Charlie
        │
        ▼
     SHUFFLE
        │
        ▼
 Deduplicated Data

⚡ Performance Tip

Don’t assume every transformation is cheap.

Operations that require data to be redistributed across partitions can be significantly more expensive.

12. dropDuplicates()

If you want uniqueness based on particular columns:

result = df.dropDuplicates(["email"])

Suppose:

id | email
---|----------------
1  | a@example.com
2  | a@example.com
3  | b@example.com

Then:

df.dropDuplicates(["email"])

keeps one row for each unique email.

This is different from:

df.distinct()

because distinct() considers the entire row.

13. groupBy()

groupBy() groups records according to one or more columns.

Example:

result = df.groupBy("department").count()

Result:

+----------+-----+
|department|count|
+----------+-----+
|IT        |2    |
|HR        |1    |
|Finance   |1    |
+----------+-----+

A typical aggregation:

result = df.groupBy("department").sum("salary")

14. groupBy() Usually Means Shuffle

This is an important performance concept.

Imagine:

Partition 1
IT → 80000
HR → 70000
Partition 2
IT → 90000
Finance → 85000

Spark needs all records belonging to the same grouping key to be brought together.

Partition 1 ──┐
              │
Partition 2 ──┼──→ Shuffle → Group by department
              │
Partition 3 ──┘

Therefore, aggregations such as groupBy() commonly involve a shuffle.

This connects directly to our earlier tutorial on Shuffle in Apache Spark.

15. agg()

agg() allows multiple aggregations.

from pyspark.sql.functions import (
    avg,
    max,
    min,
    sum,
    count
)
result = df.groupBy("department").agg(
    count("*").alias("employee_count"),
    avg("salary").alias("avg_salary"),
    max("salary").alias("max_salary"),
    min("salary").alias("min_salary")
)

Example result:

+----------+--------------+----------+---------+---------+
|department|employee_count|avg_salary|max_salary|min_salary|
+----------+--------------+----------+---------+---------+
|IT        |2             |85000     |90000    |80000    |
|HR        |1             |70000     |70000    |70000    |
|Finance   |1             |85000     |85000    |85000    |
+----------+--------------+----------+---------+---------+

16. orderBy()

Sort a DataFrame:

result = df.orderBy("salary")

Descending order:

from pyspark.sql.functions import desc
result = df.orderBy(desc("salary"))

Multiple columns:

result = df.orderBy(
    desc("department"),
    "salary"
)

17. Why Sorting Can Be Expensive

A global sort often requires data to be redistributed so that ordering can be established across the dataset.

Conceptually:

Unsorted Partitions
      │
      ▼
   Shuffle
      │
      ▼
Sorted Partitions

For large datasets, global sorting can therefore be expensive.

💡 Going Deeper

Not every ordering operation should be treated as identical.

The physical plan chosen by Spark matters.

For production debugging, use:

df.explain()

to understand what Spark actually plans to do.

18. join()

Joins combine data from two DataFrames.

Suppose:

employees
id | name   | department_id
---|--------|--------------
1  | Alice  | 10
2  | Bob    | 20
3  | Charlie| 10

and:

departments
department_id | department
--------------|-----------
10            | IT
20            | HR

We can join:

result = employees.join(
    departments,
    employees.department_id == departments.department_id,
    "inner"
)

Result:

Alice   → IT
Bob     → HR
Charlie → IT

19. Join Types

Spark supports common join types such as:

inner
left
right
full
left_semi
left_anti
cross

Example:

df1.join(
    df2,
    "id",
    "left"
)

The join type determines which rows are retained.

20. Why Joins Can Be Expensive

Consider:

DataFrame A
      │
      ├──────────┐
      │          │
      ▼          ▼
DataFrame B   Join Key
      │          │
      └────┬─────┘
           ▼
        Shuffle
           │
           ▼
       Join Result

A large join may require significant data movement.

This is why join optimization becomes one of the most important topics in production Spark.

We’ll cover:

• broadcast joins,
• join strategies,
• skewed joins,
• partitioning,
• Adaptive Query Execution,

in upcoming tutorials.

21. union()

union() combines rows from two compatible DataFrames.

df1 = spark.createDataFrame(
    [(1, "Alice")],
    ["id", "name"]
)
df2 = spark.createDataFrame(
    [(2, "Bob")],
    ["id", "name"]
)
result = df1.union(df2)

Result:

+---+-----+
|id |name |
+---+-----+
|1  |Alice|
|2  |Bob  |
+---+-----+

Important

union() combines columns by position, not by column name.

If column ordering differs, you may get incorrect results.

When you need name-based alignment, consider:

unionByName()

22. unionByName()

Suppose:

df1:
id | name | salary
df2:
name | salary | id

Using:

df1.union(df2)

can produce incorrect alignment because the columns are matched by position.

Instead:

df1.unionByName(df2)

matches columns by name.

If schemas have missing columns, depending on the use case, you can also use:

df1.unionByName(
    df2,
    allowMissingColumns=True
)

23. limit()

limit() restricts the number of rows returned:

result = df.limit(10)

This is useful for exploration and testing.

For example:

df.limit(20).show()

can be preferable to displaying a massive dataset while developing.

24. repartition()

repartition() changes the number or distribution of partitions.

result = df.repartition(10)

It generally causes a shuffle.

You can also partition by a column:

result = df.repartition(
    10,
    "department"
)

Conceptually:

Before
P0   P1   P2   P3
 \    |    |   /
      │
    Shuffle
      │
      ▼
P0  P1  P2 ... P9

25. coalesce()

coalesce() is commonly used to reduce the number of partitions.

result = df.coalesce(5)

Unlike repartition(), it typically avoids a full shuffle when reducing partitions.

100 partitions
      │
      ▼
coalesce(10)
      │
      ▼
10 partitions

This can be useful before writing data when you want fewer output files.

But don’t blindly use it.

Reducing partitions too aggressively can reduce parallelism.

26. Narrow vs Wide DataFrame Transformations

One of the most useful ways to understand transformations is to classify them by how they move data between partitions.

Narrow transformation

A narrow transformation can process each output partition using data from a limited number of input partitions without requiring a full redistribution.

Examples commonly include:

select()
filter()
withColumn()
drop()

Conceptually:

P1 ───────→ P1'
P2 ───────→ P2'
P3 ───────→ P3'

No large cross-partition redistribution is required.

Wide transformation

A wide transformation requires data to be redistributed across partitions.

Examples commonly include:

groupBy()
join()
distinct()
orderBy()
repartition()

Conceptually:

P1 ──┐
P2 ──┼──→ Shuffle ──→ P1'
P3 ──┤                P2'
P4 ──┘                P3'

This distinction matters because wide transformations often introduce shuffle boundaries and therefore can create additional stages.

27. A Transformation Pipeline

Let’s combine several operations.

result = (
    df
    .filter(df.salary > 70000)
    .select(
        "name",
        "department",
        "salary"
    )
    .withColumn(
        "bonus",
        df.salary * 0.10
    )
    .groupBy("department")
    .agg(
        {"salary": "avg"}
    )
)

Conceptually:

Read
 │
 ▼
Filter
 │
 ▼
Select
 │
 ▼
withColumn
 │
 ▼
GroupBy
 │
 ▼
Shuffle
 │
 ▼
Aggregation
 │
 ▼
Result

The first operations can often be pipelined.

The aggregation may introduce a shuffle.

28. Catalyst Optimizer

One reason DataFrames are powerful is that Spark understands their structure.

Consider:

df.filter(df.salary > 70000) \
  .select("name", "salary")

Spark doesn’t simply execute these operations exactly as written, one line at a time.

The DataFrame API contributes to a query plan that Spark SQL can analyze and optimize.

A simplified view:

DataFrame Operations
        │
        ▼
Logical Plan
        │
        ▼
Analyzed Logical Plan
        │
        ▼
Optimized Logical Plan
        │
        ▼
Physical Plan
        │
        ▼
Execution

This is one of the major reasons DataFrames are generally preferred over RDDs for structured data processing.

29. Predicate Pushdown

Suppose we write:

df.select(
    "name",
    "salary"
).filter(
    df.salary > 80000
)

Spark’s optimizer can often rearrange operations logically so that filtering happens as early as possible.

Conceptually:

Instead of:
Read everything
     ↓
Select
     ↓
Filter
Spark can often optimize toward:
Read relevant columns
     ↓
Filter early
     ↓
Continue processing

When supported by the underlying data source, predicate pushdown can also push filters closer to the source.

This can significantly reduce the amount of data that needs to be processed.

30. Column Pruning

Similarly, if your pipeline ultimately needs only:

df.select(
    "customer_id",
    "amount"
)

Spark can often eliminate unnecessary columns from the execution plan.

Source
│
├── customer_id ✓
├── amount      ✓
├── address     ✗
├── phone       ✗
├── email       ✗
└── metadata    ✗

This is called column pruning.

31. Don’t Use Python UDFs for Everything

Suppose you need to transform a column.

A common mistake is immediately writing a Python UDF.

from pyspark.sql.functions import udf

For operations already supported by Spark’s built-in functions, prefer those built-in expressions.

For example:

from pyspark.sql.functions import upper
df.withColumn(
    "name_upper",
    upper("name")
)

is generally preferable to implementing the same simple operation through a Python UDF.

Why?

Because built-in Spark expressions are visible to Spark’s optimizer and execution engine.

32. A Practical End-to-End Example

Let’s build a realistic transformation pipeline.

Suppose we have:

employees

with:

employee_id
name
department
salary
status

We want:

1. active employees only,
2. salaries above $70,000,
3. calculate bonus,
4. group by department,
5. calculate average salary.

Step 1 — Filter

active = df.filter(
    df.status == "ACTIVE"
)

Step 2 — Filter salary

high_salary = active.filter(
    active.salary > 70000
)

Step 3 — Add bonus

with_bonus = high_salary.withColumn(
    "bonus",
    high_salary.salary * 0.10
)

Step 4 — Aggregate

result = with_bonus.groupBy(
    "department"
).agg(
    avg("salary").alias("avg_salary"),
    avg("bonus").alias("avg_bonus")
)

Execution concept

Source
  │
  ▼
Filter status
  │
  ▼
Filter salary
  │
  ▼
Calculate bonus
  │
  ▼
GroupBy department
  │
  ▼
     SHUFFLE
  │
  ▼
Aggregation
  │
  ▼
Result

This is much closer to how production Spark pipelines actually look.

33. How to Inspect Transformations

When performance matters, don’t guess.

Use:

df.explain()

For more detail:

df.explain(True)

This can expose the logical and physical plans used by Spark.

For example:

result.explain(True)

can help you identify:

• filters,
• projections,
• joins,
• exchanges,
• scans,
• aggregations,
• physical operators.

An Exchange in a physical plan is often a strong signal that Spark needs to redistribute data, commonly because of a shuffle.

34. Common Mistakes

Mistake 1 — Thinking transformations execute immediately

This:

df.filter(...)

doesn’t mean Spark immediately scans and filters the entire dataset.

The operation contributes to a plan.

An action eventually triggers execution.

Mistake 2 — Assuming every transformation is cheap

These aren’t necessarily equivalent from a performance perspective:

df.select(...)

and:

df.groupBy(...).count()

The second commonly requires a shuffle.

Mistake 3 — Repeated withColumn() calls unnecessarily

Long chains of column manipulation can make code difficult to maintain and, depending on the expression structure and Spark version, may contribute to more complex plans.

When appropriate, consolidate related expressions with select().

For example:

df.select(
    "*",
    (df.salary * 0.10).alias("bonus"),
    (df.salary * 1.10).alias("total_compensation")
)

Mistake 4 — Using collect() to inspect large data

Avoid:

df.collect()

on a large DataFrame.

It brings all returned records to the Driver.

For exploration, use:

df.show()

or:

df.limit(20).show()

Mistake 5 — Ignoring shuffle

A transformation such as:

df.groupBy("department").count()

looks simple.

But internally, Spark may need to move a large amount of data across the cluster.

Always think:

Does this operation require redistribution?

35. 💡 Going Deeper: DataFrame Code Is a Declarative Description

A useful mental shift is to stop thinking of DataFrame code as:

“Execute this line, then execute this line, then execute this line.”

Instead, think:

“Describe the computation I want Spark to perform.”

For example:

result = (
    df
    .filter("salary > 70000")
    .select("name", "salary")
)

You’re describing a computation.

Spark can then analyze and optimize that computation before execution.

That’s one of the fundamental differences between using a high-level DataFrame API and manually managing low-level distributed operations.

36. ⚡ Performance Checklist for DataFrame Transformations

Before deploying a Spark transformation pipeline, ask:

1. Am I reading unnecessary columns?

Use:

select(...)

to reduce the working dataset where appropriate.

2. Can I filter earlier?

Apply selective filters as early as practical.

3. Does this operation cause a shuffle?

Look for operations such as:

groupBy
join
distinct
orderBy
repartition

4. Am I using built-in Spark functions?

Prefer:

upper()
substring()
regexp_replace()
when()

over unnecessary Python UDFs.

5. Is the partition count appropriate?

Check:

df.rdd.getNumPartitions()

and inspect the physical plan/Spark UI when tuning.

6. Is the data skewed?

One enormous partition can dominate total execution time even when the average partition size looks reasonable.

37. DataFrame Transformations Cheat Sheet

Important: “shuffle risk” is a practical rule of thumb, not an absolute guarantee. The physical plan and optimizer determine what Spark actually executes.

38. Common Interview Questions

1. What is a DataFrame transformation?

A transformation creates a new DataFrame from an existing DataFrame and is generally evaluated lazily.

2. Why are Spark transformations lazy?

Lazy evaluation allows Spark to build and optimize a complete execution plan before running it.

3. What is the difference between filter() and where()?

For DataFrames, they are equivalent APIs for filtering rows.

4. What does withColumn() do?

It adds a new column or replaces an existing column with a new expression.

5. What is the difference between distinct() and dropDuplicates()?

distinct() removes duplicate complete rows.

dropDuplicates() can remove duplicates based on specified columns.

6. Why is groupBy() expensive?

It commonly requires a shuffle so records with the same grouping key can be brought together.

7. What is the difference between repartition() and coalesce()?

repartition() redistributes data and typically causes a shuffle.

coalesce() is commonly used to reduce partitions while avoiding a full shuffle.

8. Why are DataFrames generally preferred over RDDs for structured data?

DataFrames provide schema information and allow Spark SQL’s optimizer and structured execution engine to reason about the computation.

9. What is predicate pushdown?

It is an optimization where filters can be pushed closer to the data source, reducing the amount of data that needs to be read or processed.

10. What is column pruning?

Column pruning removes unnecessary columns from the execution plan so Spark doesn’t process data that isn’t required.

11. How do you investigate a slow transformation?

Start with:

df.explain(True)

and inspect the Spark UI for:

• stages,
• task durations,
• shuffle read/write,
• partition sizes,
• skew,
• input/output sizes.

12. What is an Exchange in a Spark physical plan?

An Exchange generally represents a redistribution of data between partitions and is commonly associated with a shuffle.

39. Final Mental Model

When you write:

result = (
    df
    .filter("salary > 70000")
    .select("name", "department", "salary")
    .groupBy("department")
    .agg(avg("salary").alias("avg_salary"))
)

don’t think:

Line 1 executes
     ↓
Line 2 executes
     ↓
Line 3 executes

Think:

                DataFrame
                    │
                    ▼
                filter()
                    │
                    ▼
                select()
                    │
                    ▼
                groupBy()
                    │
                    ▼
                 agg()
                    │
                    ▼
             Execution Plan
                    │
                    ▼
              Optimization
                    │
                    ▼
                 Shuffle
                    │
                    ▼
              Tasks / Stages
                    │
                    ▼
                 Result

And nothing needs to execute until an action requires the result.

Key Takeaways

• DataFrame transformations create new DataFrames rather than modifying existing ones.
• Transformations are generally lazy.
• select(), filter(), withColumn(), and drop() are among the most frequently used transformations.
• groupBy(), join(), distinct(), orderBy(), and repartition() commonly involve data redistribution.
• Shuffle is one of the most important performance considerations in Spark.
• DataFrames allow Spark to optimize structured computations through Spark SQL’s optimizer.
• Predicate pushdown and column pruning can reduce unnecessary data processing.
• Prefer built-in Spark functions over unnecessary Python UDFs.
• repartition() typically introduces a shuffle, while coalesce() commonly reduces partitions without a full shuffle.
• Use explain(True) and the Spark UI to understand what Spark actually executes.
• Good Spark developers don’t just know transformation syntax—they understand the execution cost behind each transformation.

Continue Learning Apache Spark

We’ve now moved from simply reading and writing data to understanding how Spark transforms it:

Read Data
    │
    ▼
DataFrame
    │
    ├── select()
    ├── filter()
    ├── withColumn()
    ├── groupBy()
    ├── join()
    └── orderBy()
    │
    ▼
Execution Plan
    │
    ▼
Optimization
    │
    ▼
Shuffle / Stages / Tasks
    │
    ▼
Result

But there’s one important piece still missing.

We’ve discussed transformations extensively, but when exactly does Spark execute them?

That’s where DataFrame Actions come in.

Next Article

DataFrame Actions in Apache Spark: When Your Lazy Transformations Actually Run

We’ll cover:

• What an action is
• show()
• count()
• collect()
• take()
• first()
• head()
• reduce()
• write
• Why actions trigger Spark jobs
• Why multiple actions can recompute the same DataFrame
• When to use caching
• Driver memory risks with collect()
• How actions connect transformations → DAG → stages → tasks

Next → DataFrame Actions

Follow the Complete Apache Spark Series

Spark Fundamentals → Architecture → APIs → DataFrame Transformations → Actions → Lazy Evaluation → DAG → Stages → Shuffle → Writing → Partitioning → Skew → Join Optimization → Performance Tuning

The goal isn’t just to memorize DataFrame methods.

It’s to understand what each transformation means to Spark’s execution engine—and what it costs in production.