Understand what SparkSession is, why modern Spark applications need it, what it actually creates, and how it differs from SparkContext

If you’ve written PySpark code, you’ve probably started with something like this:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .appName("SalesAnalysis") \
    .getOrCreate()

And then immediately started working with DataFrames:

df = spark.read.parquet("sales/")

df.show()

df.groupBy("customer_id").sum("amount").show()

But what exactly is spark?

Why do we need a SparkSession?

What happens when getOrCreate() is called?

And how is SparkSession different from the older SparkContext?

These questions become important as soon as you move from writing simple PySpark scripts to building production Spark applications.

The key idea is:

SparkSession is the unified entry point for programming Spark with the DataFrame and SQL APIs.

It gives your application access to Spark’s SQL engine, DataFrame APIs, catalog, configuration, and underlying Spark context.

Apache Spark Learning Path

APACHE SPARK LEARNING PATH

✓ 1. What is Apache Spark?
✓ 2. Why Spark is Faster than Hadoop
✓ 3. Spark Architecture
✓ 4. Driver vs Executor
✓ 5. Cluster Managers
→ 6. RDD vs DataFrame vs Dataset
→ 7. SparkSession
✓ 8. Reading Data
✓ 9. Lazy Evaluation
✓ 10. DAG
✓ 11. Stages and Tasks
✓ 12. Shuffle
✓ 13. Narrow vs Wide Transformations
→ 14. Writing Data
→ 15. Partitioning
→ 16. Data Skew

[View All Apache Spark Tutorials →]

What Is SparkSession?

SparkSession is the main entry point for working with Spark’s structured APIs in modern Spark applications.

In PySpark:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .appName("MyApplication") \
    .getOrCreate()

The variable:

spark

is a SparkSession object.

You use it to perform operations such as:

spark.read

spark.sql(...)

spark.createDataFrame(...)

spark.conf

and:

spark.catalog

It also provides access to the underlying SparkContext:

spark.sparkContext

Why Was SparkSession Introduced?

Before Spark 2.0, Spark applications commonly worked with several different entry points.

For example:

SparkContext
SQLContext
HiveContext

Different APIs had different context objects.

This could make application setup more complicated.

Spark 2.0 introduced SparkSession as a unified entry point.

Conceptually:

Before Spark 2.0
SparkContext
     │
     ├── SQLContext
     │
     └── HiveContext
Spark 2.0+
          SparkSession
               │
       ┌───────┼────────┐
       ↓       ↓        ↓
    SQL/DataFrame  Catalog  Config
               │
               ↓
        SparkContext

The goal was to provide a cleaner interface for modern Spark applications.

Creating a SparkSession

The most common PySpark pattern is:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .appName("SalesAnalysis") \
    .getOrCreate()

Let’s break it down.

SparkSession.builder

SparkSession.builder

returns a builder that allows you to configure the session before creating or retrieving it.

For example:

spark = SparkSession.builder \
    .appName("SalesAnalysis") \
    .master("local[*]") \
    .getOrCreate()

You can configure things such as:

Application name
Master
Spark configuration
Extensions
Other session-level settings

appName()

.appName("SalesAnalysis")

sets the name of your Spark application.

You will typically see this name in places such as:

• Spark UI
• Cluster management interfaces
• Application logs

For example:

spark = SparkSession.builder \
    .appName("CustomerRevenuePipeline") \
    .getOrCreate()

The application will be identified as:

CustomerRevenuePipeline

This becomes especially useful when multiple Spark applications are running on the same cluster.

master()

You can specify where Spark should run.

For local development:

.master("local[*]")

means Spark can use the available CPU cores of the local machine.

For example:

spark = SparkSession.builder \
    .appName("LocalTest") \
    .master("local[*]") \
    .getOrCreate()

In a cluster environment, the deployment configuration is typically supplied by the cluster manager or submission mechanism rather than hardcoding it in application code.

For example, you may encounter:

YARN
Kubernetes
Standalone

depending on the environment.

What Does getOrCreate() Do?

This is one of the most important methods in the builder API.

.getOrCreate()

means:

Use an existing session when an appropriate one is available; otherwise create a new one.

For example:

spark = SparkSession.builder \
    .appName("MyApp") \
    .getOrCreate()

This is why you commonly see getOrCreate() instead of explicitly constructing a completely new session every time.

It is particularly convenient in:

• Jupyter notebooks
• Databricks notebooks
• Interactive environments
• Applications where a session may already exist

What Can You Do With SparkSession?

SparkSession provides access to many of the APIs you use every day in PySpark.

1. Read Data

For example:

df = spark.read.csv(
    "sales.csv",
    header=True,
    inferSchema=True
)

Or:

df = spark.read.parquet("sales/")

Or:

df = spark.read.json("events.json")

The spark.read interface is one of the most commonly used features of SparkSession.

2. Create DataFrames

You can create a DataFrame directly:

data = [
    ("Alice", 1000),
    ("Bob", 2000),
    ("Charlie", 3000)
]
df = spark.createDataFrame(
    data,
    ["name", "salary"]
)

Now:

df.show()

produces a DataFrame that can be processed using Spark’s distributed APIs.

3. Execute SQL

SparkSession provides access to Spark SQL:

spark.sql("""
    SELECT customer_id, SUM(amount)
    FROM orders
    GROUP BY customer_id
""")

You can also register a DataFrame as a temporary SQL view:

orders.createOrReplaceTempView("orders")

Then:

result = spark.sql("""
    SELECT customer_id, SUM(amount)
    FROM orders
    GROUP BY customer_id
""")

This gives you two ways to work with the same underlying Spark engine:

PySpark DataFrame API
          │
          ↓
      Spark SQL
          ↑
          │
       SQL API

4. Manage Configuration

SparkSession gives you access to runtime configuration:

spark.conf.get("spark.sql.shuffle.partitions")

You can also set configurations:

spark.conf.set(
    "spark.sql.shuffle.partitions",
    200
)

This becomes useful when tuning Spark SQL and DataFrame workloads.

For example, the shuffle configuration is directly related to the Shuffle and Partitioning concepts covered elsewhere in this series.

5. Access the Catalog

SparkSession provides access to Spark’s catalog:

spark.catalog

For example:

spark.catalog.listTables()

You can inspect available tables and other catalog information.

SparkSession and SparkContext

This is one of the most common Spark interview questions.

You may hear:

“What is the difference between SparkSession and SparkContext?”

The simplest answer is:

SparkContext is the lower-level entry point to Spark’s core execution functionality, while SparkSession is the higher-level unified entry point for Spark SQL, DataFrames, and related APIs.

Conceptually:

                 SparkSession
                      │
          ┌───────────┼────────────┐
          ↓           ↓            ↓
      DataFrame     SQL         Catalog
          │
          │
          ↓
    Spark execution
          │
          ↓
    SparkContext
          │
          ↓
   Cluster / Executors

You can access the context through:

spark.sparkContext

For example:

sc = spark.sparkContext

What Is SparkContext?

SparkContext represents the connection between your Spark application and the Spark cluster’s core execution environment.

Historically, it was the main entry point for Spark Core APIs.

For example:

sc.parallelize([1, 2, 3, 4, 5])

creates an RDD.

rdd = sc.parallelize([1, 2, 3, 4, 5])

Today, most new application code uses SparkSession and DataFrames unless there is a specific reason to work directly with RDDs.

SparkSession vs SparkContext

A useful mental model is:

SparkSession
     │
     ├── DataFrame API
     ├── SQL
     ├── Catalog
     ├── Configuration
     │
     └── SparkContext
              │
              └── Spark Core

Do We Still Need SparkContext?

Yes.

But most application developers don’t need to create it manually.

Instead:

spark = SparkSession.builder.getOrCreate()

and:

sc = spark.sparkContext

The SparkSession manages the application-level interface while the underlying SparkContext remains part of Spark’s core execution architecture.

Can You Create Multiple SparkSessions?

Yes, Spark supports multiple sessions sharing the same underlying SparkContext in appropriate circumstances.

This distinction matters because a session can provide session-specific SQL state, such as temporary views and certain configurations.

For example:

spark = SparkSession.builder \
    .appName("MyApp") \
    .getOrCreate()
spark2 = spark.newSession()

newSession() creates a new session with isolated SQL state while sharing the underlying SparkContext and related execution infrastructure.

Conceptually:

                 SparkContext
                      │
              ┌───────┴───────┐
              ↓               ↓
        SparkSession 1   SparkSession 2
              │               │
              ↓               ↓
        SQL state 1       SQL state 2

This is useful when you need session-level isolation without creating an entirely separate Spark application context.

SparkSession in a PySpark Application

A typical application may look like this:

from pyspark.sql import SparkSession
spark = (
    SparkSession.builder
    .appName("SalesPipeline")
    .getOrCreate()
)
df = spark.read.parquet("s3://bucket/sales/")
result = (
    df
    .filter("status = 'COMPLETED'")
    .groupBy("customer_id")
    .sum("amount")
)
result.show()
spark.stop()

Let’s look at what happens conceptually.

Application starts
       ↓
SparkSession created/retrieved
       ↓
Data read
       ↓
Transformations defined
       ↓
Execution plan created
       ↓
Action: show()
       ↓
Spark executes the plan
       ↓
Tasks run on executors
       ↓
Results returned
       ↓
Application ends

Notice something important:

Creating the SparkSession does not mean your DataFrame transformations immediately execute.

That brings us back to lazy evaluation.

SparkSession Does Not Execute Every Line Immediately

Consider:

df = spark.read.parquet("sales/")

Then:

filtered = df.filter("amount > 1000")

Then:

result = filtered.groupBy("customer_id").sum("amount")

At this point, Spark has built a plan.

The actual distributed computation happens when you call an action such as:

result.show()

or:

result.count()

So the relationship is:

SparkSession
     ↓
DataFrame API
     ↓
Transformations
     ↓
Logical/physical planning
     ↓
Action
     ↓
Execution

This is why SparkSession should be thought of as the entry point, not the engine that literally performs every operation itself.

SparkSession and the Driver

Another important connection is with the Spark Driver.

When your PySpark application starts, the driver process creates or obtains the SparkSession.

Conceptually:

              Spark Application
                     │
                     ↓
                  Driver
                     │
              SparkSession
                     │
                     ↓
             Spark execution
                     │
          ┌──────────┼──────────┐
          ↓          ↓          ↓
      Executor 1 Executor 2 Executor 3

The SparkSession is therefore associated with the application running on the driver.

It does not mean that every executor has its own independent SparkSession.

The driver uses Spark’s execution infrastructure to coordinate work that is ultimately executed by tasks on executors.

SparkSession in Databricks

If you’re using Databricks notebooks, you may notice something interesting.

You often write:

df = spark.read.parquet(...)

without explicitly creating:

SparkSession.builder...

Why?

Because the notebook environment typically provides an existing SparkSession for you.

So:

spark

is already available.

You can inspect it:

type(spark)

and typically see a SparkSession object.

This is why you shouldn’t blindly add another session initialization block to every Databricks notebook.

SparkSession in Jupyter

In a local Jupyter environment, you may explicitly create the session:

from pyspark.sql import SparkSession
spark = (
    SparkSession.builder
    .appName("JupyterSpark")
    .master("local[*]")
    .getOrCreate()
)

Then:

df = spark.createDataFrame(
    [(1, "Alice"), (2, "Bob")],
    ["id", "name"]
)

You can work with the DataFrame interactively.

SparkSession and Configuration

SparkSession is also a convenient place to configure Spark.

For example:

spark.conf.set(
    "spark.sql.adaptive.enabled",
    "true"
)

Or:

spark.conf.set(
    "spark.sql.shuffle.partitions",
    200
)

Then retrieve it:

spark.conf.get(
    "spark.sql.shuffle.partitions"
)

However, configuration scope and whether a setting can be changed at runtime depend on the specific Spark configuration.

For production workloads, cluster/application configuration is often managed outside the application code.

SparkSession and Temporary Views

One of the most useful SQL features exposed through SparkSession is working with temporary views.

Suppose:

df.createOrReplaceTempView("sales")

Then:

result = spark.sql("""
    SELECT
        customer_id,
        SUM(amount) AS total_amount
    FROM sales
    GROUP BY customer_id
""")

The DataFrame API and SQL API are therefore not two completely separate processing systems.

They are different interfaces that can feed into Spark SQL’s planning and execution machinery.

Conceptually:

           DataFrame API
                │
                ↓
          Spark SQL Engine
                ↑
                │
             SQL API

SparkSession vs SQLContext

You may encounter older Spark code like:

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

This is largely legacy code in modern Spark applications.

Modern applications generally use:

spark = SparkSession.builder.getOrCreate()

and:

spark.sql(...)

instead.

The introduction of SparkSession simplified the API by providing a unified entry point.

SparkSession vs HiveContext

Similarly, older applications may contain:

HiveContext

SparkSession provides the modern unified interface for SQL-related functionality, including support for Hive-related functionality when the appropriate Spark/Hive configuration and dependencies are available.

For new applications, you generally start with:

SparkSession

rather than manually creating separate context objects.

What Happens When You Call spark.read?

This is an interesting detail.

When you write:

df = spark.read.parquet("sales/")

you’re interacting with Spark’s DataFrame reader through the SparkSession.

The operation establishes a DataFrame representing the data source.

Then Spark can build an execution plan around that DataFrame.

Conceptually:

spark.read
    ↓
DataFrame
    ↓
Logical Plan
    ↓
Optimized Plan
    ↓
Physical Plan
    ↓
Execution

This distinction is important:

A SparkSession is the entry point through which you define Spark work; it is not itself the DataFrame or the executor performing the computation.

Should You Call spark.stop()?

For a standalone Spark application, it is generally good practice to stop the SparkSession when the application is finished:

spark.stop()

This allows Spark to shut down the associated resources.

A typical application structure is:

from pyspark.sql import SparkSession
spark = (
    SparkSession.builder
    .appName("MyApplication")
    .getOrCreate()
)
# Application logic
spark.stop()

In interactive environments such as notebooks, you usually don’t want to repeatedly stop and recreate the session after every cell because the environment may manage the session lifecycle for you.

Common Mistakes

Mistake 1: Creating a New SparkSession Everywhere

Avoid repeatedly doing:

SparkSession.builder.getOrCreate()

in every function without understanding session ownership.

A common application pattern is to create the session at the application entry point and pass/use it where needed.

For example:

def process_data(spark):
    df = spark.read.parquet("sales/")
    return df.filter("amount > 1000")

Then:

spark = SparkSession.builder \
    .appName("SalesApp") \
    .getOrCreate()
result = process_data(spark)

Mistake 2: Thinking SparkSession Is SparkContext

They are related but not identical.

spark

is a SparkSession.

spark.sparkContext

is the underlying SparkContext.

Mistake 3: Assuming Creating a Session Starts Every Computation

This:

spark = SparkSession.builder.getOrCreate()

doesn’t execute:

df.filter(...)

or:

df.groupBy(...)

Those operations are still subject to Spark’s execution model and lazy evaluation.

Mistake 4: Hardcoding Cluster Configuration

You may see:

.master("local[*]")

in tutorials.

That’s useful for local development.

But you generally should not blindly put:

.master("local[*]")

into production cluster applications.

The deployment environment should normally determine where the application runs.

A Production-Style Pattern

A clean application might look like:

from pyspark.sql import SparkSession
def create_spark_session():
    return (
        SparkSession.builder
        .appName("CustomerRevenuePipeline")
        .getOrCreate()
    )
def process_data(spark):
    orders = spark.read.parquet("orders/")
    return (
        orders
        .filter("status = 'COMPLETED'")
        .select("customer_id", "amount")
        .groupBy("customer_id")
        .sum("amount")
    )
if __name__ == "__main__":
    spark = create_spark_session()
    try:
        result = process_data(spark)
        result.write.mode("overwrite").parquet("output/")
    finally:
        spark.stop()

This separates:

Session creation
      ↓
Business logic
      ↓
Application lifecycle

That structure becomes especially useful as Spark applications grow.

The Complete Mental Model

Put everything together:

                 PySpark Application
                         │
                         ↓
                  SparkSession
                         │
          ┌──────────────┼──────────────┐
          ↓              ↓              ↓
      DataFrame         SQL          Catalog
          │
          ↓
    Logical Plan
          │
          ↓
    Optimized Plan
          │
          ↓
    Physical Plan
          │
          ↓
     SparkContext
          │
          ↓
       Scheduler
          │
       ┌──┴──┐
       ↓     ↓
   Executors...
       │
       ↓
     Tasks

This diagram connects several concepts you’ve already learned.

SparkSession is the application-facing entry point.

SparkContext is part of the lower-level Spark execution infrastructure.

The scheduler turns the work into jobs, stages, and tasks.

Executors execute those tasks.

SparkSession: Interview Questions

1. What is SparkSession?

SparkSession is the unified entry point for working with Spark’s DataFrame, SQL, catalog, and related structured APIs.

2. Why was SparkSession introduced?

It unified previously separate entry points such as SparkContext-related SQL contexts into a simpler API for modern Spark applications.

3. How do you create a SparkSession?

from pyspark.sql import SparkSession
spark = (
    SparkSession.builder
    .appName("MyApp")
    .getOrCreate()
)

4. What does getOrCreate() do?

It returns an existing appropriate SparkSession when one is available or creates one when necessary.

5. What is the difference between SparkSession and SparkContext?

SparkSession is the higher-level unified entry point for DataFrame/SQL APIs, while SparkContext provides lower-level Spark Core functionality.

You can access the context using:

spark.sparkContext

6. Can SparkSession execute SQL?

Yes.

spark.sql("""
    SELECT * FROM customers
""")

7. Can SparkSession create a DataFrame?

Yes.

spark.createDataFrame(data, schema)

8. Why do Databricks notebooks already have spark?

The notebook environment typically provides an existing SparkSession for the notebook’s Spark environment.

9. Can multiple SparkSessions share a SparkContext?

Yes. For example, newSession() can create a new SQL session while sharing the underlying SparkContext and execution infrastructure.

10. Should you call spark.stop()?

For standalone applications, generally yes, when the application has completed. Interactive notebook environments typically manage the session lifecycle differently.

Key Takeaways

If you’re preparing for a Spark interview or building PySpark applications, remember:

1. SparkSession is the modern unified entry point for Spark’s structured APIs.
2. It was introduced in Spark 2.0.
3. You commonly create it with SparkSession.builder.
4. getOrCreate() retrieves an existing session or creates one when needed.
5. spark.read is used to load data into DataFrames.
6. spark.sql() executes Spark SQL.
7. spark.createDataFrame() creates DataFrames.
8. spark.conf provides access to Spark SQL/runtime configuration.
9. spark.sparkContext gives access to the underlying SparkContext.
10. SparkSession is associated with the driver-side application, not an independent session on every executor.
11. Creating a SparkSession does not mean every DataFrame transformation executes immediately.
12. In Databricks and other managed environments, the spark session is often already provided.

The simplest mental model is:

SparkSession
     ↓
Entry point
     ↓
DataFrame / SQL
     ↓
Execution plan
     ↓
Spark scheduler
     ↓
Stages
     ↓
Tasks
     ↓
Executors

Once you understand this flow, SparkSession stops being just boilerplate code at the top of every PySpark notebook and becomes an important part of understanding how a Spark application is structured.

Continue Learning Apache Spark

Now that you understand the entry point of a Spark application, the next concepts connect directly to how Spark executes the work you define through that session:

→ Lazy Evaluation — why your Spark code doesn’t execute when you write it
→ DAG — how Spark represents the computation
→ Stages and Tasks — how Spark breaks the computation into executable units
→ Shuffle — how Spark redistributes data
→ Narrow vs Wide Transformations — why some operations trigger shuffle

Next Step: Lazy Evaluation →

You now know where Spark applications start.

The next question is more interesting:

If I write df.filter(...), why doesn’t Spark immediately process the data?

The answer is lazy evaluation.

That’s the mechanism that allows Spark to build and optimize an execution plan before actually running the computation.

Next: Lazy Evaluation in Apache Spark — Why Your Code Doesn’t Run When You Write It