Ace Your Databricks Data Engineer Exam: 10 Essential Practice Questions Explained Part 3

In this section, we tackle advanced ingestion with Auto Loader, Spark internals, Delta Live Tables (DLT) architectures, and Databricks Workflows.

Let’s keep the momentum going!

📥 Part 10: Advanced Auto Loader & Streaming

Question 21: Handling Evolving Schemas

A data engineer is using Auto Loader to ingest JSON files. The upstream team frequently adds new columns to the source data. The engineer wants Auto Loader to automatically detect and include these new columns in the target Delta table, but the pipeline should fail if an existing column’s data type changes. Which configuration should they use?

• A. cloudFiles.schemaEvolutionMode = "rescue"
• B. cloudFiles.schemaEvolutionMode = "addNewColumns"
• C. cloudFiles.schemaEvolutionMode = "failOnNewColumns"
• D. cloudFiles.schemaEvolutionMode = "none"

✅ Correct Answer: B

💡 Why it’s correct: By setting the evolution mode to "addNewColumns", Auto Loader automatically updates the schema to include any new fields it discovers in the source files. However, as a safety mechanism, if it detects a type mismatch on an existing column (e.g., an INT changes to a STRING), the stream will fail, alerting the engineer to the breaking change before corrupting the table.

❌ Why the others are wrong:

• A: "rescue" puts new columns into the _rescued_data column rather than evolving the table’s schema.
• C: "failOnNewColumns" will stop the pipeline the moment any new column is detected, which violates the requirement to automatically include new columns.
• D: "none" does not evolve the schema at all; new columns are simply ignored.

Question 22: Managing Late-Arriving Data

A Structured Streaming application calculates windowed aggregations of IoT sensor data. Sometimes, sensor data arrives up to 10 minutes late due to network connectivity issues. To prevent out-of-memory errors (state unbounded growth) while still successfully including this late data in the correct time window, what must the engineer implement?

• A. .trigger(processingTime="10 minutes")
• B. .awaitTermination(600)
• C. .withWatermark("timestamp", "10 minutes")
• D. .dropDuplicates(["timestamp"])

✅ Correct Answer: C

💡 Why it’s correct: A Watermark tells the Spark streaming engine how late data is allowed to be. By setting a 10-minute watermark, Spark keeps the aggregation state in memory for exactly 10 minutes past the maximum event time seen so far. Once the watermark passes, Spark clears the state from memory, preventing out-of-memory (OOM) errors while safely accommodating expected delays.

❌ Why the others are wrong:

• A: Triggers define how often the micro-batch runs, not how to handle late-arriving event time data.
• B: awaitTermination simply keeps the streaming query running; it does not manage state or late data.
• D: dropDuplicates removes identical records; it does not help Spark manage aggregation state memory for late-arriving data.

🛠️ Part 11: Delta Live Tables (DLT)

Question 23: Materialized Views vs. Streaming Tables

In a DLT pipeline, a developer needs to create a table that calculates the total lifetime value (LTV) of customers by aggregating historical sales data (SUM(amount) GROUP BY customer_id). The table should reflect the latest state of the data upon each pipeline refresh. Which DLT dataset type is required?

• A. A Streaming Table
• B. A Materialized View
• C. A View
• D. An External Table

✅ Correct Answer: B

💡 Why it’s correct: Materialized Views (formerly called Live Tables) are designed specifically for transformations that require full recomputation or complex aggregations (like GROUP BY or JOIN). When the pipeline refreshes, the materialized view incrementally or fully computes the correct aggregated state based on the underlying data.

❌ Why the others are wrong:

• A: Streaming Tables are for append-only ingestion and continuous processing. They do not support full recalculations or aggregations over the entire dataset without watermarks.
• C: A standard View does not store physical data; it computes on the fly during a query, which is not efficient for downstream querying of LTV.
• D: External Tables are a Unity Catalog concept for pointing to raw cloud storage, not a DLT dataset type used for transformations.

🧠 Part 12: Spark Internals & Performance

Question 24: Understanding Shuffles

A developer runs a JOIN operation between two large DataFrames (sales and customers) on customer_id. Neither table is partitioned, Z-ordered, or bucketed, and both are over 500GB in size. What internal Spark operation will occur, potentially causing a major performance bottleneck?

• A. Broadcast Hash Join
• B. Shuffle Exchange
• C. Map-Side Combine
• D. Predicate Pushdown

✅ Correct Answer: B

💡 Why it’s correct: Because both tables are massive, Spark cannot broadcast one to all nodes. Instead, it must perform a Sort Merge Join or Shuffled Hash Join. To ensure all records with the same customer_id end up on the same worker node to be joined, Spark must perform a Shuffle Exchange—moving massive amounts of data across the network between worker nodes. This is the most expensive operation in Spark.

❌ Why the others are wrong:

• A: A Broadcast join only happens if one of the tables is very small (usually under 10MB by default).
• C: Map-side combine is an optimization used in aggregations (like reduceByKey), not in standard SQL joins.
• D: Predicate pushdown pushes filters (like WHERE) down to the storage layer; it is an optimization, not a bottleneck.

Question 25: The Photon Engine

A data engineering team is considering enabling the Photon engine on their Databricks clusters. For which of the following workloads will Photon provide the MOST performance benefit?

• A. Heavy Python UDFs (User Defined Functions) processing text
• B. I/O bound streaming workloads blocked by network latency
• C. Heavy SQL aggregations, joins, and built-in function evaluations
• D. Machine learning model training using TensorFlow

✅ Correct Answer: C

💡 Why it’s correct: Photon is Databricks’ native, C++ based vectorized query engine. It is specifically designed to radically accelerate SQL and DataFrame operations—like joins, aggregations, and built-in Spark functions—by optimizing how data is processed at the CPU level.

❌ Why the others are wrong:

• A & D: Photon does not accelerate Python execution (Python UDFs) or non-Spark Machine Learning libraries (like TensorFlow).
• B: Photon speeds up CPU-bound tasks. If a workload is strictly I/O bound (waiting on the network or storage), Photon cannot fix that latency.

🔒 Part 13: Advanced Unity Catalog

Question 26: Unity Catalog Data Lineage

Which statement accurately describes how Unity Catalog handles data lineage?

• A. It tracks lineage at the table level, but column-level lineage requires third-party tools.
• B. It automatically captures lineage for both tables and columns across all supported workloads.
• C. It requires engineers to manually add tags to establish lineage connections.
• D. It only tracks lineage for Python DataFrames, not SQL queries.

✅ Correct Answer: B

💡 Why it’s correct: One of the most powerful features of Unity Catalog is its automated, out-of-the-box lineage tracking. It parses the actual queries running in your workspaces and automatically maps how data flows at both the table level and the column level, regardless of whether you used SQL, Python, R, or Scala.

❌ Why the others are wrong:

• A: UC natively supports column-level lineage.
• C: Lineage is tracked automatically via query parsing; manual tagging is not required.
• D: Lineage works seamlessly across SQL, Python, and other Databricks languages.

Question 27: External Location Permissions

An administrator creates a Unity Catalog External Location pointing to s3://marketing-raw-data/. What permission must be granted to the data_engineers group so they can define new external tables pointing to this exact S3 bucket?

• A. CREATE TABLE on the External Location
• B. CREATE EXTERNAL TABLE on the External Location
• C. READ FILES and WRITE FILES on the Storage Credential
• D. USE CATALOG on the External Location

✅ Correct Answer: B

💡 Why it’s correct: In Unity Catalog, an External Location acts as a governance boundary for a cloud storage path. To create an external table that points to that specific path, a user must be explicitly granted the CREATE EXTERNAL TABLE privilege on that specific External Location object.

❌ Why the others are wrong:

• A: CREATE TABLE is granted on a Schema, not an External Location.
• C: While READ/WRITE FILES grants access to the raw data, it does not grant the right to register the data as an external table in the metastore.
• D: USE CATALOG is a privilege applied to Catalogs, not External Locations.

⏱️ Part 14: Orchestration & Workflows

Question 28: Passing Variables Between Tasks

In Databricks Workflows, Task A is a Python notebook that dynamically calculates a cutoff_date. Task B is a Databricks SQL task that needs to use this cutoff_date in its WHERE clause. How can Task A pass this value to Task B without relying on external storage?

• A. Set a workspace-level environment variable in Task A.
• B. Save the value to a temporary Delta table.
• C. Use dbutils.jobs.taskValues.set() in Task A, and reference it via {{tasks.task_a.values.date}} in Task B.
• D. It is not possible to pass parameters directly between tasks; they must be merged into one notebook.

✅ Correct Answer: C

💡 Why it’s correct: Databricks Workflows supports Task Values, a feature specifically designed to pass lightweight metadata between tasks in a DAG. Task A uses the dbutils library to set the key-value pair, and Task B references it using dynamic parameter formatting {{tasks.<task_name>.values.<key>}}.

❌ Why the others are wrong:

• A: Environment variables are static per cluster/session and cannot be dynamically pushed from one job task to another in real-time.
• B: Writing to a Delta table works, but it incurs unnecessary storage costs and I/O latency. Task Values are the native, in-memory solution.
• D: Passing parameters is a fully supported and standard practice in Databricks Workflows.

🏗️ Part 15: Compute Architectures & Disaster Recovery

Question 29: Serverless SQL Warehouses

A business intelligence (BI) team relies on Databricks to serve dashboards. They need a compute option that minimizes cluster start time to just a few seconds, automatically scales, and requires zero infrastructure management. Which compute type should they choose?

• A. Classic SQL Warehouse
• B. Pro SQL Warehouse
• C. Serverless SQL Warehouse
• D. Job Compute Cluster

✅ Correct Answer: C

💡 Why it’s correct: Serverless SQL Warehouses remove the compute infrastructure from the customer’s cloud account and host it on Databricks’ side. Because Databricks maintains a warm pool of resources, Serverless warehouses start up almost instantly (in seconds), making them the absolute best choice for BI tools (like Tableau or PowerBI) that require immediate responsiveness.

❌ Why the others are wrong:

• A & B: Classic and Pro warehouses spin up VMs in the customer’s cloud account. This VM provisioning process typically takes 3 to 5 minutes, which is unacceptable for a user waiting for a dashboard to load.
• D: Job clusters are ephemeral clusters meant for batch data engineering workloads, not highly concurrent BI serving.

Question 30: Deep Clones for Compliance

A financial institution needs to create an immutable, standalone backup of their transactions table to satisfy a 7-year regulatory compliance rule. The backup must survive even if the original source table is completely dropped and PURGEd. Which command should they use?

• A. CREATE TABLE backup CLONE transactions
• B. CREATE TABLE backup SHALLOW CLONE transactions
• C. CREATE VIEW backup AS SELECT * FROM transactions
• D. OPTIMIZE transactions ZORDER BY (date)

✅ Correct Answer: A

💡 Why it’s correct: A Deep Clone (the default behavior of CLONE) physically copies all the underlying Parquet data files and metadata to a new location. It creates a completely independent table. If the original transactions table is deleted, the backup table remains fully intact, satisfying the compliance requirement.

❌ Why the others are wrong:

• B: A Shallow Clone only copies the metadata. It relies on the source table’s data files. If the source table is dropped/purged, the shallow clone breaks.
• C: A view is just a saved query. If the source table is dropped, the view fails immediately.
• D: OPTIMIZE is a performance maintenance command, not a backup strategy.

📝 Up Next

We’ve now covered 30 core exam concepts! Review the differences between Deep and Shallow clones, memorize your Unity Catalog grants, and make sure you understand exactly when to use Materialized Views vs. Streaming tables.