Ace Your Databricks Data Engineer Exam: 10 Essential Practice Questions Explained Part 2

🌊 Part 5: Delta Lake Mechanics

Question 11: Accidental Data Deletion

A junior data engineer accidentally ran a DELETE statement that removed millions of valid rows from a Delta table named sales_prod exactly two hours ago. The table has not been VACUUMed. Which Databricks SQL command should you use to recover the data?

• A. ROLLBACK TABLE sales_prod TO TIMESTAMP AS OF (current_timestamp() - INTERVAL 2 HOURS)
• B. RESTORE TABLE sales_prod TO TIMESTAMP AS OF (current_timestamp() - INTERVAL 2 HOURS)
• C. FSCK REPAIR TABLE sales_prod
• D. ALTER TABLE sales_prod RECOVER PARTITIONS

✅ Correct Answer: B

💡 Why it’s correct: The RESTORE command is the native Delta Lake feature for disaster recovery (Time Travel). Because Delta Lake uses transaction logs and keeps historical parquet files (until a VACUUM occurs), RESTORE instantly reverts the table state to the specified timestamp or version.

❌ Why the others are wrong:

• A: ROLLBACK is not a valid Delta Lake command; it is used in traditional RDBMS systems for uncommitted transactions.
• C: FSCK REPAIR is a legacy Apache Hive command used to sync partition metadata, not to recover deleted data.
• D: RECOVER PARTITIONS is also a legacy Hive command, completely irrelevant to Delta table time travel.

Question 12: Zero-Copy Testing

A data science team wants to test a new data transformation pipeline on the 10TB production customers table. They need full read access to the current data but cannot impact production performance or alter the source data. The testing environment will be torn down tomorrow. Which feature is best suited for this?

• A. Create a Deep Clone of the table
• B. Create a Shallow (Zero-Copy) Clone of the table
• C. Use CREATE TABLE ... AS SELECT * FROM customers
• D. Create a Unity Catalog View

✅ Correct Answer: B

💡 Why it’s correct: A Shallow Clone creates a new Delta table metadata layer that points to the exact same underlying data files as the source table. It takes seconds to create, costs zero additional storage, and any new writes/deletes to the clone will not affect the production table. It is perfect for short-lived experiments.

❌ Why the others are wrong:

• A & C: Both Deep Clone and CTAS will physically copy all 10TB of data. This takes a massive amount of time and doubles cloud storage costs.
• D: A View allows reading, but if the data scientists try to write, update, or test transformations that alter the table state, they cannot do so on a view.

🔄 Part 6: Streaming & Auto Loader

Question 13: Cost-Effective Streaming

A data engineer has a Structured Streaming job ingesting data from Kafka. The source only receives new data once a day at 2:00 AM. They want the pipeline to process all new data as quickly as possible and then automatically shut down the cluster to save costs. Which trigger configuration should they use?

• A. Trigger.ProcessingTime("24 hours")
• B. Trigger.Continuous("1 second")
• C. Trigger.AvailableNow()
• D. Remove the trigger to default to micro-batch processing

✅ Correct Answer: C

💡 Why it’s correct: Trigger.AvailableNow() (which replaced Trigger.Once()) tells Spark to spin up, process all unprocessed data available at the source across multiple micro-batches if necessary, and then automatically terminate the query. Combined with Databricks Job clusters, this seamlessly shuts down the compute once the data is processed.

❌ Why the others are wrong:

• A & D: These keep the streaming cluster running 24/7, just idling and wasting money while waiting for the 2:00 AM data.
• B: Continuous processing is for ultra-low latency (millisecond) requirements and also requires an always-on cluster.

Question 14: Rescuing Bad Data

When configuring Databricks Auto Loader to ingest JSON files, you notice that occasionally, upstream systems send new columns with mismatched data types (e.g., sending a string instead of an integer). How does Auto Loader handle this by default if the _rescued_data column is enabled?

• A. It drops the entire row and logs the error in the driver logs.
• B. It quarantines the corrupted files into a separate dead-letter S3 bucket.
• C. It parses the valid columns normally and stores the unparsed/mismatched data as a JSON string in the _rescued_data column.
• D. It casts the mismatched type to NULL and discards the original value.

✅ Correct Answer: C

💡 Why it’s correct: The _rescued_data column is a safety net in Auto Loader. Instead of failing the job or dropping data, it extracts whatever fields it can map to the known schema and takes the “messy” or mismatched fields (along with the original raw record) and packs them into the _rescued_data column so engineers can inspect and recover them later.

❌ Why the others are wrong:

• A & D: Auto Loader’s primary goal is to prevent data loss. Dropping rows or silently nulling out data violates this.
• B: Auto Loader does not physically move bad files to a different bucket; it handles them at the row/column level during ingestion.

🏗️ Part 7: Delta Live Tables (DLT)

Question 15: Managing Data Quality in DLT

A data engineer is building a Delta Live Tables (DLT) pipeline. They want to ensure that any record where age < 0 is dropped from the dataset, but the pipeline should continue processing all other valid records without failing. Which expectation should they use?

• A. @dlt.expect_or_drop("valid_age", "age >= 0")
• B. @dlt.expect_or_fail("valid_age", "age >= 0")
• C. @dlt.expect("valid_age", "age >= 0")
• D. @dlt.filter("valid_age", "age >= 0")

✅ Correct Answer: A

💡 Why it’s correct: expect_or_drop enforces a data quality constraint. If a record violates the condition (e.g., age is negative), DLT drops that specific row and records the violation in the DLT event log, but allows the rest of the pipeline to run successfully.

❌ Why the others are wrong:

• B: expect_or_fail will immediately halt and fail the entire pipeline the moment it sees a negative age.
• C: expect (without a modifier) simply issues a warning in the event log but keeps the bad data in the table.
• D: filter is a standard PySpark/SQL command, but not the correct syntax for DLT’s built-in data quality expectations.

🔐 Part 8: Unity Catalog Architecture

Question 16: Granting Cloud Storage Access

In Unity Catalog, a data engineer needs to allow a Databricks cluster to securely read raw CSV files stored in an AWS S3 bucket (s3://corp-raw-data/), without embedding AWS access keys in their notebooks. Which two Unity Catalog objects must be created to facilitate this?

• A. A Catalog and a Schema
• B. A Storage Credential and an External Location
• C. A Service Principal and a Personal Access Token (PAT)
• D. A Delta Table and a View

✅ Correct Answer: B

💡 Why it’s correct: In Unity Catalog, secure cloud storage access requires two steps:

1. Storage Credential: Represents the IAM role/Service Principal that has physical access to the cloud bucket.
2. External Location: Combines a specific cloud path (s3://corp-raw-data/) with that Storage Credential, allowing UC to grant fine-grained permissions to users without exposing the underlying keys.

❌ Why the others are wrong:

• A & D: These are data governance layers for structured tables, not the mechanisms for authenticating to raw cloud storage paths.
• C: PATs are for authenticating users/APIs to Databricks, not for Databricks to access AWS/Azure storage.

Question 17: Governing Unstructured Data

A machine learning team needs a governed, centralized place to store and access millions of raw image files (.png, .jpg) and PDF reports. They want to use Unity Catalog to manage permissions for these files. Which feature should they use?

• A. Unity Catalog Managed Tables
• B. Unity Catalog External Tables
• C. Unity Catalog Volumes
• D. Databricks Repos

✅ Correct Answer: C

💡 Why it’s correct: Volumes are the Unity Catalog object designed explicitly for governing non-tabular, unstructured data (like images, PDFs, audio, or raw JSON). They allow you to apply the exact same GRANT/REVOKE permissions you use for tables, but for directories of files.

❌ Why the others are wrong:

• A & B: Tables (Managed or External) are strictly for structured, tabular data (rows and columns), not loose image/PDF files.
• D: Repos (now Git Folders) are for source control (code/notebooks), not large-scale data storage.

🚀 Part 9: Transformation & Optimization

Question 18: Exploding Arrays in PySpark

You have a DataFrame with a column named transactions that contains an array of item IDs (e.g., ["item_1", "item_2"]). Which PySpark function should you use to transform this array so that each item ID gets its own separate row, duplicating the other column values?

• A. split()
• B. flatten()
• C. explode()
• D. pivot()

✅ Correct Answer: C

💡 Why it’s correct: The explode() function takes an array (or map) and creates a new row for each element in the array. This is a crucial function for flattening nested JSON or array data into a standard relational format.

❌ Why the others are wrong:

• A: split() takes a string and turns it into an array based on a delimiter.
• B: flatten() takes an array of arrays and flattens it into a single array, but does not generate new rows.
• D: pivot() reshapes data by turning distinct row values into multiple columns.

Question 19: High Cardinality Performance

A 500GB Delta table is frequently filtered by user_id, which has over 100 million distinct values. The team initially used PARTITION BY (user_id), but queries have become extremely slow, and cloud storage operations are timing out. What is the best optimization strategy?

• A. Keep user_id partitioned but run OPTIMIZE daily
• B. Remove the partition and apply Liquid Clustering on user_id
• C. Partition by user_id and apply Z-Ordering on user_id
• D. Convert the table to Parquet

✅ Correct Answer: B

💡 Why it’s correct: Partitioning on a high-cardinality column (like user_id) causes the Small File Problem. It creates 100 million tiny folders, overwhelming the cloud storage metadata system. Liquid Clustering dynamically manages data layout without rigid partitions, providing excellent read performance on high-cardinality columns without the file overhead.

❌ Why the others are wrong:

• A: OPTIMIZE compacts files within a partition. If a partition only has one tiny file (because there is only one user per partition), OPTIMIZE does nothing.
• C: You cannot Z-Order on the same column you are partitioning by.
• D: Delta Lake is already built on Parquet, but includes metadata/indexing that standard Parquet lacks. Reverting to raw Parquet would make performance worse.

Question 20: Event-Driven Orchestration

A Databricks Workflow job needs to run a data ingestion notebook. However, the job should only trigger exactly when a new raw data file is dropped into an AWS S3/Azure ADLS bucket by an external vendor. What is the most efficient way to trigger this job?

• A. Set a CRON schedule to run every 5 minutes and check for files
• B. Use a File Arrival Trigger
• C. Use a Continuous Trigger
• D. Configure Auto Loader AvailableNow

✅ Correct Answer: B

💡 Why it’s correct: Databricks Workflows supports native File Arrival Triggers. Instead of wasting compute by polling the bucket every 5 minutes, Databricks listens for storage events and automatically triggers the job the moment the file lands, optimizing both latency and compute costs.

❌ Why the others are wrong:

• A: Polling via CRON wastes compute if files don’t arrive, and causes delays if a file arrives right after a check.
• C: Continuous triggers are for low-latency streaming jobs, not event-driven batch ingestion.
• D: Auto Loader AvailableNow is a Spark read configuration inside the notebook, not a mechanism to actually trigger the Databricks Workflow DAG to start.