Databricks Certified Associate Developer for Apache Spark practice questions

Question 1

A logistics analytics team is migrating a join between two continuous streaming sources to Structured Streaming, and a developer must decide how Databricks classifies that join and what it implies for retained state. Per the Databricks joins documentation, which option matches?

Accepted Answer

It is a stateful join, so Databricks tracks information about the two sources and iteratively updates the results as new records arrive.

Answer

It is a stateless join, because every join processes immediately and only ever reflects the exact source data present at the instant the query is run.

Answer

It is a batch join, so the engine recomputes the full result from scratch on each micro-batch and keeps no information between runs.

Answer

It is a broadcast join, so the smaller stream is cached in memory once and the engine never retains any prior records from either side.

Question 2

A retail analytics team is debugging a Spark join between two large tables where the query appears stuck finishing the last few tasks, and an engineer must decide whether data skew is the likely cause. Per the Databricks skew join optimization documentation, which option best matches?

Accepted Answer

Skew is likely, since joins shuffle data and skew creates an extreme imbalance, leaving a query stuck on a few final tasks.

Answer

A stage stuck on its final tasks usually means the driver lost heartbeat with the cluster manager rather than data skew.

Answer

Skew mainly affects narrow transformations like filters, so a stuck join points to an undersized broadcast threshold.

Answer

Skew typically surfaces as driver out-of-memory errors rather than a stage where a few tasks run far longer than all the rest.

Question 3

A retail analytics team is designing a Spark SQL report that PIVOTs sales rows using two aggregate expressions (a SUM and an AVG) over the same column_list, and a developer needs to know how Spark names the generated pivot columns. Per the Databricks PIVOT documentation, which option matches the column-naming behavior?

Accepted Answer

Because two aggregate expressions are used, each pivot column is named column_alias_agg_column_alias, keeping SUM and AVG distinct.

Answer

Because two aggregate expressions are used, every pivot column is named with just the column_alias, so the SUM and AVG values collide.

Answer

Because two aggregate expressions are used, Spark rejects the query, since PIVOT accepts at most one aggregate expression.

Answer

Because two aggregate expressions are used, only grouping columns survive and aggregate columns keep the bare column_alias.

Question 4

A media stream analytics team is migrating a streaming write step to Structured Streaming and a developer must decide how to write each micro-batch to multiple sinks at once and apply non-idempotent operations. Per the Databricks Structured Streaming watermarks documentation, which option matches the documented use case for foreachBatch?

Accepted Answer

Foreachbatch allows applying arbitrary DataFrame operations to each micro-batch output, enabling complex write patterns like writing to multiple sinks or applying non-idempotent operations.

Answer

foreachBatch sends every micro-batch to the Spark UI only, where the developer can manually approve or reject the batch before any sink-side write is executed on a production cluster.

Answer

foreachBatch is a deprecated function on Databricks Structured Streaming and developers must use writeStream.format('kafka') with multiple option keys to fan out a micro-batch to several sinks at once.

Answer

foreachBatch automatically routes the streaming output to exactly one Delta sink based on the sink's path; multi-sink writes and non-idempotent operations are not part of its documented capabilities.

Question 5

An IoT telemetry processing team is migrating a production batch ETL job and a developer must decide how to recover from a recurring driver OutOfMemoryError that appears whenever .collect() is called on a large DataFrame. Per the Databricks AQE optimizations documentation, which option matches the documented anti-pattern and remediation?

Accepted Answer

Calling collect() or toPandas() on large DataFrames pulls all data to the driver, often causing OutOfMemoryError; instead use take(n), show(n), or write results to storage.

Answer

Driver OOM from collect() is expected behaviour in modern Spark and the documented remediation is to triple the driver memory and disable AQE so the optimizer does not generate a final shuffle to the driver process.

Answer

Driver OOM from collect() should be remediated by raising spark.executor.memory; the driver process is not actually involved in collect() and increasing executor memory is the only published lever for this anti-pattern.

Answer

Driver OOM from collect() must be remediated by switching to RDD APIs; DataFrame.collect() is the only documented driver-side action and cannot be replaced with take(n) or show(n) in a published Databricks runtime.

Question 6

A developer is preparing a clean local Python virtual environment to build an application with Databricks Connect, but the install instructions warn about a package conflict. Per the Databricks Connect installation documentation, what must the developer do before installing the databricks-connect client package?

Accepted Answer

Uninstall any existing PySpark, since databricks-connect conflicts with a local PySpark installation.

Answer

Install PySpark first with pip3 so Databricks Connect can reuse its bundled local Spark JARs at runtime.

Answer

Remove the Databricks CLI so its embedded Spark session does not shadow the databricks-connect client package.

Answer

Export the SPARK_REMOTE variable first so pip can resolve the databricks-connect client against the cluster.

Question 7

A data science team is converting a Spark DataFrame to a pandas-on-Spark DataFrame with pandas_api(), then converting it back to Spark and again to pandas-on-Spark inside an iterative feature-engineering loop. A developer is concerned about what happens to the index across that cycle. Per the Databricks pandas_api documentation, what is the documented behaviour of the index across this round trip?

Accepted Answer

The index information is lost on the round trip, and the original index is turned into a normal column in the result.

Answer

The index column is preserved automatically because pandas_api() stores the original index as hidden table metadata in Spark.

Answer

The conversion raises an error unless an index_col argument is passed to pandas_api() on every call inside the loop.

Answer

The index is retained only while Arrow execution is enabled; otherwise the whole DataFrame is collected to the driver.

Question 8

A media stream analytics team is migrating multi-dimensional reporting to a single Spark SQL query and a developer must decide which operators produce subtotals and grand totals without multiple UNION ALL passes. Per the Databricks PySpark basics documentation, which option matches?

Accepted Answer

The grouping sets, cube, and rollup operations generate subtotals and grand totals for multi-dimensional aggregation analysis.

Answer

Multi-dimensional aggregations require the developer to write nested groupBy().count().unionByName() chains; Spark SQL does not provide rollup, cube, or grouping sets in the documented PySpark functions module.

Answer

Multi-dimensional aggregations are implemented through pivot only; the rollup and cube operators were removed from Spark SQL in favour of the unified pivot DSL in Databricks Runtime 13.x.

Answer

Multi-dimensional aggregations require an external BI tool such as Tableau or PowerBI; Spark SQL exposes no built-in operators for subtotal or grand-total calculation against a DataFrame on Databricks clusters.

Question 9

A retail real-time analytics team is designing a join between a large fact DataFrame and a small lookup table and an engineer is being asked what broadcast variables actually do in Spark. Per the Databricks Spark concepts documentation, which option matches?

Accepted Answer

Broadcast variables allow the driver to send a read-only copy of a shared variable to each executor once, avoiding the cost of shipping it with every task, which is particularly useful for joining a large DataFrame with a small lookup table.

Answer

Broadcast variables allow the driver to mutate a shared variable from any executor concurrently, providing distributed locks that synchronize state across all tasks during wide transformations.

Answer

Broadcast variables stream a continuously updated value from the driver to executors over a Kafka topic, so each task pulls the latest snapshot before it begins to process its input partition.

Answer

Broadcast variables enforce per-executor write quotas based on the spark.executor.broadcast.maxMB property and reject any task that attempts to update the shared variable from inside the executor JVM.

Question 10

A media stream analytics team is migrating a production batch ETL job to Spark SQL and a developer must decide how to limit which files Spark SQL reads from a large nested S3 directory hierarchy. Per the Databricks PySpark basics documentation, which option matches the file-based reader configuration?

Accepted Answer

For file-based sources, the pathGlobFilter option limits which files are read using glob patterns, and recursiveFileLookup traverses nested directory structures.

Answer

Spark file-based sources support only the path option and do not provide any way to filter by glob, by directory depth, or by file modification timestamp at read time on Databricks Runtime.

Answer

Spark file-based sources require a separate Auto Loader job that pre-curates the file list into a manifest, which DataFrameReader then consumes; glob filtering and recursive directory scans are not part of DataFrameReader itself.

Answer

Spark file-based sources expose a fileFilterRegex option that accepts arbitrary regular expressions, but recursive directory traversal must always be implemented in client code by calling dbutils.fs.ls and passing the discovered paths individually.

Question 11

A streaming ingestion team is migrating a high-volume Kafka pipeline onto Databricks and a developer must decide how to cap how many records are pulled into each micro-batch so a backlog burst cannot overwhelm the cluster. Per the Structured Streaming index documentation, which readStream option applies backpressure?

Accepted Answer

maxOffsetsPerTrigger, which caps the number of records pulled into each micro-batch to apply backpressure.

Answer

startingOffsets, which sets where the Kafka source begins reading but does not bound the records per micro-batch.

Answer

processingTime, which sets the micro-batch trigger interval but does not limit how many records each batch reads.

Answer

minOffsetsBehindLatest, which reports per-partition offset lag but does not throttle the records read per batch.

Question 12

A retail data engineering pipeline is being optimized for query latency on large Delta Lake tables, and a developer must decide which statements about dynamic file pruning on Databricks are accurate. Per the Databricks dynamic file pruning documentation, select the TWO correct statements.

Accepted Answer

The query optimizer triggers dynamic file pruning for queries that contain filter statements or WHERE clauses on the table.

Accepted Answer

Dynamic file pruning in MERGE, UPDATE, and DELETE statements requires Photon-enabled compute in order to take effect at runtime.

Answer

Dynamic file pruning only works on plain Parquet tables and is automatically disabled whenever it runs against Delta Lake tables.

Answer

Dynamic file pruning requires Adaptive Query Execution to be turned off so the static plan can prune the data files up front.

Question 13

A platform team is migrating from Spark Classic to Spark Connect for a retail real-time analytics application and a developer is being asked when name resolution and type checking happen. Per the Databricks Spark Connect vs Classic documentation, which option matches?

Accepted Answer

The client constructs unresolved logical plans locally and sends them to the server for analysis and execution, unlike Spark Classic which performs eager analysis during plan construction.

Answer

Spark Connect performs eager analysis on the client and rejects any reference to a nonexistent column at the moment the developer writes the DataFrame transformation in code, identical to Spark Classic semantics.

Answer

Spark Connect performs eager analysis on the server every time a transformation is added to the DataFrame, with a round-trip gRPC call against the Databricks cluster between every method invocation in the application code.

Answer

Spark Connect skips analysis entirely; type checking and name resolution are deferred indefinitely and only validated through external static analysis tooling such as MyPy or PyCharm on the developer's machine.

Question 14

A platform team is being asked how the Pandas API on Spark differs from native pandas when running a data ops workload against a multi-terabyte healthcare dataset. Per the Databricks pandas-on-spark documentation, which option matches the documented behaviour?

Accepted Answer

The Pandas API on Spark (pyspark.pandas) provides pandas-compatible APIs that run on Apache Spark, enabling users familiar with pandas to scale their workloads to distributed clusters.

Answer

The Pandas API on Spark is a thin wrapper around CPython's pandas package that runs entirely inside the Spark driver process and offers no documented scaling benefit over native pandas in a Databricks notebook.

Answer

The Pandas API on Spark is a synonym for koalas and is deprecated in every modern Databricks runtime; new applications must use only the PySpark DataFrame API to access distributed pandas-style processing.

Answer

The Pandas API on Spark is a SQL-only DSL exposed exclusively through spark.sql('pandas.DataFrame(...)'); it cannot be imported as pyspark.pandas in any published Databricks runtime since 14.x.

Question 15

A data-warehouse modernization team is migrating analytical tables to PySpark and a developer must decide which file format lets the engine read only the needed columns and skip irrelevant row groups. Per the Databricks Parquet documentation, which option matches?

Accepted Answer

Parquet, a columnar format that reads only the needed columns and skips irrelevant row groups

Answer

CSV, a row-oriented text format that forces the engine to scan every field of every line per query

Answer

JSON, a semi-structured text format whose nested records must be parsed in full before reading a column

Answer

Avro, a row-based binary format built mainly for record-by-record streaming serialization across systems

Databricks Certified Associate Developer for Apache Spark

Sample questions

Sources

Similar exams