Spark SQL

Data EngineeringTopic 5 of 123 questions

Q1What is the difference between Flink's DataSet API and DataStream API, and when do you use DataStream vs Table API/Flink SQL

DataSet is the deprecated batch-only low-level API; DataStream is the recommended low-level API unifying streaming and batch; the Table API and Flink SQL are the declarative layer for SQL-like analytics.

Read Answer

Q2Adaptive Query Execution (AQE) and how it complements Catalyst

Catalyst optimizes the plan before execution using estimates, and AQE re-optimizes the physical plan during execution using real runtime statistics at shuffle boundaries : the two work together.

Read Answer

Q3The Catalyst Optimizer

Catalyst turns a DataFrame or SQL query into an efficient plan through parse, analyze, optimize, and physical planning, cutting data read via predicate pushdown and column pruning and speeding execution with whole-stage code generation.

Read Answer