Logical steps and physical plans¶
An HQL program describes a computation before choosing how to execute it. Its types, operations, data dependencies and ordered effects form the logical program. Planning turns that checked program into work for a particular backend. The physical plan describes how that work will run: which operations share a traversal, where data moves, and which work can run in parallel.
The same HQL expression should be able to run locally, be pushed down into
SQL, or become a distributed Spark plan while keeping the meaning of |.
This is the design contract for alternative backends.
Current implementation and design direction
localhost is the only execution backend in the current release.
Checking, local planning and hql plan work today. Parallel execution,
HQL-to-SQL translation and distributed backends are design direction.
Their examples below explain the intended separation; they are not
commands for selecting an available backend.
Logical steps come before planning¶
Consider two steps over a list:
[3, 1, 2] | filter(n => n > 1) | map(n => n * 10)
[30, 20]
The head supplies three integers. The filter step retains 3 and 2;
the map step multiplies each retained value by ten. The checker establishes
the operations and their types before planning begins. The result has type
List<Int> and preserves the order of the retained elements.
Each | composes the value on its left with the operation on its right.
These two logical steps remain the explanation of the result under every
execution strategy. Their source positions identify computation; the planner
chooses where that computation runs and how much work to combine.
| Term | Meaning | Established by |
|---|---|---|
| Step | One operation or presentation at one position in a pipeline | The program, resolved by the checker |
| Stage | A region of planned work with an execution strategy | The planner |
| Task | One scheduled attempt to execute a stage, over a partition where applicable | The backend's scheduler |
| Job | One run of the whole program's plan | The runtime |
The stage belongs to physical planning. The task and job describe execution
of the plan. The task and job vocabulary here describes the intended model;
the current CLI exposes stage annotations through hql plan.
From the checked program to execution¶
Planning resolves execution choices as far as the selected backend allows. For a local evaluator, it can select in-process evaluation directly. For a SQL backend, it can select a query to submit and delegate the remaining physical decisions to the database engine.
HQL program
-> checking: typed steps, data dependencies, effects
-> backend planning and translation
-> localhost: in-process evaluation
-> parallel backend: work divided among local workers
-> SQL backend: SQL -> engine planning -> physical execution
-> distributed backend: stages -> scheduled tasks on workers
SQL pushdown and distributed execution can be parts of the same route: a Databricks backend could translate HQL to Spark SQL, then let the engine plan and distribute the work. A backend may also refine a plan during a run. The logical program supplies the contract throughout those decisions.
Local, parallel and distributed execution¶
Today, localhost puts all steps of each pipeline in stage 0, marked
in process. For the finite list above, it evaluates the filter and then
the map. hql plan shows the chosen stages without running the program.
For streams, values are consumed on demand, as described in
chapter 7.
An alternative parallel backend could divide the list into parts and apply both operations in one traversal of each part. It would assemble the retained results in their original order. A distributed backend could do the same work on several machines, often where the input already resides. Parallel execution describes work that runs concurrently; distributed execution adds placement and communication across machines.
More demanding operations need other plans. A reduction may combine partial results from several workers; grouping may move values so that equal keys reach the same worker. Several logical steps can share a stage, and the work of one logical step may span several stages. Those choices depend on the operation's contract, the data and the backend's capabilities.
SQL translation and CTEs¶
Design direction. A backend for Databricks could translate a supported region of HQL into Spark SQL. A chain of common table expressions (CTEs) is one way to express the intermediate relations in a single SQL statement. For the small expression above, a possible translation is:
WITH input(pos, n) AS (
VALUES (0, CAST(3 AS BIGINT)),
(1, CAST(1 AS BIGINT)),
(2, CAST(2 AS BIGINT))
), filtered AS (
SELECT pos, n FROM input WHERE n > 1
), mapped AS (
SELECT pos, n * 10 AS value FROM filtered
)
SELECT value FROM mapped ORDER BY pos;
This is an illustrative translation, written by hand. HQL does not generate
it today. The backend would decode the returned column as List<Int>.
The explicit position column and final ordering preserve the HQL list's
order, producing 30 followed by 20.
A CTE
names a result within a SQL statement. CTE dependencies express how the
relations build on one another; their number does not prescribe the number
of physical stages or tasks. Spark plans the submitted SQL and chooses its
physical operators. Its
EXPLAIN EXTENDED
shows the logical plans and the physical plan separately.
Generating SQL therefore completes HQL's translation for that region while leaving physical planning to the engine. A single submitted statement can involve many scheduled tasks. HQL's job describes the whole program run; the engine may organize that work into its own jobs and stages. The backend integration will need to relate those identities when reporting progress.
What every plan must preserve¶
The reference behavior is localhost's result, including its type, sequence
positions and failures. Planning also preserves the order of observable
effects and the scope of transactions. Reading, writing and logging carry
those constraints into the plan.
Pure operations give a planner room to combine or distribute work when their contracts permit it. Data dependencies and failure behavior still apply: changing the order of two pure functions can change the result. A retry of an effectful task also needs the endpoint's delivery guarantees or a transaction protocol that makes the retry valid.
For SQL translation, matching operation names is only a starting point. The backend must preserve HQL's treatment of missing values, numeric precision, ordering and diagnostics. It must reject unsupported work before execution. Which expressions can be translated, and how SQL and local work could share a plan, remain decisions for a future backend specification.
An endpoint and an execution backend¶
The current spark endpoint connects HQL to a table on Databricks or a
Spark Thrift Server. It sends the SQL needed for endpoint operations;
the HQL pipeline's filter, map and grouping still run in hql.
A future Spark execution backend would also run translated pipeline computation on the engine. Endpoint configuration identifies the data; execution configuration selects where the checked computation runs. Chapter 17 uses the existing endpoint, and chapter 15 explains how endpoints are configured.