Skip to content

Logical steps and physical plans

An HQL program describes a computation before choosing how to execute it. Its types, operations, data dependencies and ordered effects form the logical program. Planning turns that checked program into work for a particular backend. The physical plan describes how that work will run: which operations share a traversal, where data moves, and which work can run in parallel.

The same HQL expression should be able to run locally, be pushed down into SQL, or become a distributed Spark plan while keeping the meaning of |. This is the design contract for alternative backends.

Current implementation and design direction

localhost is the only execution backend in the current release. Checking, local planning and hql plan work today. Parallel execution, HQL-to-SQL translation and distributed backends are design direction. Their examples below explain the intended separation; they are not commands for selecting an available backend.

Logical steps come before planning

Consider two steps over a list:

[3, 1, 2] | filter(n => n > 1) | map(n => n * 10)
[30, 20]

The head supplies three integers. The filter step retains 3 and 2; the map step multiplies each retained value by ten. The checker establishes the operations and their types before planning begins. The result has type List<Int> and preserves the order of the retained elements.

Each | composes the value on its left with the operation on its right. These two logical steps remain the explanation of the result under every execution strategy. Their source positions identify computation; the planner chooses where that computation runs and how much work to combine.

Term Meaning Established by
Step One operation or presentation at one position in a pipeline The program, resolved by the checker
Stage A region of planned work with an execution strategy The planner
Task One scheduled attempt to execute a stage, over a partition where applicable The backend's scheduler
Job One run of the whole program's plan The runtime

The stage belongs to physical planning. The task and job describe execution of the plan. The task and job vocabulary here describes the intended model; the current CLI exposes stage annotations through hql plan.

From the checked program to execution

Planning resolves execution choices as far as the selected backend allows. For a local evaluator, it can select in-process evaluation directly. For a SQL backend, it can select a query to submit and delegate the remaining physical decisions to the database engine.

HQL program
    -> checking: typed steps, data dependencies, effects
    -> backend planning and translation
        -> localhost: in-process evaluation
        -> parallel backend: work divided among local workers
        -> SQL backend: SQL -> engine planning -> physical execution
        -> distributed backend: stages -> scheduled tasks on workers

SQL pushdown and distributed execution can be parts of the same route: a Databricks backend could translate HQL to Spark SQL, then let the engine plan and distribute the work. A backend may also refine a plan during a run. The logical program supplies the contract throughout those decisions.

Local, parallel and distributed execution

Today, localhost puts all steps of each pipeline in stage 0, marked in process. For the finite list above, it evaluates the filter and then the map. hql plan shows the chosen stages without running the program. For streams, values are consumed on demand, as described in chapter 7.

An alternative parallel backend could divide the list into parts and apply both operations in one traversal of each part. It would assemble the retained results in their original order. A distributed backend could do the same work on several machines, often where the input already resides. Parallel execution describes work that runs concurrently; distributed execution adds placement and communication across machines.

More demanding operations need other plans. A reduction may combine partial results from several workers; grouping may move values so that equal keys reach the same worker. Several logical steps can share a stage, and the work of one logical step may span several stages. Those choices depend on the operation's contract, the data and the backend's capabilities.

SQL translation and CTEs

Design direction. A backend for Databricks could translate a supported region of HQL into Spark SQL. A chain of common table expressions (CTEs) is one way to express the intermediate relations in a single SQL statement. For the small expression above, a possible translation is:

WITH input(pos, n) AS (
    VALUES (0, CAST(3 AS BIGINT)),
           (1, CAST(1 AS BIGINT)),
           (2, CAST(2 AS BIGINT))
), filtered AS (
    SELECT pos, n FROM input WHERE n > 1
), mapped AS (
    SELECT pos, n * 10 AS value FROM filtered
)
SELECT value FROM mapped ORDER BY pos;

This is an illustrative translation, written by hand. HQL does not generate it today. The backend would decode the returned column as List<Int>. The explicit position column and final ordering preserve the HQL list's order, producing 30 followed by 20.

A CTE names a result within a SQL statement. CTE dependencies express how the relations build on one another; their number does not prescribe the number of physical stages or tasks. Spark plans the submitted SQL and chooses its physical operators. Its EXPLAIN EXTENDED shows the logical plans and the physical plan separately.

Generating SQL therefore completes HQL's translation for that region while leaving physical planning to the engine. A single submitted statement can involve many scheduled tasks. HQL's job describes the whole program run; the engine may organize that work into its own jobs and stages. The backend integration will need to relate those identities when reporting progress.

What every plan must preserve

The reference behavior is localhost's result, including its type, sequence positions and failures. Planning also preserves the order of observable effects and the scope of transactions. Reading, writing and logging carry those constraints into the plan.

Pure operations give a planner room to combine or distribute work when their contracts permit it. Data dependencies and failure behavior still apply: changing the order of two pure functions can change the result. A retry of an effectful task also needs the endpoint's delivery guarantees or a transaction protocol that makes the retry valid.

For SQL translation, matching operation names is only a starting point. The backend must preserve HQL's treatment of missing values, numeric precision, ordering and diagnostics. It must reject unsupported work before execution. Which expressions can be translated, and how SQL and local work could share a plan, remain decisions for a future backend specification.

An endpoint and an execution backend

The current spark endpoint connects HQL to a table on Databricks or a Spark Thrift Server. It sends the SQL needed for endpoint operations; the HQL pipeline's filter, map and grouping still run in hql.

A future Spark execution backend would also run translated pipeline computation on the engine. Endpoint configuration identifies the data; execution configuration selects where the checked computation runs. Chapter 17 uses the existing endpoint, and chapter 15 explains how endpoints are configured.