Subjects and lakehouse tables¶
A nats endpoint is one core NATS subject, a nats_jetstream endpoint is
one subject of a NATS JetStream stream, and a spark endpoint is one table
behind a Spark SQL endpoint: a Databricks SQL warehouse, or an open-source
Spark Thrift Server over Delta Lake tables.
They are configured as a topic and a table
are, and they sit on the same two sides of the line source and tap draw:
a subject is a log that does not end, and a table ends.
| Kind | What it is | Read by | Written by |
|---|---|---|---|
kafka |
one topic | tap, as a Stream |
sink, target |
nats |
one core subject, kept by no one | tap, as a Stream |
sink, target |
nats_jetstream |
one subject of a JetStream stream | tap, as a Stream |
sink, target |
postgres |
one table | source, as a Seq |
target, sink |
spark |
one table of a lakehouse | source, as a Seq |
target, sink |
The programs on this page need a server each, so none of them runs here. Each runs in the hql-demo repository, which starts NATS, MinIO and a Spark Thrift Server with Docker Compose and compares what the programs print with what this page shows.
A core subject¶
NATS delivers a message to whoever is subscribed to its subject when it is
published, and keeps nothing. A nats endpoint names the server and a
subject:
[endpoint.orders]
kind = "nats"
url_env = "NATS_URL"
subject = "orders.>"
idle_ms = 3000
The address is written as url, or held by the environment variable
url_env names. An address may hold a user and a password, and one that does
is refused in .hql/config.toml, so it is never written there.
tap("orders") subscribes when the stream is first traversed, and is given
every message published from then on, in the order it arrives. A message
published before is not there to be given, so a reader must be subscribed
before the writer publishes; nothing is read from the earliest message, and
the endpoint takes no from. A subject may hold the wildcards *, one
token, and >, every token that follows, so orders.> reads every subject
under orders.:
events : Stream<Event> = tap("orders")
events
| filter(e => e.kind == "order_placed")
| map(e => {id: e.id, customer: e.customer, amount: e.amount})
sink publishes each value as one message to one subject, so an endpoint
with a wildcard is read only. Nothing confirms a core message: sink ends
once the server has the messages, whether or not anyone was subscribed.
A subject of a stream¶
JetStream is the part of the server that keeps messages: a stream stores
every message published to the subjects it is configured with, in order. A
nats_jetstream endpoint names the server, the stream, and one subject
within it:
[endpoint.events]
kind = "nats_jetstream"
url_env = "NATS_URL"
stream = "EVENTS"
subject = "events.orders"
idle_ms = 2000
sink("events") publishes each value as one message, a JSON document. A
message is written once the stream answers that it stored it, so a program
that ends without a failure has lost nothing:
type Event {
id : Int
kind : String
customer : String
amount : Decimal
at : String
}
events : Seq<Event> = source("mock_events")
events | sink("events")
A subject that no stream stores is an evaluation failure, not a message that disappears. Creating the stream is the server's business, as creating a topic is a cluster's.
tap("events") reads the stream's messages under that subject in the order
the stream holds them, from the earliest it keeps, or from the next one
published under from = "latest". It is a Stream, and source("events")
is a type error, as it is for a topic:
events : Stream<Event> = tap("events")
events
| filter(e => e.kind == "order_placed")
| map(e => {id: e.id, customer: e.customer, amount: e.amount})
{amount: 120.50, customer: "ada", id: 1}
{amount: 42.00, customer: "grace", id: 2}
{amount: 7.99, customer: "linus", id: 4}
{amount: 15.25, customer: "ada", id: 6}
The read leaves nothing on the server: no position is recorded, so reading a
subject twice gives the same messages twice. Without idle_ms the stream
waits for messages without end; with it, the stream ends once none has
arrived for that many milliseconds.
A subject may hold the wildcards *, one token, and >, every token that
follows. subject = "events.>" reads every subject of the stream under
events.. Such an endpoint is read only: a message is published to one
subject, and sink to a wildcard is an evaluation failure that says so.
A position the server keeps¶
A stream stores messages. A consumer is the server's record of how far one reader has come in a stream, and a durable consumer is one the server keeps under a name. An endpoint that names one reads from where the read before it ended:
[endpoint.billing]
kind = "nats_jetstream"
url_env = "NATS_URL"
stream = "EVENTS"
subject = "events.orders"
consumer = "billing"
idle_ms = 2000
events : Stream<Event> = tap("billing")
events | map(e => e.id)
The first run reads every event the stream keeps:
1
2
3
4
5
6
A second run prints nothing, and a run after two more events are published
prints those two. The first read under a name creates the consumer, starting
where from says; every later read finds it, and from has no effect then.
Two endpoints with different consumers each keep their own position over the
same messages. A consumer keeps one position over one subject, so naming it
with another subject is an evaluation failure.
A message is acknowledged once the program asks for the one after it, and
the last one when the stream ends. A message is therefore delivered at least
once: a program that fails while it holds a message, or that stops early as
take(2) does, is given that message again by the next run.
tap("billing") | take(2) | map(e => e.id)
[1, 2]
The next run of the program above starts with 2.
A subject without configuration¶
Each kind is also a transport type with a constructor, in a module of its
own: Nats and nats(...) in io.msg.nats, NatsJetStream and
nats_jetstream(...) in io.msg.nats_jetstream. A constructor's parameters
are its kind's keys, so a program reaches a subject with no .hql/config.toml:
from io.msg.nats_jetstream import *
orders = nats_jetstream("events.orders", stream = "EVENTS", url_env = "NATS_URL", idle_ms = 2000)
tap(orders) | filter(e => e.kind == "order_placed") | map(e => e.id)
1
2
4
6
A message is one document, so the format may be left out and is then Json.
Both are transports on the network that carry messages, which is what their
supertypes state:
from io.msg.nats import *
nats("orders.order_placed", url_env = "NATS_URL") | is(Network)
Bool
A module is imported by its full path, and naming it in the program's first
lines is how a program states that it reaches NATS:
from io.msg.nats import *. Each module declares only its own transport:
nats(...) takes no stream, and io.msg.nats has no nats_jetstream to
import. The address is
still url or the variable url_env names: a constructor takes no secret,
and a url that holds credentials is refused.
A constructed transport has no schema boundary: fields, mode and
strictness are stated only in configuration. A message is read as Data,
or as the type an annotation names. The boundary of a configured endpoint
writes a Decimal as JSON text so that its digits survive, and reads it
back; to a constructed transport that field is text.
A lakehouse table¶
A lakehouse keeps tables as files in object storage, with a transaction log beside them, and answers SQL over them. Databricks is one; Apache Spark with Delta Lake over an S3 bucket is another, built from open-source parts. Both answer the same protocol at an HTTP address, so one kind reaches either:
[endpoint.events]
kind = "spark"
url_env = "LAKEHOUSE_URL"
table = "default.events"
order = ["id"]
mode = "replace"
[endpoint.events.fields.amount]
type = "DECIMAL(12,2)"
| Where | LAKEHOUSE_URL |
token_env |
|---|---|---|
| A Databricks SQL warehouse | https://<workspace host>/sql/1.0/warehouses/<warehouse id> |
names the variable holding a personal access token |
| A Spark Thrift Server in HTTP mode | http://<host>:10001/cliservice |
left out |
A table name holds its catalog and schema where it has them,
main.sales.events. token is a secret key: written in .hql/config.toml it
refuses the vault, naming token_env.
target("events") writes a Seq or a Table of a declared record. A table
that does not exist is created, a column per field: BIGINT for an Int,
DOUBLE for a Float, DECIMAL(38,18) for a Decimal, BOOLEAN for a
Bool, and STRING otherwise. A DECIMAL column states how many digits it
holds, so the boundary above names the type amount is written as, and the
amounts read back with two decimal places. A list or a record field has no
column type and is a check error:
events : Seq<Event> = source("mock_events")
events | target("events")
Rows are written in statements of up to 500, and each statement is one
commit to the table's log. A target of up to 500 rows is therefore atomic,
and under mode = "replace" it puts its rows in place of the ones the table
held in that same commit, so a reader never sees the table empty.
source("events") reads the table through the
schema boundary, as a postgres table
is read: the columns are compared with the record before any row is decoded,
and only the columns the record reads are selected. A DECIMAL column reads
as a Decimal with its digits and a NULL as an absent field:
events : Seq<Event> = source("events")
events
| filter(e => e.kind == "order_placed")
| group(by = e => e.customer)
| map(g => {customer: g.key, orders: g.items | size, total: g.items | map(e => e.amount) | sum})
{{customer: "ada", orders: 2, total: 135.75}, {customer: "grace", orders: 1, total: 42.00}, {customer: "linus", orders: 1, total: 7.99}}
The endpoint supplies the rows of one table; hql evaluates the filter and
grouping locally. A future execution backend could translate supported
pipeline steps to Spark SQL. Logical steps and physical plans
explains that design direction and how engine planning fits into it.
From a subject to a lakehouse¶
The two meet in the pipeline most platforms have: events arrive on a bus, the valid ones land in a table, the invalid ones go somewhere they can be repaired, and a second table holds what is reported.
producer ──▶ orders.raw ──▶ validate ──ok───▶ orders_silver ──▶ customer_revenue
(subject) │ (Delta table) (Delta table)
└──err──▶ orders.rejected
(subject)
One stream holds both subjects, and two tables hold the rest:
[endpoint.orders_raw]
kind = "nats_jetstream"
url_env = "NATS_URL"
stream = "ORDERS"
subject = "orders.raw"
idle_ms = 2000
[endpoint.orders_rejected]
kind = "nats_jetstream"
url_env = "NATS_URL"
stream = "ORDERS"
subject = "orders.rejected"
idle_ms = 2000
[endpoint.orders_silver]
kind = "spark"
url_env = "LAKEHOUSE_URL"
table = "default.orders_silver"
order = ["id"]
mode = "append"
[endpoint.customer_revenue]
kind = "spark"
url_env = "LAKEHOUSE_URL"
table = "default.customer_revenue"
order = ["customer"]
Streaming. The raw subject is read as Data, so no message fails to
decode. validate
gives one Result per message, and fork routes each to one of two sinks,
in order, ending neither:
type EventData : DataContract = {
id : Int
kind : String
customer : String
amount : Decimal
at : String
require amount >= 0 as non_negative
require ["order_placed", "order_shipped", "payment_failed"] | contains(kind) as known_kind
}
type Rejected {
id : Int
message : Data
rules : List<String>
}
tap("orders_raw")
| validate(EventData)
| fork {
silver =
oks()
| map(e => Event(id = e.id, kind = e.kind, customer = e.customer, amount = e.amount, at = e.at))
| sink("orders_silver"),
rejected =
errs()
| map(e => Rejected(
id = match e.value.id { some(Int n) => n, _ => 0 },
message = e.value,
rules = e.violations | map(v => v.rule)))
| sink("orders_rejected"),
}
sink("orders_silver") appends the rows as they arrive, a commit each 500
rows and one at the end. sink("orders_rejected") publishes each rejected
message as the producer sent it, with the rules it broke, so the dead-letter
subject is itself a stream a later program taps:
rejected : Stream<Rejected> = tap("orders_rejected")
rejected | map(r => {id: r.id, rules: r.rules})
{id: 3, rules: ["EventData.amount.type"]}
{id: 5, rules: ["EventData.customer.missing"]}
{id: 7, rules: ["EventData.non_negative"]}
{id: 8, rules: ["EventData.known_kind"]}
Transformation. The silver table ends, so it is read whole, and the report is written whole over the one before it:
type Revenue {
customer : String
orders : Int
total : Decimal
}
events : Seq<Event> = source("orders_silver")
events
| filter(e => e.kind == "order_placed")
| group(by = e => e.customer)
| map(g => Revenue(customer = g.key, orders = g.items | size, total = g.items | map(e => e.amount) | sum))
| target("customer_revenue")
The two programs share no state but the table between them. The first runs for as long as messages arrive; the second runs when a report is wanted, and reads whatever the first has committed by then.
What is not here¶
One reader reads through one consumer: several programs sharing a consumer to divide its messages is not supported. Credential files for NATS are not read. A lakehouse table holds no nested column and has no constructor yet, and a Databricks endpoint is reached with a personal access token only.