The shop's day in one file¶
Orders arrive on the JetStream subject orders.raw. Each is appended to an
archive file as it arrived. Each is validated: the valid ones land in the
silver Delta table, and a placed order of 100 or more is also published as
an alert to a core NATS subject; the invalid ones go to the dead-letter
subject with the rules they broke. The configuration is the shop's
.hql/config.toml; the pipeline, day.hql, is this:
type EventData : DataContract = {
id : Int
kind : String
customer : String
amount : Decimal
at : String
require amount >= 0 as non_negative
require ["order_placed", "order_shipped", "payment_failed"] | contains(kind) as known_kind
}
type Event {
id : Int
kind : String
customer : String
amount : Decimal
at : String
}
type Rejected {
id : Int
message : Data
rules : List<String>
}
tap("orders_raw")
| tee archived = sink(file("archive/orders.jsonl").end, Json)
| validate(EventData)
| fork {
silver =
oks()
| map(e => Event(id = e.id, kind = e.kind, customer = e.customer, amount = e.amount, at = e.at))
| tee large =
filter(e => e.kind == "order_placed" and e.amount >= 100)
| map(e => {customer: e.customer, amount: e.amount, text: fmt("large order from {}", e.customer)})
| sink("alerts")
| sink("orders_silver"),
rejected =
errs()
| map(e => Rejected(
id = match e.value.id { some(Int n) => n, _ => 0 },
message = e.value,
rules = e.violations | map(v => v.rule)))
| sink("orders_rejected"),
}
{silver: Step, rejected: Step}
Read top to bottom, it is the dataflow diagram a platform would draw:
orders.raw ──┬──▶ archive/orders.jsonl
│
└──▶ validate ──ok──┬──▶ orders_silver
│ └──▶ alerts.orders (≥ 100, placed)
└──err─────────▶ orders.rejected
One input, three kinds of fan-out, four outputs. tee branches a pipeline
without ending it: the archive receives every order before validation, and
the alert branch receives each valid order before the silver table does.
fork sends every value to every branch, and oks() and errs() keep one
side each. A branch of either is typed like the trunk, so a branch that
ended in target over this stream would be refused, as any target of a
stream is.
There is no orchestration around the file: no activity names, no cluster,
no connection strings. hql check day.hql reads .hql/config.toml, resolves every
id to its kind and every kind to a type, checks the whole file, and opens
no connection; the result above is what it prints. Every system in the file
is in the current release, and the
nats-lakehouse case
of hql-demo runs the same shape against NATS and a lakehouse.
The gold report of chapter 17 is the second file, and the subject of chapter 20: it reads the silver table whole when a report is wanted, and replaces the gold table in one commit. The two share no state but the silver table.