17. Subjects and the lakehouse¶
The shop's Kafka topic of chapter 16 carries every order, and three needs have grown around it. The warehouse wants to hear about each shipment as it happens and does not care about the ones it missed. Billing wants every order exactly where it left off yesterday. And the owner wants reports over all of history without anyone's pipeline slowing down to produce them.
This chapter meets the three with NATS and a lakehouse. NATS is two kinds:
nats, a subject whose messages nobody keeps, and nats_jetstream, a
subject of a stream the server keeps. A lakehouse is tables kept as files
in object storage, reached by kind = "spark". Each key of each kind is on
the subjects and lakehouse tables page
of chapter 9; this chapter is what the shop does with them.
The programs need servers. They are checked here against the shop's
configuration, and run in the
hql-demo repository, whose
nats-core, nats and nats-lakehouse cases start NATS, MinIO and a Spark
Thrift Server with Docker Compose. The outputs shown are theirs.
type Event {
id : Int
kind : String
customer : String
amount : Decimal
at : String
}
Notifications nobody keeps¶
Core NATS delivers a message to whoever is subscribed to its subject when it is published, and keeps nothing. That is what a notification is: the warehouse screen shows what is happening now, and an event from before the screen was switched on is not news.
The shop publishes each event to a subject named for its kind,
orders.order_placed, orders.payment_failed, orders.order_shipped. The
subject is computed, so the program writes the transport itself, with the
constructor of io.msg.nats:
from io.msg.nats import *
type Event {
id : Int
kind : String
customer : String
amount : Decimal
at : String
}
events : Seq<Event> = source("events")
for_kind = (kind: String) => events | filter(e => e.kind == kind)
subject = (kind: String) => nats("orders." + kind, url_env = "NATS_URL")
for_kind("order_placed") | sink(subject("order_placed"))
for_kind("payment_failed") | sink(subject("payment_failed"))
for_kind("order_shipped") | sink(subject("order_shipped"))
Step
A reader names one subject or a family of them. A subject may hold the
wildcards *, one token, and >, every token that follows, so the shop's
notifications endpoint reads all three:
[endpoint.notifications]
kind = "nats"
url_env = "NATS_URL"
subject = "orders.>"
idle_ms = 3000
events : Stream<Event> = tap("notifications")
events
| filter(e => e.kind == "order_placed")
| map(e => {id: e.id, customer: e.customer, amount: e.amount})
{amount: 120.50, customer: "ada", id: 1}
{amount: 42.00, customer: "grace", id: 2}
{amount: 7.99, customer: "linus", id: 4}
{amount: 15.25, customer: "ada", id: 6}
The subscription starts when the stream is first traversed, so the
nats-core case starts this program before the publisher, and a message
published before it would not be there to be given. Nothing confirms a core
message: sink ends once the server has the messages, whether or not
anyone was listening. A wildcard endpoint is read only, since a message is
published to one subject.
A log the server keeps¶
Billing cannot miss an order. JetStream is the part of the NATS server that
keeps messages: a stream stores every message published to its
subjects, in order, and a nats_jetstream endpoint names the stream and one
subject in it. The shop's checkout publishes raw orders to orders.raw, in
the stream ORDERS:
[endpoint.orders_raw]
kind = "nats_jetstream"
url_env = "NATS_URL"
stream = "ORDERS"
subject = "orders.raw"
idle_ms = 2000
source("raw_events") | sink("orders_raw")
Step
A message is written once the stream answers that it stored it, so a program that ends without a failure has lost nothing. A subject no stream stores is an evaluation failure rather than a message that disappears.
tap("orders_raw") reads the stream's messages under that subject in the
order the stream holds them, from the earliest it keeps, or from the next
one published under from = "latest". Like a Kafka read, it records no
position: two runs give the same messages twice.
Billing's place¶
A consumer is the server's record of how far one reader has come in a stream, and a durable consumer is one the server keeps under a name. Billing's endpoint is the same subject with a consumer named:
[endpoint.billing]
kind = "nats_jetstream"
url_env = "NATS_URL"
stream = "ORDERS"
subject = "orders.raw"
consumer = "billing"
idle_ms = 2000
The first run through billing reads every message the stream keeps. The
next run reads only what was published since; a run with nothing new
prints nothing. Another endpoint with another consumer keeps its own place
over the same messages, so the warehouse's reports and billing's invoices
never move each other's position.
A message is acknowledged once the program asks for the one after it, and the last one when the stream ends. A message is therefore delivered at least once: a program that fails while it holds a message, or stops early, is given that message again by its next run.
tap("billing") | take(2) | map(e => e.id)
[1, 2]
take(2) asked for no third message, so 2 was never acknowledged, and the
next run of this program starts with 2. A program that writes what it
reads, an invoice per order, writes it so that writing it twice does no
harm.
The lakehouse¶
The owner's reports need history, and a broker is not where history is
queried. A lakehouse keeps tables as files in object storage, with a
transaction log beside them, and answers SQL over them. Databricks is one;
Apache Spark with Delta Lake over an S3 bucket is another, built from
open-source parts, and the spark kind reaches either, by the URL of a
Databricks SQL warehouse or of a Spark Thrift Server:
[endpoint.orders_silver]
kind = "spark"
url_env = "LAKEHOUSE_URL"
table = "default.orders_silver"
order = ["id"]
mode = "append"
[endpoint.orders_silver.fields.amount]
type = "DECIMAL(12,2)"
A lakehouse table ends, so it sits on the table side of the line: read by
source through the schema boundary, written by target in commits of up
to 500 rows, or by sink as rows arrive. A DECIMAL column states how
many digits it holds, so the boundary names the type each amount is written
as. The valid orders of the subject above land in this table by the
quarantine of chapter 16 with the ids changed, a subject in place of the
topic and this table in place of the clean one:
type EventData : DataContract = {
id : Int
kind : String
customer : String
amount : Decimal
at : String
require amount >= 0 as non_negative
require ["order_placed", "order_shipped", "payment_failed"] | contains(kind) as known_kind
}
type Rejected {
id : Int
message : Data
rules : List<String>
}
tap("orders_raw")
| validate(EventData)
| fork {
silver =
oks()
| map(e => Event(id = e.id, kind = e.kind, customer = e.customer, amount = e.amount, at = e.at))
| sink("orders_silver"),
rejected =
errs()
| map(e => Rejected(
id = match e.value.id { some(Int n) => n, _ => 0 },
message = e.value,
rules = e.violations | map(v => v.rule)))
| sink("orders_rejected"),
}
{silver: Step, rejected: Step}
That table is the shop's first analytical one, and what it is for, the layers a lakehouse is organized in, and what a report built over it needs from it are chapter 20. This chapter's business with it ends at the transport: a subject in, a table out, and what each promises.
Where the work runs¶
The filter and the grouping above run where hql runs. The spark endpoint
reaches a table over Spark SQL and sends the statements needed to read or
write it. The surrounding HQL pipeline is evaluated locally.
A future Spark execution backend could translate supported pipeline steps
into SQL and let the engine plan their execution on the cluster. The
companion Logical steps and physical plans
explains that handoff and the requirement to preserve the meaning of |.
What the current release does not do: several programs sharing one consumer to divide its messages, NATS credential files, a lakehouse column holding a list or a record, and a Databricks token other than a personal access token.
What is next¶
Chapter 18 is where the shop's data comes from outside: a payment provider calling a webhook, a carrier's REST API, and an archive in object storage. It is design, not yet in the release.