Streams against tables¶
Design only
This section describes functionality that is not in the current release. It is written as if it existed: these programs do not run, and names and types may change before they ship.
The commonest enrichment is a stream against a lookup: each order joined to
its customer's record in a table. The finite side is read whole; the live
side passes through one value at a time, and no value waits for another, so
join takes a Stream on the left and a Seq on the right:
type Customer { customer : String, region : String, vip : Bool }
customers : Seq<Customer> = source("customers")
tap("orders_raw")
| Event
| join(customers, left = o => o.customer, right = c => c.customer,
into = (o, c) => {id: o.id, region: c.region, vip: c.vip, amount: o.amount})
| filter(x => x.vip)
| sink("vip_orders")
Two streams joined would wait for each other without end, and the checker
says so: join over two Streams is refused until one of them is bounded.
A window is collect by time instead of by count, a
Stream<List<T>> whose batches close when the span does:
tap("orders_silver_changes")
| window(1d, by = e => e.at)
| map(day => day
| group(by = e => e.customer)
| map(g => Revenue(customer = g.key, orders = g.items | size, total = g.items | map(e => e.amount) | sum)))
| emit
| sink("daily_revenue")
Inside a window every whole-input step applies, which is how a daily total
is a group and a sum over an input that never ends.