Skip to content

Incremental loads

A load that reads everything every time is simple and, past a certain size, impossible. An incremental load reads only what is new since the last one. There are two places the pipeline can learn where "new" starts: from the table it writes, or from the system it reads.

A watermark the table holds

If every fact carries a timestamp, the latest one in the table is the watermark: everything after it is new. The load reads the watermark, filters the incoming rows by it, and appends:

type Event {
    id       : Int
    kind     : String
    customer : String
    amount   : Decimal
    at       : String
}
current  : Seq<Event> = source("ledger")
incoming : Seq<Event> = source("silver_replayed")

watermark = current | map(o => o.at) | max

incoming
| filter(o => o.at > watermark.unwrap_or(""))
| map(o => {id: o.id, at: o.at})
[{at: "2026-11-02T11:50:00Z", id: 6}]

The ledger holds the six orders as first reported, so its watermark is order 6's first timestamp, and the only row after it is the correction. max over an empty table is none, and unwrap_or("") makes the first load a full one.

A watermark is only as good as the timestamp: a row that arrives late, with an at before the watermark, is never loaded. The replay has one, the second copy of order 4, which here is harmless because it is an exact duplicate of a row the table holds, and in general is the reason a watermark load is paired with the merge of the previous page, or replaced by a position the reading side keeps.

A position the server keeps

The durable consumer of chapter 17 is an incremental load with no watermark in the program at all. The server records how far the consumer has read, so each run of the load is given only what was published since the run before, and the program is the plain append:

events : Stream<Event> = tap("billing")
events | sink("orders_silver")
Step

A message is acknowledged once the program asks for the next one, so a run that fails is given its last message again, and the silver table it appends to needs the deduplication of this chapter exactly once: at that message. A late message is no problem, because the position is in the stream's order, not the message's timestamp.

The two mechanisms suit the two sides. A watermark is what an analytical table can offer a batch load over another table, where no server keeps a position for it. A consumer is what a transactional system offers a streaming load, and it is the better of the two whenever it is there.

The three loads together

Load Reads Writes Guards against
full the whole source replace nothing; it starts over
incremental by watermark rows after the table's latest timestamp append reading what is already held
incremental by consumer messages after the server's position append or sink reading what is already held, and late arrival

None of them guards against a duplicate on its own; that is the deduplication in the load, or the merge mode of the table. Bronze is loaded in full and never again; silver is loaded incrementally and merged; gold is rebuilt in full from silver. The layers of the first page are the three loads, one each.