Skip to content

19. OLTP

The shop's systems so far are its OLTP side: online transaction processing, the side where things happen. An order is placed and a row is committed; a payment fails and a message is published; the warehouse reads the next shipment and acknowledges it. Each fact is one row or one message, written once as it occurs, and what the system holds is the current state of the business: these orders, these customers, this stock. The PostgreSQL table, the Kafka topic and the JetStream subject of chapters 16 and 17 are all of this kind, and so is the first lakehouse table the raw orders land in.

The pipelines of this side share a shape. They read one value at a time, tap, and write one value at a time, sink, as each arrives; they decide each value on its own, validate, filter, map, and keep nothing between two values; and they run for as long as values arrive. The guarantees they rest on are the ones of chapter 16 and chapter 17: a row committed as it is written, a message stored before the write ends, a message delivered at least once and so written so that twice does no harm.

This chapter is the shop's whole transactional day as one file, and what follows from its being a file. The other side, the history those transactions leave and the questions a report asks of it, is chapter 20.

The pages

Page What it covers
The shop's day in one file one pipeline from the raw subject to an archive, a silver table, an alert subject and a dead-letter subject, checked against the configuration with nothing connected
When a mistake is found the phase each broken binding is found in, and what is said
A replay is the same file the pipeline as a function of its input: a test over a file, a replay over the archive, production over the subject, and a stream bounded into a batch
Streams against tables enrichment of a stream by a lookup table, and windows, as design
A transfer is one transaction insert, update and delete changing rows in place, and with transaction making several of them one transaction, with its rollback, its retries and its failure as data

What is next

Chapter 20 turns from the transactions to what they leave behind: bronze, silver and gold tables, the grain of a row, duplicates and how they are removed, and a table kept up to date by key.