Skip to content

A replay is the same file

The pipeline names one input. Bound to a function, it takes any stream of Data, and the input is chosen where it is called:

type EventData : DataContract = {
    id       : Int
    kind     : String
    customer : String
    amount   : Decimal
    at       : String

    require amount >= 0 as non_negative
    require ["order_placed", "order_shipped", "payment_failed"] | contains(kind) as known_kind
}
route = (raw: Stream<Data>) => raw
| validate(EventData)
| fork {
    clean    = oks()  | map(e => fmt("ok   {}", e.id)) | sink(stdout, Text),
    rejected = errs() | map(e => fmt("err  {} {}", e.value.id, e.violations | map(v => v.rule))) | sink(stdout, Text),
}

tap("raw_events") | route
ok   1
ok   2
ok   4
ok   6
err  3 ["EventData.amount.type"]
err  5 ["EventData.customer.missing"]
err  7 ["EventData.non_negative"]
err  8 ["EventData.known_kind"]
{clean: (), rejected: ()}

tap over a file is a Stream that ends after the file's last order, so the routing written for a subject runs over a file of recorded messages, with no broker at all. That is a test. The archive day.hql writes is such a file, so the same call over tap(file("archive/orders.jsonl")) is a replay: last week's orders through this week's rules. And the call over tap("orders_raw") is production. The platform words, test, replay, backfill, name three inputs to one pipeline, not three pipelines.

The other direction is as short. A stream becomes something finite by bounding it: take(n) for a count, collect(n) for batches of a size, each batch a List every whole-input step of chapter 6 applies to:

tap("orders_raw") | take(100) | size
Int

A snapshot is the first hundred, or the first day, of a stream written whole by target; a micro-batch is collect(n) and one write per batch.