Skip to content

18. APIs, webhooks and stores

Design only

This chapter describes functionality that is not in the current release. It is written as if it existed: apart from http, websocket and tcp, which the network page of chapter 9 shows running, these programs do not run against the current binary, and names, keys, types and outputs may change before they ship.

Everything the shop has read so far, it wrote first: its own files, tables, topics and subjects. A business also reads what others publish. The payment provider calls the shop when a payment settles. The carrier answers questions about parcels, a page of answers at a time. The supplier's catalogue is a GraphQL API. And the auditor wants every order kept for seven years, somewhere cheaper than a lakehouse.

Each of those is one more transport, and the model of chapter 15 does not change for any of them: an endpoint by id or by constructor, read by source if it ends and by tap if it does not, written by target or sink, its type known from configuration before anything connects.

Three transports over HTTP

http(url) in the current release reads one resource as bytes, with a format to say what the bytes are. Most APIs are more than one resource: a collection split into pages, a query posted to one address, or a server that calls back. Three transports of io.net name those shapes, and each gives messages, one item per document, Json unless a format is named:

Transport What one read is Ends Read by
rest(url, items, next) the items of a collection, every page in turn yes source
graphql(url, query, variables, items) the nodes one query returns yes source
webhook(port, path) the body of each request the host receives no tap

The shapes are the same two as before. A paginated collection and a query result are finite, read once per evaluation and replayable within it. A webhook is live: requests arrive when the sender decides, and the stream is as endless as a topic.

The carrier's API

The carrier lists parcels at https://api.carrier.example/parcels. Each response holds a page of them under results, and the Link header names the next page. The shop configures it once, with the token in the environment:

[endpoint.parcels]
kind = "rest"
url = "https://api.carrier.example/parcels"
items = "results"
next = "link"
token_env = "CARRIER_TOKEN"
type Parcel {
    order   : Int
    status  : String
    updated : String
}

parcels : Seq<Parcel> = source("parcels")
parcels
| filter(p => p.status == "delivered")
| map(p => p.order)

source sends a GET for each page in turn until none is named, and gives every item in page order. items is a field path into each response body, "data.results" for one nested deeper; next is "link" for the rel="next" target of the Link header, or a field path to a URL in the body. The token is sent as Authorization: Bearer, and the pages are fetched once per evaluation, so a program that reads parcels twice asks the carrier once.

The parcels are now one more table to the shop. Joining them to the silver orders of chapter 17, each an Event of chapter 16, is a join of two Seqs, which chapter 6 already showed:

orders : Seq<Event> = source("orders_silver")

orders
| filter(o => o.kind == "order_placed")
| left_join(parcels, left = o => o.id, right = p => p.order,
            into = (o, p) => {id: o.id, customer: o.customer, status: p.map(x => x.status).unwrap_or("not sent")})

A rest endpoint is read only. Creating something on the carrier's side is a request by another method, which is a write to an http endpoint.

The supplier's catalogue

The supplier answers GraphQL. A graphql endpoint is one query: one POST of the query and its variables, and the items are the array at items below the response's data:

[endpoint.catalogue]
kind = "graphql"
url = "https://supplier.example/graphql"
query = "{ products(inStock: true) { sku name price } }"
items = "products"
token_env = "SUPPLIER_TOKEN"
type Product {
    sku   : String
    name  : String
    price : Decimal
}

products : Seq<Product> = source("catalogue")
products | filter(p => p.price < 50.00) | map(p => p.name)

A response whose errors is not empty is an evaluation failure naming the URL and the first error's message: a GraphQL server answers 200 for a query it could not run, and the endpoint does not mistake that for an empty catalogue.

The payment provider's webhook

The payment provider does not wait to be asked. When a payment settles or fails, it sends a POST to an address the shop gave it. A webhook endpoint is that address, and tap over it is a Stream of request bodies:

[endpoint.payments]
kind = "webhook"
port = 8080
path = "/payments"
token_env = "PAYMENTS_TOKEN"
type Payment {
    order  : Int
    status : String
    amount : Decimal
}

payments : Stream<Payment> = tap("payments")
payments
| filter(p => p.status == "failed")
| map(p => {id: p.order, kind: "payment_failed", amount: p.amount})
| sink("orders_raw")

The program listens from the first traversal until it ends, and each accepted body is one value. The sender is answered as HTTP has it: 202 once the body is read and decoded, 400 for a body that does not decode, 401 for a request without the token, 404 for another path. A body that does not decode is answered and dropped; the stream goes on, because the sender, not the shop, made the mistake.

A webhook listens on the loopback address unless host says otherwise, so a program on a developer's machine is not reachable from outside by accident. It is the one transport that accepts connections, and like every Network transport it is refused in a document being rendered.

The failed payments go onto the shop's own raw subject, where the bronze pipeline of chapter 17 validates them with every other order. The webhook is an edge; what comes through it joins the flow the shop already has.

The archive

The auditor wants every raw order kept for seven years, as files, in object storage. An object store is a transport whose documents are objects under keys, and its module is in the io.store family: s3, which also reaches MinIO and Ceph, azure_blob, adls, gcs, sftp and hdfs.

[endpoint.archive]
kind = "s3"
bucket = "shop-archive"
prefix = "orders/raw/"
region = "eu-central-1"
format = "json_lines"
credentials_env = "ARCHIVE_CREDENTIALS"
tap("orders_raw") | collect(1000) | sink("archive")

collect(1000) turns the stream into batches, and sink writes each batch as one object under the prefix, so the archive is files of a thousand orders each rather than a million files of one. Read back, the prefix is a finite set of objects, and source("archive") is a Seq of every order in it: a replay of last year is a pipeline over the archive instead of over the subject, as chapter 19 shows.

A format says what the bytes of one object are. Beside the text formats of chapter 9, binary formats decode bytes to values with a schema of their own: Parquet for columns of a table, Avro and Protobuf for records on a wire. An archive written as Parquet is smaller and is read by every analytics tool there is:

import io.store.s3

orders : Seq<Event> = source("orders_silver")
orders | target(s3("shop-archive", key = "orders/2026.parquet", credentials_env = "ARCHIVE_CREDENTIALS"), Parquet)

A table format is one level up: Delta Lake and Apache Iceberg keep a table as Parquet objects and a transaction log in a store. Where the spark kind of chapter 17 asks a SQL server for a table's rows, a delta or iceberg kind reads the log and the objects itself, through the store's transport, so a table in a bucket is readable with no server running at all.

Further brokers and databases

The families have room for the systems a shop meets next, each a module named for its system, each a transport with a constructor and a kind:

Family Systems
io.msg rabbitmq, pulsar, mqtt, redis_streams, sqs, azure_service_bus, artemis, rocketmq, nsq, zeromq
io.db duckdb, clickhouse, mongodb, couchdb, cassandra, opensearch, redis, neo4j
io.net grpc, feed
io.store s3, azure_blob, adls, gcs, sftp, hdfs

A queue such as RabbitMQ or SQS is a tap and a sink, as a topic is; a document database is a source and a target, as a table is. Whether a new system ends is the one fact about it a program depends on, and the kind states it.

What is next

Chapter 19 puts every system of Part II into one file: the shop's whole day as a pipeline, checked before it runs, and where that pipeline may run.