15. Endpoints¶
Every system in Part II is reached the same way. A file, a table, a topic, a
subject and a webhook differ in where their data is and whether it ends; to
a program each is an endpoint, and four steps read and write all of
them. This chapter is that model, told over the shop of
chapter 14. The programs run with
--project examples/pipelines/shop.
One file, two spellings¶
An endpoint is a value that states where documents are. Constructing one
performs no I/O, and a program can reach the shop's events by writing the
endpoint itself, file(path):
source(file("examples/pipelines/shop/data/events.json")) | size
1
or by naming the id the shop's .hql/config.toml declares it under:
[endpoint.events]
kind = "json"
path = "data/events.json"
source("events") | size
6
The two answers differ because the two spellings say different things. A
.json file read by file is one JSON document, the whole array, so the
Seq holds one value. The json kind is a file whose one document is an
array of records, and it gives the array's elements. A constructed endpoint
states where the bytes are and leaves the rest to the program; a configured
endpoint states the system, its documents, and, on the
schema boundary, the names and
types that system uses.
The rest of Part II uses ids, for the reason the second spelling exists: a
program that names an id is not tied to one machine's paths, one server's
address, or one system. Moving the events from a file into a table is a
change to .hql/config.toml, not to the pipeline.
Four steps¶
Two questions sort every read and write. Does the endpoint end, and does the step take the data whole or one value at a time?
| Whole | One at a time | |
|---|---|---|
| Read | source(e): a Seq, every document, from an endpoint that ends |
tap(e): a Stream, one document held at a time |
| Write | target(e): one completed value, written once |
sink(e): one document per value, written as each arrives |
A table ends, so source reads it. A topic does not, so it is read by
tap; source of one is a check error, because a Seq is a value that
ends and a topic has no last message:
source("order_events")
check error[type-mismatch]: `order_events` is a `kafka` endpoint, read by `tap` as a `Stream`, not by `source` as a `Seq`
--> <expression>:1:8
|
1 | source("order_events")
| ^^^^^^^^^^^^^^
The writing side is the mirror. A stream has no last value for target to
wait for, so a stream is written with sink:
type Event {
id : Int
kind : String
customer : String
amount : Decimal
at : String
}
events : Stream<Event> = tap("order_events")
events | target("orders")
check error[type-mismatch]: `target("orders")` writes a Table or a sequence of a declared record, not Stream<Event>
--> <expression>:10:10
|
10 | events | target("orders")
| ^^^^^^^^^^^^^^^^
type Event {
id : Int
kind : String
customer : String
amount : Decimal
at : String
}
events : Stream<Event> = tap("order_events")
events | sink("orders")
Step
tap reads an endpoint that ends as well: tap("ledger") gives the table's
rows one at a time, which is how a table larger than memory is read. The
type of each step is fixed before the program runs, so a pipeline that would
wait forever, tap("order_events") | size, is a check error and not a
process that hangs (finite or live).
The kind states the type¶
source("orders") is a PostgreSQL table, and the program above was checked
without a database. Checking reads .hql/config.toml, finds kind = "postgres",
and types orders from the kind alone: it ends, source reads it as a
Seq of rows, and target writes it in one transaction or sink row by
row. No connection is made until the program runs, and hql check
never runs it:
source("customer_revenue")
Seq<Data>
customer_revenue is a Delta table in a lakehouse; the shop's configuration
names it, and that is all the checker needs. The kinds in the current
release, and the chapter each is told in:
| Kind | What one endpoint is | Ends | Chapter |
|---|---|---|---|
csv, json, json_lines, yaml |
a file in that format | yes | 9 |
vault |
a folder of documents | yes | 20 |
sqlite |
one table of a database file | yes | 16 |
postgres |
one table of a PostgreSQL database | yes | 16 |
kafka |
one Kafka topic | no | 16 |
nats |
one core NATS subject | no | 17 |
nats_jetstream |
one subject of a JetStream stream | no | 17 |
spark |
one table of a lakehouse, over Spark SQL | yes | 17 |
The id is a literal. A computed id, a connection string in place of an id, and an id the configuration does not declare are each refused at checking, on the endpoints-by-id page of chapter 9.
Secrets live in the environment¶
The shop's .hql/config.toml holds no password. An endpoint whose address may carry
a secret names the environment variable that holds it, dsn_env for a
database, brokers_env for Kafka, url_env for NATS and the lakehouse, and
token_env for a token:
[endpoint.orders]
kind = "postgres"
dsn_env = "SHOP_DSN"
table = "orders"
order = ["id"]
An address with credentials written into the configuration refuses the whole vault, before any program is read:
$ hql --vault shop eval '1'
hql: shop: `[endpoint.x]`: `url` holds credentials; name the environment variable that holds the address, `url_env = "VARIABLE"`
A variable that is not set is found when the endpoint is first read, and the failure names the endpoint and the variable, never its value:
$ hql --project examples/pipelines/shop eval 'source("orders")'
execute failure[io-failure]: endpoint `orders`: the environment variable `SHOP_DSN`, which `dsn_env` names, is not set
--> <expression>:1:1
|
1 | source("orders")
Every failure past checking has that form: the endpoint by its id, then the system's own words, then the row or message by its position. A row that does not decode stops the evaluation there; chapter 16 routes such rows instead of stopping.
Transports¶
An endpoint that reaches outside the evaluation is a transport. Two properties of a transport matter to a program and to whoever runs it:
- its locality:
Direct, on this machine, such as a file or standard input, orNetwork, another process over a connection; - its carriage: what one read moves, bytes that a format splits into documents, a framed message that is one document, or a row of named columns that needs no format at all.
Sqlite and the two NATS transports are declared with both today. Each is a
module of its own, imported by its full path, and its constructor takes the
keys its kind takes, so a program reaches a subject without any
configuration:
from io.msg.nats_jetstream import *
tap(nats_jetstream("orders.raw", stream = "ORDERS", url_env = "NATS_URL", idle_ms = 2000))
Stream<Data>
A message is one document, so no format is named, and it is Json unless
one is. A table is rows of named columns, so it takes no format at all, and
naming one is a check error:
from io.db.sqlite import *
source(sqlite("data/shop.db", table = "orders", order = ["id"]))
Seq<Data>
A constructor takes no secret, only the name of the variable that holds
one; and a constructed transport has no schema boundary, which only
configuration states. A row is read as Data, or as the record an
annotation names, each column decoded as its type holds it, and a record
is written to a constructed table a column per field, replacing the rows
it held. A database file is written whole, by target; sink writes
rows to a server's table, not to a file's.
Design only
The rest of this section describes functionality that is not in the current release. It is written as if it existed: names and types may change before it ships.
Every kind is a transport with a constructor of its name, and every transport states its locality and carriage:
| Transport | Locality | Carriage |
|---|---|---|
File, Stdin, Stdout |
Direct |
bytes |
Sqlite |
Direct |
rows |
Http, Tcp |
Network |
bytes |
WebSocket, Kafka, Nats, NatsJetStream, Rest, Graphql, Webhook |
Network |
messages |
Postgres, Spark |
Network |
rows |
The carriage settles whether a step takes a format: a byte transport needs
one, a message transport may name one and is Json without, and a row
transport takes none, so source(postgres(dsn_env = "SHOP_DSN", table =
"orders"), Json) is a check error. The locality is what a host grants: a
document being rendered reaches no Network transport, constructed or
configured, and the check reads that from the type, not from a list of
names (effects and what a host allows).
The transports are grouped into modules by the system they reach:
io file, stdin, stdout, stderr, source, tap, target, sink
├── net http, websocket, tcp, rest, graphql, webhook
├── db sqlite, postgres, spark, and further databases
├── msg kafka, nats, nats_jetstream, and further brokers
└── store s3, azure_blob, gcs, sftp, hdfs
from io.db.postgres import * brings postgres(...) into scope, as
from io.db.sqlite import * brings sqlite(...) and
from io.msg.nats import * brings nats(...). A configured id needs no
import: the configuration names the kind, and the kind names its module.
A program that constructs a network transport names its module in its
first lines, so what a program may reach is visible before reading it.
Chapter 18 is the net and store
transports the shop reaches next.
What is next¶
Chapter 16 moves the shop's events out of a file: into a SQLite table, then a PostgreSQL table, then onto a Kafka topic and back.