Endpoints by id¶
A program that names a file path is tied to one machine's files. A vault
declares its endpoints in .hql/config.toml instead, each under an id, and a program
names only the id. The field station of the
endpoints examples
is such a vault, and the programs on this page run with
--vault examples/endpoints/station:
[endpoint.journal]
kind = "vault"
root = "journal"
[endpoint.counts]
kind = "csv"
path = "data/counts.csv"
[endpoint.sightings]
kind = "json"
path = "data/sightings.json"
journal = source("journal")
journal.cards | map(c => c.name)
["kite", "owl", "wren"]
A vault is a folder of documents, each a card with a name, the subject of
chapter 21; journal.cards is the cards of that one.
Here it is one endpoint among others, read by its id like the rest.
The kind states the type¶
source("journal") is a Vault, with its own cards. source("counts") is
a Seq<Data>, a row per element, and a json endpoint is the elements of its
file's one array:
source("counts")
[{count: "2", species: "wren"}, {count: "1", species: "owl"}, {count: "1", species: "kite"}]
source("sightings") | map(s => s.species)
["wren", "owl", "kite"]
The kinds are vault, csv, json, json_lines, yaml, sqlite, one
table of a database file, postgres, one table of a PostgreSQL database,
kafka, one Kafka topic, nats, one core NATS subject, nats_jetstream,
one subject of a NATS JetStream stream, and spark, one table of a
lakehouse. postgres and kafka are the subject of
topics and tables, and nats, nats_jetstream and
spark of
subjects and lakehouse tables. A path resolves against the directory of
.hql/config.toml. The type comes from the configuration, so checking a program
reads .hql/config.toml and opens nothing else.
An endpoint read as a declared record decodes each element as the record, as a
file does, and tap("id") gives a finite endpoint one element at a time, as a
Stream:
type Tally {
species : String
count : Int
}
tallies : Seq<Tally> = source("counts")
tallies | map(t => t.count) | sum
4
The id is a literal¶
The type an endpoint is read as is known only from its configuration, so the id is written in the program as a literal, and a connection string is not an id:
id = "counts"
source(id)
check error[type-mismatch]: `source` names a configured endpoint by a literal id, `source("field_notes")`: a computed id would leave its type unknown until evaluation
--> <expression>:2:8
|
2 | source(id)
| ^^
source("postgres://db/birds")
check error[type-mismatch]: "postgres://db/birds" is not an endpoint id: connection details belong in configuration, `[endpoint.<id>]` in .hql/config.toml, and a program names the id
--> <expression>:1:8
|
1 | source("postgres://db/birds")
| ^^^^^^^^^^^^^^^^^^^^^
An id the configuration does not declare is a name error that names the nearest one:
source("count")
check error[unknown-name]: no endpoint `count` is configured; did you mean `counts`?
--> <expression>:1:8
|
1 | source("count")
| ^^^^^^^
The schema boundary¶
The names and types an external system uses are facts about that system, not
about the program's types, so they are configured beside the endpoint. The
station's ledger is a CSV whose headers are Species Name, Total and
Observer. [endpoint.<id>.fields.<field>] maps a field of the record a
program reads to its external field, by name for a column or a key, and by
a dotted path for a key below the top level of a JSON or YAML document:
[endpoint.ledger]
kind = "csv"
path = "data/ledger.csv"
[endpoint.ledger.fields.species]
name = "Species Name"
[endpoint.ledger.fields.count]
name = "Total"
[endpoint.profiles]
kind = "json"
path = "data/profiles.json"
[endpoint.profiles.fields.id]
name = "user_id"
[endpoint.profiles.fields.name]
path = "profile.display_name"
type Profile {
id : String
name : String
}
profiles : Seq<Profile> = source("profiles")
profiles | map(p => p.name)
["Ada", "Grace"]
A field's table also takes type, the external type where the format's
default is not it; absent, the external values read as an absent optional
field, an empty cell in CSV unless written; and write = false for a field a
write leaves out.
strictness says what a read does with an external field no field of the
record reads. It is compatible unless written: the field is passed to the
type, so a closed record does not decode a ledger with an Observer column.
Under open the field is not read, under warn it is not read and is
reported, and under strict it is an error:
type Tally {
species : String
count : Int
}
tallies : Seq<Tally> = source("ledger")
tallies | map(t => t.count) | sum
execute failure[io-failure]: endpoint `ledger`: document 1 does not decode as Tally: field `Observer`: Tally declares no such field
--> <expression>:6:24
|
6 | tallies : Seq<Tally> = source("ledger")
| ^^^^^^^^^^^^^^^^
[endpoint.ledger_open]
kind = "csv"
path = "data/ledger.csv"
strictness = "open"
[endpoint.ledger_open.fields.species]
name = "Species Name"
[endpoint.ledger_open.fields.count]
name = "Total"
type Tally {
species : String
count : Int
}
tallies : Seq<Tally> = source("ledger_open")
tallies | map(t => t.count) | sum
4
hql boundary <file> <id> --as <Type> prints an endpoint's effective boundary,
each property with the layer it comes from, the format's default or the
configuration:
$ hql --vault examples/endpoints/station boundary tally.hql ledger --as Tally
endpoint ledger (csv), read as Tally; strictness compatible, mode replace
species : String
external Species Name configuration
type text format
position 1 format
read true format
write true format
absent empty, missing format
count : Int
external Total configuration
type text format
position 2 format
read true format
write true format
absent empty, missing format
Writing by id¶
A configured endpoint is written by its id as well. xs | target("id") writes a
Table<R>, or a List<R> or Seq<R>, of a declared record R through the
same boundary: each field under its external name, in position order, a
field with write = false left out. The kind fixes the format, so a format
beside the id is a type error, and a vault is read and never written:
type Tally {
species : String
count : Int
}
tallies : Seq<Tally> = source("counts")
tallies | filter(t => t.count > 1) | target("counts")
Step
source("counts") | target("counts")
check error[type-mismatch]: `target("counts")` writes rows of a declared record, and Data is not one
--> <expression>:1:20
|
1 | source("counts") | target("counts")
| ^^^^^^^^^^^^^^^^
source("counts") | target("counts", Csv)
check error[type-mismatch]: a configured endpoint's format is its kind's, set in its configuration; name none
--> <expression>:1:20
|
1 | source("counts") | target("counts", Csv)
| ^^^^^^^^^^^^^^^^^^^^^
The endpoint's table takes mode, how a write treats what the endpoint already
holds. It is replace unless written:
mode |
A file | A SQLite table |
|---|---|---|
create |
written where none exists; an existing file is an evaluation failure | created where none exists; an existing table is an evaluation failure |
append |
the rows or documents added after the ones held; created where none exists | rows inserted; the table created where none exists |
replace |
rewritten whole | dropped and created again with the rows written |
Reading an endpoint the same evaluation has replaced is an evaluation failure, as for a file, and a document being rendered writes nothing (effects and what a host allows).
Topics and tables¶
A postgres endpoint is one table of a PostgreSQL database, and a kafka
endpoint is one Kafka topic. The connection string of a database may hold a
password, so the configuration names the environment variable that holds it,
dsn_env; a topic's brokers are written as brokers or named the same way
by brokers_env:
[endpoint.orders]
kind = "postgres"
dsn_env = "DEMO_DSN"
table = "orders"
order = ["id"]
[endpoint.events]
kind = "kafka"
brokers_env = "KAFKA_BROKERS"
topic = "events"
idle_ms = 2000
Checking a program reads this configuration and connects to neither system. The variables are read, and the connections made, when an endpoint is read or written.
A table ends. source("orders") is a Seq of rows, read through the
schema boundary as a sqlite table is: the columns are compared with the
record before any row is decoded, and only the columns the record reads are
selected. A numeric column reads as a Decimal with its digits, a jsonb
column as the tree it holds, and a NULL as an absent field:
type Event {
id : Int
kind : String
customer : String
amount : Decimal
at : String
}
events : Seq<Event> = source("orders")
events
| filter(e => e.kind == "order_placed")
| group(by = e => e.customer)
| map(g => {customer: g.key, total: g.items | map(e => e.amount) | sum})
{{customer: "ada", total: 135.75}, {customer: "grace", total: 42.00}, {customer: "linus", total: 7.99}}
target("orders") writes a Seq or a Table of a declared record in one
transaction. A table that does not exist is created, a column per field:
BIGINT for an Int, DOUBLE PRECISION for a Float, NUMERIC for a
Decimal, BOOLEAN for a Bool, JSONB for a list or a record, and TEXT
otherwise. Under mode = "replace" the rows the table holds are removed and
its definition, indexes and grants are kept.
A topic does not end. tap("events") is a Stream, one JSON document
per message, decoded as the element type. source("events") is a type
error, since a Seq is a value that ends:
source("events")
check error[type-mismatch]: `events` is a `kafka` endpoint, read by `tap` as a `Stream`, not by `source` as a `Seq`
--> <expression>:1:8
|
1 | source("events")
| ^^^^^^^^
The stream starts at the earliest message the topic keeps, or at the next
one under from = "latest". Without idle_ms it waits for messages without
end; with it, the stream ends once no message has arrived for that many
milliseconds.
sink writes as values arrive. sink("events") writes each value as one
message, and sink("orders") each row as it arrives, committed one at a
time. A stream has no last value for target to wait for, so moving a topic
into a table is a tap and a sink:
events : Stream<Event> = tap("events")
events | sink("orders")
and moving a table into a topic is a source and a sink:
orders : Seq<Event> = source("orders")
orders | sink("events")
sink takes no format beside an id, and writes to a kafka, nats, nats_jetstream,
postgres or spark endpoint only; a file or an SQLite table is written
whole by target. A NATS subject and a lakehouse table take the places of
the topic and the table on the
next page.