HQL Book¶
This is a book about HQL, read from the first page to the last. It starts with
a number and ends with a web of vaults, and every chapter builds on the ones
before it. Every program in it prints what the book shows. Chapter 18 and
chapters 25 to 27 describe APIs and object stores, the ai module and the
direction beyond one vault, which are design and not in the current
release, as do the marked sections of chapters 14, 15, 19 and 37. Chapter 14's
companion on planning separates the local backend from future execution
strategies. The programs of chapters 16, 17, 19 and 20 that reach a
server are checked in the book and run in the
hql-demo repository.
HQL is a strongly typed language for querying, transforming, connecting and
computing over data, knowledge and AI workflows. A program is checked before it
runs, and values flow through pipelines of typed steps: over records and lists,
as jq works over JSON; over tables, as SQL does; over files, standard input
and streams that do not end; over a vault of Markdown documents, their links
and their meaning; and through the context, prompts and runs of a language
model.
The book is in five parts. The language is the steps, the values they
pass and the types that join them, from the first expression to the type
system and the boundary where data from outside becomes typed. Data flows
is the pipeline those steps form: what | does, when a program is checked,
where it runs, and the systems a pipeline connects. Knowledge and AI is
a vault of documents as a namespace, its graph, retrieval by meaning,
documents that compute themselves, agents whose runs are values, documents
that run them, and the web they point at. The standard library is the
modules a program imports for work that is neither a type nor I/O:
statistics, random values, digests and encodings, and cryptography.
Extending HQL is how that library is made and how to add to it: a
function declared in HQL and implemented in Rust, the package that carries
it, the library hql loads it from, and the registry that names packages.
Following along¶
Install hql, clone the repository for the reading-club vault,
and run each program with hql eval:
git clone https://github.com/ewiger/hql && cd hql
hql --vault examples/reading-club eval 'cards | size'
The reading-club vault, examples/reading-club/,
is a small reading club: seven books, five authors, the links between them,
one claim about which book inspired which, and a librarian.
The programs that reach a server — Kafka, PostgreSQL, NATS, a lakehouse —
run in the hql-demo repository. Each
of its cases is a vault with the Docker Compose file that starts what it
needs and a run.sh that compares the output with the expected one. With
Docker and hql on the PATH:
git clone https://github.com/ewiger/hql-demo && cd hql-demo
./run-all.sh
Part I: The language¶
- Hello world — the first program, running it, and the check before a program runs.
- Data types —
Int,Float,StringandBoolin the REPL,Bytes,Dataand its JSON-shaped trees,Optionfor a value that may be missing, and how sets,Dataand maps differ. - Collections — lists, sets and maps, and the steps they share.
- Functions — lambdas, declared functions, the
two body forms
= exprand{ … }, generic functions, aliases, and partial application. - Modules — modules of your own, imports, receiver modules, public and private declarations, how a name resolves, and the packages modules come in.
- SQL-like Queries — where, order by, group by and the aggregates.
- Streams that do not end — generators, laziness, and batches.
- Writing results out — presentation formats and who chooses them.
- I/O — sources and targets: endpoints,
source,tap,targetandsink, finite or live, formats, endpoints by id and the schema boundary, channels, logging, the network, files and folders, the environment and arguments, andhqlas a command-line data tool. - The type system — records, unions and match, results, subtypes and narrowing, generics, receiver calls, operators and traits.
- Advanced data typing — the type
boundary where loose data becomes typed, open and closed types,
TypeContractandDataContract, validators, validation as a value, and what a contract states that a schema cannot. - Errors — failures kept as data, diagnostics and the
run's diagnostic channel,
strictandcollect. - Working with
hql— the commands, the reports, and what to do when something is wrong.
Part II: Data flows¶
- ETL pipelines — the wiring as a program:
|and., the three phases, logical steps and effects,hqlin a shell pipe, the shop whose orders the rest of Part II follows, and partitioning, schedules and backends. Its companion, Logical steps and physical plans, follows a checked program into local execution and the design direction for parallel workers, SQL translation and distributed execution. - Endpoints — one model for every system: a file by
path or by id,
source,tap,targetandsink, the kind that states the type, secrets in the environment, and transports. - Tables and topics — the shop's orders in a SQLite table, a PostgreSQL table and a Kafka topic, what a write promises, and a topic whose messages are not all orders.
- Subjects and the lakehouse — core NATS notifications, a JetStream log and a durable consumer, and bronze, silver and gold Delta tables.
- APIs, webhooks and stores — a paginated REST API, a GraphQL catalogue, a webhook, an archive in object storage, and the systems the transport families reserve.
- OLTP — the shop's transactional side: its whole day in one checked file, where its mistakes are found, a replay as the same file, and streams against tables.
- OLAP — the history those transactions leave: bronze, silver and gold, the grain of a row and its keys, deduplication, a table kept up to date by key, and incremental loads.
Part III: Knowledge and AI¶
- A vault of documents — cards, their metadata, a reading list nobody maintains, and the vault as a namespace.
- Links and graphs — following links, building graphs, drawing them.
- Finding cards by meaning — rankings, hits, and the evidence behind a score.
- Queries inside documents — cells, documents built from them, and cells that maintain a vault.
- Agents and runs — agents as cards, context and prompts, tools, and runs kept as values.
- Agents inside documents — a document whose cells retrieve evidence, run an agent over the draft they sit in, and remember the answer.
- The hyper web — namespaces beyond one tree, providers, retrieval and agents across them, and a reader who asks.
Part IV: The standard library¶
- Statistics — means, spread, quantiles, the mode and correlation, over a list and over a query's column.
- Random values — sampling, shuffling, choice, numbers and UUIDs, and a seed that makes them reproducible.
- Digests and encodings — digests of text
and bytes, the fingerprint of a
Datatree, and hex, base64 and the other text encodings. - Keys and cryptography — keys and their formats, secrets that print redacted, signatures, MACs and password hashes, and the encryption and key vault that are design.
Part V: Extending HQL¶
- Functions and declarations — a function with a body in HQL, a declaration whose body is Rust, and how the standard library is built from the second.
- Packages and extensions — an
importfollowed from the module it names to the package, the extension and the file that runs. - Writing an extension — a package from an empty directory: the crate, the contract, the Rust functions, values, errors, and functions passed in.
- How a library is loaded — static and
dynamic libraries, the one symbol
hqlcalls, and why a library fits onehqland one compiler. - Installing packages —
hql package, what an installed package is, a package under development, and the settings of.hql/config.toml. - The registry and publishing — official and known packages, version ranges and exact versions, and publishing a crate.
- The ml package — datasets, models as values, prediction and metrics, from the first official package.