Skip to content

HQL Book

This is a book about HQL, read from the first page to the last. It starts with a number and ends with a web of vaults, and every chapter builds on the ones before it. Every program in it prints what the book shows. Chapter 18 and chapters 25 to 27 describe APIs and object stores, the ai module and the direction beyond one vault, which are design and not in the current release, as do the marked sections of chapters 14, 15, 19 and 37. Chapter 14's companion on planning separates the local backend from future execution strategies. The programs of chapters 16, 17, 19 and 20 that reach a server are checked in the book and run in the hql-demo repository.

HQL is a strongly typed language for querying, transforming, connecting and computing over data, knowledge and AI workflows. A program is checked before it runs, and values flow through pipelines of typed steps: over records and lists, as jq works over JSON; over tables, as SQL does; over files, standard input and streams that do not end; over a vault of Markdown documents, their links and their meaning; and through the context, prompts and runs of a language model.

The book is in five parts. The language is the steps, the values they pass and the types that join them, from the first expression to the type system and the boundary where data from outside becomes typed. Data flows is the pipeline those steps form: what | does, when a program is checked, where it runs, and the systems a pipeline connects. Knowledge and AI is a vault of documents as a namespace, its graph, retrieval by meaning, documents that compute themselves, agents whose runs are values, documents that run them, and the web they point at. The standard library is the modules a program imports for work that is neither a type nor I/O: statistics, random values, digests and encodings, and cryptography. Extending HQL is how that library is made and how to add to it: a function declared in HQL and implemented in Rust, the package that carries it, the library hql loads it from, and the registry that names packages.

Following along

Install hql, clone the repository for the reading-club vault, and run each program with hql eval:

git clone https://github.com/ewiger/hql && cd hql
hql --vault examples/reading-club eval 'cards | size'

The reading-club vault, examples/reading-club/, is a small reading club: seven books, five authors, the links between them, one claim about which book inspired which, and a librarian.

The programs that reach a server — Kafka, PostgreSQL, NATS, a lakehouse — run in the hql-demo repository. Each of its cases is a vault with the Docker Compose file that starts what it needs and a run.sh that compares the output with the expected one. With Docker and hql on the PATH:

git clone https://github.com/ewiger/hql-demo && cd hql-demo
./run-all.sh

Part I: The language

  1. Hello world — the first program, running it, and the check before a program runs.
  2. Data types — Int, Float, String and Bool in the REPL, Bytes, Data and its JSON-shaped trees, Option for a value that may be missing, and how sets, Data and maps differ.
  3. Collections — lists, sets and maps, and the steps they share.
  4. Functions — lambdas, declared functions, the two body forms = expr and { … }, generic functions, aliases, and partial application.
  5. Modules — modules of your own, imports, receiver modules, public and private declarations, how a name resolves, and the packages modules come in.
  6. SQL-like Queries — where, order by, group by and the aggregates.
  7. Streams that do not end — generators, laziness, and batches.
  8. Writing results out — presentation formats and who chooses them.
  9. I/O — sources and targets: endpoints, source, tap, target and sink, finite or live, formats, endpoints by id and the schema boundary, channels, logging, the network, files and folders, the environment and arguments, and hql as a command-line data tool.
  10. The type system — records, unions and match, results, subtypes and narrowing, generics, receiver calls, operators and traits.
  11. Advanced data typing — the type boundary where loose data becomes typed, open and closed types, TypeContract and DataContract, validators, validation as a value, and what a contract states that a schema cannot.
  12. Errors — failures kept as data, diagnostics and the run's diagnostic channel, strict and collect.
  13. Working with hql — the commands, the reports, and what to do when something is wrong.

Part II: Data flows

  1. ETL pipelines — the wiring as a program: | and ., the three phases, logical steps and effects, hql in a shell pipe, the shop whose orders the rest of Part II follows, and partitioning, schedules and backends. Its companion, Logical steps and physical plans, follows a checked program into local execution and the design direction for parallel workers, SQL translation and distributed execution.
  2. Endpoints — one model for every system: a file by path or by id, source, tap, target and sink, the kind that states the type, secrets in the environment, and transports.
  3. Tables and topics — the shop's orders in a SQLite table, a PostgreSQL table and a Kafka topic, what a write promises, and a topic whose messages are not all orders.
  4. Subjects and the lakehouse — core NATS notifications, a JetStream log and a durable consumer, and bronze, silver and gold Delta tables.
  5. APIs, webhooks and stores — a paginated REST API, a GraphQL catalogue, a webhook, an archive in object storage, and the systems the transport families reserve.
  6. OLTP — the shop's transactional side: its whole day in one checked file, where its mistakes are found, a replay as the same file, and streams against tables.
  7. OLAP — the history those transactions leave: bronze, silver and gold, the grain of a row and its keys, deduplication, a table kept up to date by key, and incremental loads.

Part III: Knowledge and AI

  1. A vault of documents — cards, their metadata, a reading list nobody maintains, and the vault as a namespace.
  2. Links and graphs — following links, building graphs, drawing them.
  3. Finding cards by meaning — rankings, hits, and the evidence behind a score.
  4. Queries inside documents — cells, documents built from them, and cells that maintain a vault.
  5. Agents and runs — agents as cards, context and prompts, tools, and runs kept as values.
  6. Agents inside documents — a document whose cells retrieve evidence, run an agent over the draft they sit in, and remember the answer.
  7. The hyper web — namespaces beyond one tree, providers, retrieval and agents across them, and a reader who asks.

Part IV: The standard library

  1. Statistics — means, spread, quantiles, the mode and correlation, over a list and over a query's column.
  2. Random values — sampling, shuffling, choice, numbers and UUIDs, and a seed that makes them reproducible.
  3. Digests and encodings — digests of text and bytes, the fingerprint of a Data tree, and hex, base64 and the other text encodings.
  4. Keys and cryptography — keys and their formats, secrets that print redacted, signatures, MACs and password hashes, and the encryption and key vault that are design.

Part V: Extending HQL

  1. Functions and declarations — a function with a body in HQL, a declaration whose body is Rust, and how the standard library is built from the second.
  2. Packages and extensions — an import followed from the module it names to the package, the extension and the file that runs.
  3. Writing an extension — a package from an empty directory: the crate, the contract, the Rust functions, values, errors, and functions passed in.
  4. How a library is loaded — static and dynamic libraries, the one symbol hql calls, and why a library fits one hql and one compiler.
  5. Installing packages — hql package, what an installed package is, a package under development, and the settings of .hql/config.toml.
  6. The registry and publishing — official and known packages, version ranges and exact versions, and publishing a crate.
  7. The ml package — datasets, models as values, prediction and metrics, from the first official package.