Skip to content

13. Working with hql

The language is half of what you use; the other half is the command and what it says back. This chapter is about the tools: the commands, the diagnostics, and the messages hql writes when a program is wrong.

The commands

Command What it does
hql eval <program> check and evaluate a program given as an argument
hql run <file> check and evaluate a program in a file; - reads standard input
hql check <file> write the program's type, and the signature of every alias, without evaluating
hql repl read, evaluate and print, keeping bindings between lines
hql render <file> build a document from its cells (chapter 24)
hql builtins list every step, by the provider that implements it, with its signature
hql config the settings in force, and where each came from
hql context [file] the execution context: the mode, the project, the vault, the configuration files and module roots, and where each came from
hql doctor check the semantic-search chain and name the first missing part

--vault <dir> names the vault, --to <format> the output format, and --envelope json wraps the answer for another program. --project <dir>, --bare, --import <module> and --context <name> set the execution context, below.

The REPL

hql repl keeps its bindings from one input to the next. The arrow keys edit a line and recall earlier inputs, from a history kept between sessions. An input goes on over several lines while a bracket is open, or while the next line opens with |, so every program in this book can be pasted in as it is written. :type asks for the type of an expression, or, given a function's name, prints the standard library's declarations of it, one per overload. :quit leaves, as do quit, exit and Ctrl-D:

$ hql repl
hql 0.5.1. `:help` for commands, `:quit` to leave.
hql> shelf = [412, 280]
()
hql> shelf | sum
692
hql> :type shelf
List<Int>
hql> :type range
fn range(start: Int, end: Int, step: Int = 1) : List<Int>    // std/seq
hql> :quit

What a step takes

hql builtins lists every step with its signature, one line per form, grouped by the module declaring it. Each heading names the module, the package providing it and whether a program has it without an import:

$ hql builtins
collections (std 0.6.0, always available)
  size
    Collection<T> | size<T>() -> Int
    Window<T> | size<T>() -> Int
    pure, reduction
    The number of element occurrences in `self`, an `Int`: `[1, 2, 2, 3] | size` is `4`, and `Set([1, 2, 2]) | size` is `2`.
  contains
    Collection<T> | contains<T>(value: T (positional)) -> Bool
    pure, reduction
    `true` when `value` occurs in `self`, and `false` otherwise: `[1, 2, 3] | contains(2)` is `true`.
  …

A function of a module that the program has not imported is listed too, so the import it needs can be looked up. import stats binds the module's name, so its functions are called stats.mean(…); from stats import * binds mean and the rest by their own names (chapter 5). The modules of lib/std/prelude.hql, core, collections, string, seq, math, option, result, graph, format and io among them, are imported both ways in every program unless the project's .hql/config.toml says prelude = false under [modules].

Diagnostics

A run produces a value and the diagnostics its phases wrote: warnings that change nothing, errors that make a result wrong, failures that make one impossible. Each is headed by the phase that wrote it, its severity and its kind, and names where it points, by line and column. Chapter 12 is the whole error model.

By default hql stops at the first error. --report collect reports every error it can find:

$ hql --report collect eval 'a = nosuchthing
b = alsomissing'
check error[unknown-name]: `nosuchthing` is not bound
 --> <expression>:1:5
  |
1 | a = nosuchthing
  |     ^^^^^^^^^^^

check error[unknown-name]: `alsomissing` is not bound
 --> <expression>:2:5
  |
2 | b = alsomissing
  |     ^^^^^^^^^^^

2 diagnostics: 0 failure, 2 error, 0 warning

A misspelled step suggests the nearest one, and a retired name says what replaced it, on a help line under the caret:

$ hql eval '[1, 2] | mapp(n => n)'
check error[unknown-step]: `mapp` is not a pipeline step
 --> <expression>:1:10
  |
1 | [1, 2] | mapp(n => n)
  |          ^^^^^^^^^^^^
  = help: did you mean `map`?

A warning never changes the exit status. Names have one casing per kind — snake_case for bindings, functions and fields, PascalCase for types — and a name off its kind's convention is a warning where it is written:

$ hql eval 'bookCount = 7
bookCount'
7
check warning[naming-convention]: `bookCount` is a binding, written in snake_case
 --> <expression>:1:1
  |
1 | bookCount = 7
  | ^^^^^^^^^
  = help: write it `book_count`

--envelope json carries every diagnostic, whatever the level, as data, and --diagnostics-format json writes each one to standard error as one JSON document per line. A program that wants to report on itself, apart from its result, writes to a logger: the logging page of chapter 9.

The execution context

Every invocation runs in an execution context: a project or none, a vault or none, and the configuration gathered for it. The same program text checks and evaluates the same way in every context; the context determines the module roots, packages, endpoints and settings it is given, and, where a run imports hmd, the documents it reads as cards.

Projects and vaults

A project is HQL's: a directory holding .hql/config.toml, which configures the programs run in it. A vault is HyperMarkdown's: a directory holding .hmd/, whose documents are cards. The two are found apart, and one directory may be both.

The anchor is the directory discovery starts from: the source directory of the program, the directory of its file for hql run, check, plan, boundary and context <file>, and the working directory for an expression or standard input. hql walks from the anchor upward to the nearest directory holding .hql/config.toml, the project, and separately to the nearest holding .hmd/, the vault. The walk for a project skips the directory whose .hql/ is the HQL home, so ~/.hql/config.toml is never a project. The mode is project where a project was found, and bare where none was.

The working directory is not consulted when a file is the anchor, so a program finds its project wherever hql is run from. A relative path in a program resolves against its source directory too, never against the working directory (the files and folders page of chapter 9).

--project <dir> names the project and replaces its walk; the directory must hold .hql/config.toml. --vault <dir> names the vault the hmd package reads, and the directory must hold .hmd/: it is the vault root, not the directory its cards start in, which the vault's wiki setting names (chapter 21). A directory inside a vault that holds no .hmd/ is refused, naming the vault root above it. Where no project is named, the project is the nearest at or above the vault. --bare runs with no project and no discovery at all, and --bare with --project is a usage error. hql render finds the vault of each document from that document's directory, and under --bare renders none.

hql context writes the context, each part with the layer it came from:

$ cd ~/tmp
$ hql context ../notes/scripts/report.hql
mode:      project
context:   work (user configuration)
project:   /home/ada/notes (discovered from the anchor)
vault:     /home/ada/notes (discovered from the anchor)
anchor:    /home/ada/notes/scripts
cwd:       /home/ada/tmp
hql-home:  /home/ada/.hql (the default)
prelude:   on (the default)
config:
  /etc/hql/config.toml (system configuration, absent)
  /home/ada/.hql/config.toml (user configuration)
  /home/ada/notes/.hql/config.toml (the project's .hql/config.toml)
module roots:
  /home/ada/.hql/lib (user configuration)
imports:
  hmd (the project's .hql/config.toml)

--envelope json writes the same as one JSON document.

Configuration layers

Bare means no project, not no configuration. These layers contribute in every mode, each read in turn:

Layer Where
system /etc/hql/config.toml on Unix, %ProgramData%\hql\config.toml on Windows; $HQL_SYSTEM_CONFIG names another file
user config.toml in the HQL home, ~/.hql/; $HQL_HOME names another directory
named context the selected [context.<name>] table of the system or user file
HQL home modules ~/.hql/modules/, a module root when it exists
project the project's .hql/config.toml
$HQL_PATH further module roots, separated as $PATH is
command line --modules <dir> and --import <module>

The system and user files take the keys a project's .hql/config.toml takes: [modules] roots, import and prelude, [packages] dependencies and paths, [report] mode and level, and [execution] backend. A relative path in a project's file is read against the project root, the directory holding .hql/; in the system or user file, against the directory holding that file. A configuration file that does not parse, or that states a key its layer does not take, fails every run, naming the file and the key.

Module roots and imports gather across the layers, each once. A gathered root is a place to look, not a priority: a module that two roots provide is still an error. prelude is the last layer's that states it. The HQL home is the user's environment, not a project, and never a vault.

Packages come from the project and from the HQL home, in that order, each holding its own packages/ (chapter 36).

Named contexts

A named context is a [context.<name>] table of the system or user file: settings grouped under a name and reused across projects. It takes the keys above, and project, which binds a project, read against the directory of the declaring file. A user file's definition of a name replaces the system file's; a project's .hql/config.toml defines no context.

current-context = "work"

[modules]
roots = ["lib"]

[context.work.report]
level = "info"

[context.archive]
project = "../archive"

One context is selected: --context <name>, else $HQL_CONTEXT, else current-context, else a context named default when one is defined. A selected name that no file defines fails every run, naming the layer that selected it. A context that binds a project replaces the walk for one; a context without one leaves discovery as it is. A context selected by --context or $HQL_CONTEXT that binds a project, together with --bare or a --project naming another directory, is an error; under a context selected by current-context, the command line wins.

hql config lists and changes the selection; use-context writes current-context into the user file and keeps every other line of it:

$ hql config get-contexts
CURRENT  NAME     PROJECT            DEFINED IN
         archive  /home/ada/archive  /home/ada/.hql/config.toml
*        work     -                  /home/ada/.hql/config.toml
$ hql --context archive context
mode:      project
context:   archive (--context)
project:   /home/ada/archive (context `archive`)
…
$ hql --context nope eval 1
hql: no context is named `nope`, which --context selects; the defined contexts are `archive`, `work`

hql config current-context writes the persistent current-context, which --context and $HQL_CONTEXT leave unchanged, hql config use-context <name> sets it, and hql config view writes the system and user files merged, a table key by key and any other value from the user file.

Settings

The report mode and level come from the command line, then the project's .hql/config.toml, then the environment ($HQL_REPORT, $HQL_REPORT_LEVEL), then the selected context, the user file and the system file, then the defaults. The backend follows the same order with --backend, [execution] backend and $HQL_BACKEND. hql config says which applies, and why:

$ hql config
report mode: strict (from the default)
report level: warning (from the default)
backend: localhost (from the default)
vault: none

backend selects the execution strategy described in Logical steps and physical plans, and localhost is the only one. Run inside a vault, or with --vault, the last line names it and counts its cards.

What is next

Every chapter so far was about a step. Chapter 14 is about the pipeline the steps form: what | does, when a program is checked, where it runs, and the systems a pipeline will connect.