Skip to content

21. A vault of documents

The shelf of chapter 6 was a list written by hand. Here the same books are documents in a folder, and HQL reads the folder. This part of the book is about knowledge: documents, the links between them, their meaning, and the agents that read them.

A vault is a folder

A vault is a directory of Markdown and HyperMarkDown documents, marked as one by an .hmd/ directory at its root. The reading-club vault looks like this:

examples/reading-club/
├── authors/
│   ├── frank-herbert.hmd
│   ├── jrr-tolkien.hmd
│   ├── mary-shelley.hmd
│   ├── virginia-woolf.hmd
│   └── walt-whitman.hmd
├── books/
│   ├── a-room-of-ones-own.hmd
│   ├── dune.hmd
│   ├── frankenstein.hmd
│   ├── leaves-of-grass.hmd
│   ├── mrs-dalloway.hmd
│   ├── the-common-reader.hmd
│   └── the-hobbit.hmd
├── club.hmd
├── frankenstein-inspires-dune.hmd
├── librarian.hmd
├── .hmd/config.toml
└── .hql/config.toml

.hmd/ makes the directory a vault. .hql/config.toml makes it an HQL project too, which configures the programs run in it; the two are separate, and a vault need not be a project (chapter 13).

The vault's .hmd/config.toml says where its cards start, with the key wiki: a path against the vault root, doc/wiki when the file does not set it. The reading club keeps its cards at the vault root, so it says so:

wiki = "."

That directory is the namespace root. A document is a card exactly when it lies below it, and a card's path is written from it, so books/dune.hmd above. A file of the vault outside it, a README.md beside .hmd/ in a vault whose wiki is doc/wiki, is not a card. A card is a .hmd file, or a .md or .markdown file whose body is plain Markdown.

Each document is a card. A card has a header of metadata and a body someone wrote:

---
title: Dune
metadata:
  category: book
  author: Frank Herbert
  year: 1965
  genre: novel
  pages: 412
  status: reading
---
# Dune

A novel by [[frank-herbert]]. A desert planet, a spice, and a family that inherits both.

A program run inside a vault reads it: hql takes the nearest directory at or above the program's file that holds .hmd/, or at or above the working directory for an expression or standard input. --vault <directory> names one instead, and --bare runs with none. The cards are read through the builtin package hmd, which the reading-club's project imports for every program ("The vault itself" below):

hql --vault examples/reading-club eval 'cards | size'
cards | size
15

cards

cards is every card of the vault, as a sequence. Its order is the order a sorted walk of the folder reaches each document — the entries of each directory by name, depth first — so a program answers the same on every machine:

cards | take(4) | map(c => c.name)
["frank-herbert", "jrr-tolkien", "mary-shelley", "virginia-woolf"]

A card has the fields of a document: its name (the file stem), title, path, format, body, its kind, and metadata, the header's tree.

@[[name]] refers to a card by name: @ followed by the link a document writes, [[name]]. A link may point at a document nobody has written yet, so its type is Option<Card>, and reading a field through it reads through the option:

@[[dune]].title
"Dune"
@[[nowhere]].title
none

Metadata is Data

A card's metadata is the Data of chapter 2, so every question from chapters 2 and 6 applies. The books are the cards whose category says so:

cards
| filter(c => c.metadata.category == "book")
| map(c => c.title)
["A Room of One's Own", "Dune", "Frankenstein", "Leaves of Grass", "Mrs Dalloway", "The Common Reader", "The Hobbit"]

The reading list

The club's reading list is the books still to read, by title:

cards
| filter(c => c.metadata.status == "todo")
| sort(by = c => c.title)
| map(c => c.title)
["Leaves of Grass", "The Common Reader", "The Hobbit"]

Read it as three steps, each consuming the one before: filter decides each card on its own, sort needs every card before it can place the first, and map turns each card into its title. Nothing is stored in between, and nothing is a loop.

No one maintains this list. Mark a book status: done in its header and the next run of the query no longer lists it.

Writing cards

The formats of chapter 8 apply to a card as to any value. A card written as Markdown is its body; a list of cards, a list of links:

cards | filter(c => c.metadata.category == "author") | markdown
- [[frank-herbert]]
- [[jrr-tolkien]]
- [[mary-shelley]]
- [[virginia-woolf]]
- [[walt-whitman]]

Grouping cards

group partitions the books by a header value, as it partitioned the shelf:

cards
| filter(c => c.metadata.category == "book")
| group(by = c => c.metadata.status)
| map(g => {status: g.key, books: g.items | map(c => c.title)})
{{books: ["A Room of One's Own", "Frankenstein"], status: "done"}, {books: ["Dune", "Mrs Dalloway"], status: "reading"}, {books: ["Leaves of Grass", "The Common Reader", "The Hobbit"], status: "todo"}}

A typed view of a card works as the views of chapter 6 did, and the pages per author are a map over each group and a sum — map, then reduce:

fn pages(c: Card) : Int = match c.metadata.pages { some(Int n) => n, _ => 0 }

cards
| filter(c => c.metadata.category == "book")
| group(by = c => c.metadata.author)
| map(g => {author: g.key, pages: g.items | map(pages) | sum})
{{author: "Virginia Woolf", pages: 610}, {author: "Frank Herbert", pages: 412}, {author: "Mary Shelley", pages: 280}, {author: "Walt Whitman", pages: 145}, {author: "J. R. R. Tolkien", pages: 310}}

Kinds of card

A card whose header declares a relation between two other cards is a RelationCard; every other card is a ConceptCard. of_type keeps the cards of one kind:

cards | of_type(RelationCard) | map(c => c.title)
["Frankenstein inspires Dune"]

Chapter 22 is about what that relation means: cards and the links between them form a graph.

The vault itself

cards is a binding of the builtin package hmd, cards = vault.cards, over vault, the value the host seeds when it is given a vault:

[vault.name, vault.cards | size | text]
["reading-club", 15]

A vault is HyperMarkdown's, and so are vault and cards. They are bound while the package hmd is active, and a run that has a vault activates it: the vault found at or above the program, or the one --vault names. A program in a vault writes cards with no import.

A run with no vault activates hmd by importing it: import hmd in the program, --import hmd on the command line, or [modules] import in the project's .hql/config.toml:

[modules]
import = ["hmd"]

With no vault and no import, cards names what is missing:

cards | size
check error[unknown-name]: `cards` is HyperMarkdown's: write `import hmd`, or run with `--import hmd`, to bind it
 --> <expression>:1:1
  |
1 | cards | size
  | ^^^^^

With hmd imported and no vault, cards has nothing to read, and the check says so:

import hmd
cards | size
check error[unknown-name]: `cards` needs a vault: run inside one or pass --vault <directory>
 --> <expression>:2:1
  |
2 | cards | size
  | ^^^^^

A vault is a namespace

A card is a file with a place in a vault, and the place is what makes it addressable by name: [[dune]] names a card because there is one tree to resolve the name in. That tree is a namespace, rooted at the vault's namespace root, and a vault is exactly one namespace. cards holds what a wikilink can reach and nothing else, and a card's path is its place in the vault, relative to the namespace root, so the cards of one folder are a filter on the path:

cards | filter(c => c.path | starts_with("books/")) | size
7

HQL resolves a reference exactly as HyperMarkdown does, through HyperMarkdown's own resolver, so hql and hmd lint always name the same card. A [[name]] in a card is looked up from that card: a path that opens with /, ./ or ../ is followed; otherwise a name the card imports, then the card's own folder and each folder above it, nearest first, then the folders it imports with *, and last a search of the whole vault. A search that finds two cards chooses neither, and the reference has to be qualified, [[books/dune]]. @[[dune]] in a program is resolved as though written at the namespace root. Where dune.hmd and dune.md sit side by side, [[dune]] is the .hmd card and [[dune.md]] the other, whose name keeps its extension.

Other vaults

A second vault is a second namespace. It is reached as a configured endpoint of kind vault (endpoints by id), with its own cards and graph, and its cards are never merged into cards: a wikilink in one vault never resolves into another, and a relation between cards of two vaults is a join's result, not a link. The field station of the data sources examples configures two:

journal = source("journal")
archive = source("archive")
[journal.cards | map(c => c.name), archive.cards | map(c => c.name)]
[["kite", "owl", "wren"], ["finch", "heron"]]

What is next

A document that writes [[frank-herbert]] links to another. Chapter 22 follows the links, and the cards and links together are a graph.