Skip to content

23. Finding cards by meaning

filter answers a question with a yes or no per card. Retrieval answers a question in words with a ranking: every card that matches, best first, each with a score. HQL has two retrievals over a vault's cards, both HyperMarkdown's, in the builtin package hmd.

Lexical retrieval

lexical matches the words of a query against the words of each card. It needs nothing built in advance:

import hmd
cards | lexical("desert spice") | take(3)
[dune (0.2928), frankenstein-inspires-dune (0.2778), frankenstein (0.2471)]

Both retrievals are modules of the builtin package hmd, hmd.search.lexical and hmd.search.semantic, and hmd's prelude imports them: import hmd makes lexical and semantic steps of the program. The reading-club's project imports hmd for every program, so the import hmd above restates it. Without hmd, the name is not a step, and the error says which import provides it:

["desert spice"] | lexical("desert spice")
check error[unknown-name]: `lexical` is declared in the module `hmd.search.lexical`: write `import hmd`
 --> <expression>:1:20
  |
1 | ["desert spice"] | lexical("desert spice")
  |                    ^^^^^^^^^^^^^^^^^^^^^^^

A ranking holds hits

A ranking is a sequence of hits. A hit holds the card it retrieved, as value, and its score:

import hmd
cards
| lexical("desert spice")
| take(2)
| map(h => {card: h.value.name, score: h.score})
[{card: "dune", score: 0.2927674902397191}, {card: "frankenstein-inspires-dune", score: 0.2777641653663699}]

A score is evidence that a card is relevant to the query, not relevance itself, so every hit carries the retrieval that produced it — the model, the index, the metric, and whether the search was approximate:

import hmd
ranked = cards | lexical("desert")
{model: ranked.retrieval.model, metric: ranked.retrieval.metric, approximate: ranked.retrieval.approximate}
{approximate: false, metric: "cosine", model: "hql.hashbag.v1"}

take keeps a ranking a ranking, in the retrieval's order. Any other order is the program's own: sorting a ranking by another key makes it a plain sequence of hits.

Retrieval, then the graph

A ranking is a collection of cards, so every step of the earlier chapters applies. expand grows a graph from the best hits, and the nodes it adds say they were reached, not retrieved:

import hmd
cards
| lexical("a creature and its maker")
| take(2)
| expand(depth = 1)
| text
club  (expanded from frankenstein at depth 1)
  -> dune
  -> frankenstein
  -> leaves-of-grass
dune  (expanded from frankenstein at depth 1)
frankenstein  (retrieved 0.2064)
  -> mary-shelley
  -> dune
leaves-of-grass  (expanded from walt-whitman at depth 1)
  -> walt-whitman
mary-shelley  (expanded from frankenstein at depth 1)
  -> frankenstein
walt-whitman  (retrieved 0.1261)
  -> leaves-of-grass

Semantic retrieval

lexical matches spellings: a card that never uses the query's words scores nothing, and a card that uses common ones scores something. semantic scores by meaning, against embeddings a language model computed ahead of time into an index the vault names in .hql/config.toml:

[semantics]
index = ".hmd/index.sqlite"

Asked for a creature and its maker, semantic ranks Frankenstein, the relation card that names it, and Mary Shelley, though none of the three contains the word "creature":

import hmd
cards | semantic("a creature and its maker") | take(3) | map(h => h.value.title)
["Frankenstein", "Frankenstein inspires Dune", "Mary Shelley"]

The same question, by spelling, finds Frankenstein and then two cards that share only its small words:

import hmd
cards | lexical("a creature and its maker") | take(3) | map(h => h.value.title)
["Frankenstein", "Walt Whitman", "Librarian"]

A semantic ranking reports the model that embedded the cards, so two indexes built by two models are never compared by score:

import hmd
ranked = cards | semantic("a creature and its maker")
{model: ranked.retrieval.model, metric: ranked.retrieval.metric, approximate: ranked.retrieval.approximate}
{approximate: false, metric: "cosine", model: "sentence-transformers/all-MiniLM-L6-v2"}

The binary never loads a model: HyperMarkdown's hmd-semantics embeds the cards and builds the index, and embeds each question the index will be asked, named at build time with --query. The reading-club vault's index holds the three questions this book asks. A question it does not hold is a diagnostic naming the build command, not a ranking of nothing:

import hmd
cards | semantic("night hunting birds") | take(1)
execute failure[invalid-argument]: the index holds no vector for `night hunting birds`, and this binary has no model to make one: rerun `hmd-semantics build --query "night hunting birds"`
 --> <expression>:2:9
  |
2 | cards | semantic("night hunting birds") | take(1)
  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

hql doctor checks the chain — vault, configuration, index, model, query vectors — and names the first part that is missing.

What is next

A ranking, a graph and a reading list are values a program prints. Chapter 24 writes them into the document that asks for them, and builds the document from its queries.