23. Finding cards by meaning¶
filter answers a question with a yes or no per card. Retrieval answers a
question in words with a ranking: every card that matches, best first,
each with a score. HQL has two retrievals over a vault's cards, both
HyperMarkdown's, in the builtin package hmd.
Lexical retrieval¶
lexical matches the words of a query against the words of each card. It
needs nothing built in advance:
import hmd
cards | lexical("desert spice") | take(3)
[dune (0.2928), frankenstein-inspires-dune (0.2778), frankenstein (0.2471)]
Both retrievals are modules of the builtin package hmd,
hmd.search.lexical and hmd.search.semantic, and hmd's prelude imports
them: import hmd makes lexical and semantic steps of the program. The
reading-club's project imports hmd for every program, so the import hmd
above restates it. Without hmd, the name is not a step, and the error says
which import provides it:
["desert spice"] | lexical("desert spice")
check error[unknown-name]: `lexical` is declared in the module `hmd.search.lexical`: write `import hmd`
--> <expression>:1:20
|
1 | ["desert spice"] | lexical("desert spice")
| ^^^^^^^^^^^^^^^^^^^^^^^
A ranking holds hits¶
A ranking is a sequence of hits. A hit holds the card it retrieved, as
value, and its score:
import hmd
cards
| lexical("desert spice")
| take(2)
| map(h => {card: h.value.name, score: h.score})
[{card: "dune", score: 0.2927674902397191}, {card: "frankenstein-inspires-dune", score: 0.2777641653663699}]
A score is evidence that a card is relevant to the query, not relevance itself, so every hit carries the retrieval that produced it — the model, the index, the metric, and whether the search was approximate:
import hmd
ranked = cards | lexical("desert")
{model: ranked.retrieval.model, metric: ranked.retrieval.metric, approximate: ranked.retrieval.approximate}
{approximate: false, metric: "cosine", model: "hql.hashbag.v1"}
take keeps a ranking a ranking, in the retrieval's order. Any other order is
the program's own: sorting a ranking by another key makes it a plain sequence
of hits.
Retrieval, then the graph¶
A ranking is a collection of cards, so every step of the earlier chapters
applies. expand grows a graph from the best hits, and the nodes it adds say
they were reached, not retrieved:
import hmd
cards
| lexical("a creature and its maker")
| take(2)
| expand(depth = 1)
| text
club (expanded from frankenstein at depth 1)
-> dune
-> frankenstein
-> leaves-of-grass
dune (expanded from frankenstein at depth 1)
frankenstein (retrieved 0.2064)
-> mary-shelley
-> dune
leaves-of-grass (expanded from walt-whitman at depth 1)
-> walt-whitman
mary-shelley (expanded from frankenstein at depth 1)
-> frankenstein
walt-whitman (retrieved 0.1261)
-> leaves-of-grass
Semantic retrieval¶
lexical matches spellings: a card that never uses the query's words scores
nothing, and a card that uses common ones scores something. semantic scores
by meaning, against embeddings a language model computed ahead of time
into an index the vault names in .hql/config.toml:
[semantics]
index = ".hmd/index.sqlite"
Asked for a creature and its maker, semantic ranks Frankenstein, the
relation card that names it, and Mary Shelley, though none of the three
contains the word "creature":
import hmd
cards | semantic("a creature and its maker") | take(3) | map(h => h.value.title)
["Frankenstein", "Frankenstein inspires Dune", "Mary Shelley"]
The same question, by spelling, finds Frankenstein and then two cards that share only its small words:
import hmd
cards | lexical("a creature and its maker") | take(3) | map(h => h.value.title)
["Frankenstein", "Walt Whitman", "Librarian"]
A semantic ranking reports the model that embedded the cards, so two indexes built by two models are never compared by score:
import hmd
ranked = cards | semantic("a creature and its maker")
{model: ranked.retrieval.model, metric: ranked.retrieval.metric, approximate: ranked.retrieval.approximate}
{approximate: false, metric: "cosine", model: "sentence-transformers/all-MiniLM-L6-v2"}
The binary never loads a model: HyperMarkdown's hmd-semantics
embeds the cards and builds the index, and embeds each question the index
will be asked, named at build time with --query. The reading-club vault's index
holds the three questions this book asks. A question it does not hold is a
diagnostic naming the build command, not a ranking of nothing:
import hmd
cards | semantic("night hunting birds") | take(1)
execute failure[invalid-argument]: the index holds no vector for `night hunting birds`, and this binary has no model to make one: rerun `hmd-semantics build --query "night hunting birds"`
--> <expression>:2:9
|
2 | cards | semantic("night hunting birds") | take(1)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hql doctor checks the chain — vault, configuration, index, model, query
vectors — and names the first part that is missing.
What is next¶
A ranking, a graph and a reading list are values a program prints. Chapter 24 writes them into the document that asks for them, and builds the document from its queries.