37. The registry and publishing¶
A package on your machine installs from its directory. This chapter is how a package reaches other machines: the repository that holds the official packages, the registry that names them, and publishing a package's crate to crates.io.
Official and known packages¶
hql-extensions is the
repository of official packages. It is a Cargo workspace with one crate
per package under crates/, and one file at its root,
hql-packages.yaml, the registry.
| Kind | Where its crate is kept | Who maintains it |
|---|---|---|
| official | under crates/ of hql-extensions |
the HQL project |
| known | anywhere, pinned by the registry to a crates.io version or a git revision | its author |
| a path package | on your machine | you; it is in no registry |
The registry is a list of names. It decides what ml means when someone
writes hql package install ml, and it is reviewed, because a package is
native code with the rights of the hql process
(chapter 35).
A registry entry¶
registry: 1
packages:
- name: ml
crate: hql-ext-ml
kind: official
version: 0.1.0
hql: ">=0.7.0, <0.8.0"
provides: [ml, ml.metrics]
description: Datasets from records, linear and logistic regression and k-means as values, prediction, and metrics
| Key | States |
|---|---|
name |
the package's name, unique in the registry and distinct from every builtin package |
crate |
its crate, hql-ext-<name> |
kind |
official or known |
version |
the version the entry pins; for an official package the tag <crate>-v<version> of the repository |
hql |
the hql versions the package builds against |
provides |
the modules it provides, all under its own name |
The hql key is read before anything is fetched. A package whose range
excludes the running hql is refused there, with both versions named.
The same range is in the crate's own Cargo.toml, as hql = "0.7".
Version ranges and exact versions¶
Two version statements concern a package, and they answer different questions.
| Statement | Where | Answers |
|---|---|---|
a range, hql = "0.7" |
the crate's Cargo.toml, and hql: in the registry |
which hql versions is this source written for? |
an exact version, hql = "0.7.0" |
[build] in an installed package.toml |
which hql was this library built against? |
A crate is published with the range. One release of hql-ext-ml serves
every hql 0.7, because each install builds it again against the hql
that runs the install. The exact version exists only in a build
(the version rule).
Installing by name¶
hql package install ml reads the registry, checks the entry's hql
range, fetches the crate at the pinned version, and then does what
install --path does: builds it against the running hql and installs
the result.
hql package search metrics
ml 0.1.0 (official, ml, ml.metrics)
Datasets from records, linear and logistic regression and k-means as values, prediction, and metrics
hql package install ml
| Command | What it does |
|---|---|
hql package install <name> |
install the registry's pinned version of a package |
hql package install <name>@<version> |
install another released version of an official package |
hql package search <text> |
registry entries whose name, modules or description hold the text |
hql package update [<name>] |
install the version the registry now pins, where the installed one differs |
hql package registry [--refresh] |
where the registry is read from, when it was fetched, and how many packages it lists |
The source is kept under .hql/cache/src/<name>/ of the project or the HQL
home it was installed into, which is what rebuild
builds again. For an official package it is a checkout of the repository at
the entry's tag, holding that one crate; fetching it needs git.
The registry is fetched once a day and kept under ~/.hql/registry/. A
command that needs it uses the copy while it is less than a day old, and
also when the fetch fails, so a machine that is offline still lists and
rebuilds what it has. hql package registry --refresh fetches it now.
A package installed from a path is not the registry's: update leaves it
alone, and rebuild builds it from its directory.
Publishing a crate¶
A package's crate is published to crates.io under its
crate name, so that a pinned version of it can be fetched by anyone. The
Cargo.toml of a published package states what crates.io requires and
what ships:
[package]
name = "hql-ext-ml"
version = "0.1.0"
description = "HQL package `ml`: datasets, fitted models as values, prediction and metrics"
license = "MIT"
repository = "https://github.com/ewiger/hql-extensions"
readme = "README.md"
include = ["/src", "/modules", "/examples/*.hql", "/examples/.hql/config.toml", "/README.md"]
[dependencies]
hql = "0.7"
The module root is part of the published crate: include lists
/modules, since a crate without its contracts is not a package.
Three rules follow from how Cargo publishes:
hqlis a version range from crates.io. A crate with a path or git dependency cannot be published, so thehqlversion a package names is on crates.io before the package is.- Everything else is ordinary. A package depends on any crate, and those dependencies are built on the installing machine with the rest.
- A crate does not enable features of the crates it shares with
hql. The values the two exchange hold types of those crates, and a feature can change a type (every refusal).
cargo publish --dry-run builds the crate as crates.io would, from the
packaged files alone, and is worth running before a tag:
cargo publish -p hql-ext-ml --dry-run
A release of an official package¶
In hql-extensions a package is released by a tag:
- Set
versionin the crate'sCargo.tomland in its registry entry. - Merge to
main. CI tests the crate and runs every program under itsexamples/against a build ofhql. - Push the tag
<crate>-v<version>,hql-ext-ml-v0.1.0.
The release workflow checks that the tag, the crate and the registry
entry state one version, and runs cargo publish for that crate alone.
Registering a package of your own¶
A known package is added by a pull request to hql-extensions that adds
one entry to hql-packages.yaml. The review checks that:
- the crate is named
hql-ext-<name>, the name is one lowercase word no other package uses, and the modules are under that name; [package.metadata.hql]states the package, its modules and its module root, and the contracts under the root declare those modules;- the entry is pinned, to a crates.io version or a git revision, and
states the
hqlversions it builds against; - the crate builds and its examples run against those versions.
A package nobody registers is still a package. It installs with
--path, and a team shares it as it shares any repository.
Prebuilt libraries¶
Design only
This section describes functionality that is not in the current release. It is written as if it existed, and names may change before it ships.
Building from source needs a Rust toolchain and a minute. A prebuilt
library removes both for official packages: each hql release is built
with one exact compiler, and the official packages are built with that
same compiler for that same release, per platform, and published with
checksums. hql package install then fetches the library that matches
the running hql exactly, verifies its checksum, and falls back to
building from source where none matches.
The version rule is unchanged. A prebuilt library is one that happens to
have been built for this hql already, and the header check of
chapter 35 holds
it to that.
What is next¶
Chapter 38 uses a real package: ml, with
datasets, three kinds of model and their metrics.