30. Digests and encodings¶
A digest is the fixed-size output of a cryptographic hash function over a
sequence of octets: equal input gives an equal digest, and a change to any
octet changes it. An encoding writes octets as text-safe characters and
reads them back. This chapter covers the two modules that provide them,
crypto.digest and encoding, over a String, a
Bytes value and a Data tree.
The digest module¶
crypto.digest is not in the prelude. A program writes
import crypto.digest and reaches the module as digest, and its
receivers, "abc".digest(Sha256); without the import, the name is a check
error:
"abc".digest(Sha256)
check error[unknown-name]: `digest` is declared in the module `crypto.digest`: write `import crypto.digest`
--> <expression>:1:1
|
1 | "abc".digest(Sha256)
| ^^^^^^^^^^^^^^^^^^^^
An algorithm is named by a type, as a format is: Md5, Sha1, Sha256,
Sha512 and Blake3. s.digest(A) is the digest of a text's UTF-8 octets
under A, written as lowercase hexadecimal, two digits per octet:
import crypto.digest
s = "abc"
[s.digest(Md5), s.digest(Sha1), s.digest(Sha256), s.digest(Blake3)]
["900150983cd24fb0d6963f7d28e17f72", "a9993e364706816aba3e25717850c26c9cd0d89d", "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", "6437b3ac38465133ffb63b75273a8db548c558465d79db03fd359c6cd5bd9d85"]
The length of the text follows from the algorithm: 32 hex digits for Md5,
40 for Sha1, 64 for Sha256 and Blake3, 128 for Sha512. Md5 and
Sha1 are broken for collision resistance: two different inputs with one
digest can be constructed. They serve to compare with digests other tools
wrote, not to protect anything new.
b.digest(A) takes a Bytes value, the form for octets that are not text.
The two forms agree: the digest of Bytes(s) is the digest of s:
import crypto.digest
from encoding import *
Bytes("68656c6c6f", Hex).digest(Sha256) == "hello".digest(Sha256)
true
Fingerprints of data¶
fingerprint(x) is the SHA-256 digest of a Data tree's canonical
JSON, the text canonical(x) gives. Canonical JSON follows RFC 8785:
keys sorted, no whitespace, a number as its digits without trailing zeros,
a time value as its ISO 8601 text and a Bytes leaf as its base64 text:
from crypto.digest import *
canonical({title: "Dune", pages: 412, price: 9.50, on: Date("2026-10-24")})
"{\"on\":\"2026-10-24\",\"pages\":412,\"price\":9.5,\"title\":\"Dune\"}"
Two trees equal under HQL equality have one canonical text, so one
fingerprint. Key order, whitespace and 1.10 against 1.1 in the source
text change neither:
from crypto.digest import *
a = Data("{\"b\": 1, \"a\": [1.10, \"x\"]}", Json)
b = Data("{ \"a\" : [1.1, \"x\"],\n \"b\": 1.0 }", Json)
[fingerprint(a), fingerprint(b)]
["5b07d3c10bbccec19fd995c80fb3393b8fd39691dbcc412ac55c0d08262ed5af", "5b07d3c10bbccec19fd995c80fb3393b8fd39691dbcc412ac55c0d08262ed5af"]
A value's type does count: the number 9.50 and the text "9.50" are
different leaves, with different fingerprints:
from crypto.digest import *
[fingerprint({price: 9.50}) == fingerprint({price: 9.5}), fingerprint({price: 9.50}) == fingerprint({price: "9.50"})]
[true, false]
fingerprint(x) is canonical(x).digest(Sha256), so a program in another
language reproduces it with any RFC 8785 implementation and SHA-256.
The encoding module¶
encoding is not in the prelude either: a program writes import encoding,
which reaches its receivers, "hé".encode(Hex).
An encoding is named by a type:
| Encoding | Writes octets as |
|---|---|
Hex |
two hexadecimal digits each, lowercase; either case reads |
Base64 |
RFC 4648 base64, standard alphabet, padded |
Base64Url |
RFC 4648 base64, URL-safe alphabet, unpadded; padding optional on reading |
Percent |
RFC 3986 percent-encoding: A-Z a-z 0-9 - . _ ~ as they are, every other octet as %XX |
s.encode(E) writes a text's UTF-8 octets in E:
import encoding
["hé".encode(Hex), "??>".encode(Base64), "??>".encode(Base64Url), "a b/é~".encode(Percent)]
["68c3a9", "Pz8+", "Pz8-", "a%20b%2F%C3%A9~"]
s.decode(E) reads the text back as an Option<String>. It is none where
the input is not valid in E, and also where the octets it holds are not
UTF-8, as /w== in base64 is the single octet 0xff:
import encoding
["68C3A9".decode(Hex), "aGk/".decode(Base64), "6g".decode(Hex), "/w==".decode(Base64)]
["hé", "hi?", none, none]
Percent-decoding reads %20 as a space and leaves + as it is:
import encoding
"a+b%20%C3%A9".decode(Percent)
"a+b é"
Between text and Bytes¶
Where the octets are not text, Bytes holds them. b.encode(E) writes a
Bytes value in E, and s.to_bytes(E) reads it back as an
Option<Bytes>, with no UTF-8 requirement; Bytes(s, E) reads it as a
value and fails on invalid text. The page Bytes
shows each:
from encoding import *
b = Bytes("ff00", Hex)
[b.encode(Hex), b.encode(Base64), b.encode(Base64Url)]
["ff00", "/wA=", "_wA"]
import encoding
["/w==".decode(Base64), "/w==".to_bytes(Base64)]
[none, 0xff]
In practice¶
A record already seen¶
A fingerprint is a key that holds outside the program: in a column, a file
name, a cache. Records that arrive with their keys in another order, or as
another program wrote them, still match the keys of an earlier load. Here
loaded holds a 12-digit prefix of each fingerprint an earlier run stored,
and only the new record passes:
from crypto.digest import *
loaded = {"106815777291"}
incoming = [{to: "ada", id: 1}, {id: 2, to: "bo"}]
incoming | filter(r => not loaded.contains(fingerprint(r)[0:12]))
[{id: 2, to: "bo"}]
Within one program, distinct already removes records equal under HQL
equality, and Deduplication covers duplicates
by key. The fingerprint is for the comparison that crosses a run.
A file's digest¶
source(file(p), Text) reads a file whole as one String, as
Source shows. Its digest is the one shasum -a 256
prints for the file, so a program checks a download against a published
digest:
import crypto.digest
expected = "ade391083e197c4575f5ccb436b35c499665e77202518e6873a02fa6b54cc8e3"
source(file("examples/io/counts.csv"), Text) | map(t => t.digest(Sha256) == expected)
[true]
The reference states every function of the two modules with its signature: crypto.digest and encoding.