Skip to content

30. Digests and encodings

A digest is the fixed-size output of a cryptographic hash function over a sequence of octets: equal input gives an equal digest, and a change to any octet changes it. An encoding writes octets as text-safe characters and reads them back. This chapter covers the two modules that provide them, crypto.digest and encoding, over a String, a Bytes value and a Data tree.

The digest module

crypto.digest is not in the prelude. A program writes import crypto.digest and reaches the module as digest, and its receivers, "abc".digest(Sha256); without the import, the name is a check error:

"abc".digest(Sha256)
check error[unknown-name]: `digest` is declared in the module `crypto.digest`: write `import crypto.digest`
 --> <expression>:1:1
  |
1 | "abc".digest(Sha256)
  | ^^^^^^^^^^^^^^^^^^^^

An algorithm is named by a type, as a format is: Md5, Sha1, Sha256, Sha512 and Blake3. s.digest(A) is the digest of a text's UTF-8 octets under A, written as lowercase hexadecimal, two digits per octet:

import crypto.digest
s = "abc"
[s.digest(Md5), s.digest(Sha1), s.digest(Sha256), s.digest(Blake3)]
["900150983cd24fb0d6963f7d28e17f72", "a9993e364706816aba3e25717850c26c9cd0d89d", "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", "6437b3ac38465133ffb63b75273a8db548c558465d79db03fd359c6cd5bd9d85"]

The length of the text follows from the algorithm: 32 hex digits for Md5, 40 for Sha1, 64 for Sha256 and Blake3, 128 for Sha512. Md5 and Sha1 are broken for collision resistance: two different inputs with one digest can be constructed. They serve to compare with digests other tools wrote, not to protect anything new.

b.digest(A) takes a Bytes value, the form for octets that are not text. The two forms agree: the digest of Bytes(s) is the digest of s:

import crypto.digest
from encoding import *
Bytes("68656c6c6f", Hex).digest(Sha256) == "hello".digest(Sha256)
true

Fingerprints of data

fingerprint(x) is the SHA-256 digest of a Data tree's canonical JSON, the text canonical(x) gives. Canonical JSON follows RFC 8785: keys sorted, no whitespace, a number as its digits without trailing zeros, a time value as its ISO 8601 text and a Bytes leaf as its base64 text:

from crypto.digest import *
canonical({title: "Dune", pages: 412, price: 9.50, on: Date("2026-10-24")})
"{\"on\":\"2026-10-24\",\"pages\":412,\"price\":9.5,\"title\":\"Dune\"}"

Two trees equal under HQL equality have one canonical text, so one fingerprint. Key order, whitespace and 1.10 against 1.1 in the source text change neither:

from crypto.digest import *
a = Data("{\"b\": 1, \"a\": [1.10, \"x\"]}", Json)
b = Data("{ \"a\" : [1.1, \"x\"],\n  \"b\": 1.0 }", Json)
[fingerprint(a), fingerprint(b)]
["5b07d3c10bbccec19fd995c80fb3393b8fd39691dbcc412ac55c0d08262ed5af", "5b07d3c10bbccec19fd995c80fb3393b8fd39691dbcc412ac55c0d08262ed5af"]

A value's type does count: the number 9.50 and the text "9.50" are different leaves, with different fingerprints:

from crypto.digest import *
[fingerprint({price: 9.50}) == fingerprint({price: 9.5}), fingerprint({price: 9.50}) == fingerprint({price: "9.50"})]
[true, false]

fingerprint(x) is canonical(x).digest(Sha256), so a program in another language reproduces it with any RFC 8785 implementation and SHA-256.

The encoding module

encoding is not in the prelude either: a program writes import encoding, which reaches its receivers, "hé".encode(Hex). An encoding is named by a type:

Encoding Writes octets as
Hex two hexadecimal digits each, lowercase; either case reads
Base64 RFC 4648 base64, standard alphabet, padded
Base64Url RFC 4648 base64, URL-safe alphabet, unpadded; padding optional on reading
Percent RFC 3986 percent-encoding: A-Z a-z 0-9 - . _ ~ as they are, every other octet as %XX

s.encode(E) writes a text's UTF-8 octets in E:

import encoding
["hé".encode(Hex), "??>".encode(Base64), "??>".encode(Base64Url), "a b/é~".encode(Percent)]
["68c3a9", "Pz8+", "Pz8-", "a%20b%2F%C3%A9~"]

s.decode(E) reads the text back as an Option<String>. It is none where the input is not valid in E, and also where the octets it holds are not UTF-8, as /w== in base64 is the single octet 0xff:

import encoding
["68C3A9".decode(Hex), "aGk/".decode(Base64), "6g".decode(Hex), "/w==".decode(Base64)]
["hé", "hi?", none, none]

Percent-decoding reads %20 as a space and leaves + as it is:

import encoding
"a+b%20%C3%A9".decode(Percent)
"a+b é"

Between text and Bytes

Where the octets are not text, Bytes holds them. b.encode(E) writes a Bytes value in E, and s.to_bytes(E) reads it back as an Option<Bytes>, with no UTF-8 requirement; Bytes(s, E) reads it as a value and fails on invalid text. The page Bytes shows each:

from encoding import *
b = Bytes("ff00", Hex)
[b.encode(Hex), b.encode(Base64), b.encode(Base64Url)]
["ff00", "/wA=", "_wA"]
import encoding
["/w==".decode(Base64), "/w==".to_bytes(Base64)]
[none, 0xff]

In practice

A record already seen

A fingerprint is a key that holds outside the program: in a column, a file name, a cache. Records that arrive with their keys in another order, or as another program wrote them, still match the keys of an earlier load. Here loaded holds a 12-digit prefix of each fingerprint an earlier run stored, and only the new record passes:

from crypto.digest import *
loaded = {"106815777291"}
incoming = [{to: "ada", id: 1}, {id: 2, to: "bo"}]
incoming | filter(r => not loaded.contains(fingerprint(r)[0:12]))
[{id: 2, to: "bo"}]

Within one program, distinct already removes records equal under HQL equality, and Deduplication covers duplicates by key. The fingerprint is for the comparison that crosses a run.

A file's digest

source(file(p), Text) reads a file whole as one String, as Source shows. Its digest is the one shasum -a 256 prints for the file, so a program checks a download against a published digest:

import crypto.digest
expected = "ade391083e197c4575f5ccb436b35c499665e77202518e6873a02fa6b54cc8e3"
source(file("examples/io/counts.csv"), Text) | map(t => t.digest(Sha256) == expected)
[true]

The reference states every function of the two modules with its signature: crypto.digest and encoding.