7. Streams that do not end

Every collection so far had an end: a size, a last element. A stream is a succession of values whose production need not end — a log, a feed, the natural numbers. HQL types the difference, so a program cannot ask an endless stream a question that needs its end.

A function that yields

A fn whose body is a block, in braces, produces its values with yield, in the order each yield is reached. Its result type says whether production ends: Seq<T> promises an end, Stream<T> does not. These are the natural numbers:

fn naturals(from: Int) : Stream<Int> {
    yield from
    for n in naturals(from + 1) {
        yield n
    }
}
naturals(1) | take(5)

[1, 2, 3, 4, 5]

for name in collection { … } runs its block once per value. A block that yields is an iterating block: it holds yield, for and bindings, one per line (function bodies).

Calling naturals(1) evaluates nothing. Its value is a recipe for the numbers, and take(5) asks for five of them and then stops asking — which is what makes an endless producer usable.

One value at a time

map, filter and drop take a stream and give a stream. Each value passes through the whole pipeline before the next is produced, so only as many are made as the end of the pipeline asks for:

naturals(1)
| filter(n => n % 7 == 0)
| map(n => n * n)
| take(3)

[49, 196, 441]

Questions that need the end

sum, size, sort, group and the other whole-input steps need the last value before they can answer. Over a stream there may be none, and the checker says so before anything runs:

fn naturals(from: Int) : Stream<Int> {
    yield from
    for n in naturals(from + 1) {
        yield n
    }
}
naturals(1) | sum

check error[type-mismatch]: `sum` needs a finite input, and Stream<Int> may not end; bound it first with `take(n)`, or batch it with `collect(n)`
 --> <expression>:7:15
  |
7 | naturals(1) | sum
  |               ^^^

A stream becomes finite through take(n), or is answered in pieces through collect(n), which gathers consecutive values into batches of n:

naturals(1)
| collect(3)
| map(batch => batch | sum)
| take(3)

[6, 15, 24]

emit is the inverse of collect: it passes on every member of every batch, in order. flat_map(f) is map(f) | emit.

[[1, 2], [3]] | emit

[1, 2, 3]

A sequence over the shelf

A generator that promises an end is a Seq. This one yields the title of each book on the shelf of chapter 6, in title order — a sequence the shelf produces on demand:

shelf = [
    {title: "Frankenstein", pages: 280},
    {title: "Dune", pages: 412},
    {title: "The Hobbit", pages: 310},
    {title: "Leaves of Grass", pages: 145},
    {title: "A Room of One's Own", pages: 112},
    {title: "The Common Reader", pages: 304},
    {title: "Mrs Dalloway", pages: 194},
]
fn title(b: Data) : String = match b.title { some(String t) => t, _ => "" }

fn titles() : Seq<String> {
    for b in shelf | sort(by = title) {
        yield title(b)
    }
}
titles() | take(3)

["A Room of One's Own", "Dune", "Frankenstein"]

A list is such a sequence already. So are the documents of a file read with source, in chapter 9, produced as each one is read.

Writing a stream

Formatting an endless stream writes it piece by piece as it arrives: a line of text, a JSON value per line, a CSV row. hql flushes each piece, and a reader that stops reading ends the stream:

naturals(1) | map(n => n * n) | json

$ hql eval '…naturals… | map(n => n * n) | json' | head -3
1
4
9

A diagram format describes a whole, so | mermaid over a stream is a type error until take(n) bounds it.

What is next

Chapter 8 is about every format a value can be written in, and who chooses it.