Skip to content

28. Statistics

This chapter is about the module stats: descriptive statistics over a collection of numbers. It covers the centre of the values (the mean, the median and the mode), their spread, their quantiles, the correlation of two columns, and what a statistic is over an input too small to define it. Each statistic applies to a list written by hand and to a column a query reads.

The module

stats is not in the prelude. A program writes import stats and calls stats.mean(xs), or binds them all, from stats import *, and calls mean(xs). Without an import, a statistic is an unknown name, and the checker names the import it needs:

[1, 2, 3, 4] | mean
check error[unknown-name]: `mean` is declared in the module `stats`: write `from stats import mean`
 --> <expression>:1:16
  |
1 | [1, 2, 3, 4] | mean
  |                ^^^^

The centre

mean is the sum divided by the count, median the middle value of the sorted values, and the mean of the two middle values where the count is even, and mode the most frequent value:

from stats import *
xs = [2, 4, 4, 4, 5, 5, 7, 9]
{mean: xs | mean, median: xs | median, mode: xs | mode}
{mean: 5.0, median: 4.5, mode: 4}

A statistic of Int or Float values is a Float. A statistic of Int values is computed exactly and rounded to a Float once. A statistic of Decimal values is a Decimal, exact except where a quotient does not terminate, which is rounded half to even to 100 significant digits as Decimal / is. A literal with a decimal point is a Decimal (Decimal), so a list of prices keeps its cents:

from stats import *
prices = [1.10, 2.20, 3.30, 4.45, 5.05]
[prices | mean, prices | median]
[3.22, 3.30]

mode takes values of any Orderable type, not only numbers. Among equally frequent values, the mode is the least by the type's order:

from stats import *
[[3, 1, 3, 1, 2] | mode, ["novel", "essay", "novel", "poetry"] | mode]
[1, "novel"]

Spread

variance is the sample variance, the sum of squared deviations from the mean divided by n - 1, and stdev its square root. pvariance is the population variance, the same sum divided by n, and pstdev its square root:

import stats
from stats import *
xs = [2, 4, 4, 4, 5, 5, 7, 9]
[xs | stats.variance, xs | stdev, xs | pvariance, xs | pstdev]
[4.571428571428571, 2.138089935299395, 4.0, 2.0]

The prelude declares a variance too: the population variance of Int and Float values. With variance imported from stats, a bare variance over those is ambiguous, and the checker asks for the module's name:

from stats import *
[1, 2, 3, 4] | variance
check error[unknown-name]: `variance` over List<Int> is declared by `collections` and `stats`, so the call is ambiguous; qualify it, `collections.variance(…)`
 --> <expression>:2:16
  |
2 | [1, 2, 3, 4] | variance
  |                ^^^^^^^^

stats.variance is the sample variance and collections.variance the population variance. Over Decimal values only stats declares one, and a bare variance is the sample variance.

Quantiles

quantile(q) is the value below which the fraction q of the sorted values lies, by linear interpolation between the two nearest values: the method R and NumPy use by default. quantiles(n) is the n - 1 cut points that divide the sorted values into n groups of equal probability, so quantiles(4) is the three quartiles and quantiles(100) the percentiles:

from stats import *
xs = [2, 4, 4, 4, 5, 5, 7, 9]
{p90: xs | quantile(0.9), quartiles: xs | quantiles(4)}
{p90: 7.6, quartiles: [4.0, 4.5, 5.5]}

median is quantile(0.5). Over Int and Decimal values a quantile is computed exactly, so the 95th percentile of the prices is a Decimal:

from stats import *
[1.10, 2.20, 3.30, 4.45, 5.05] | quantile(0.95)
4.9300

A q outside 0.0 to 1.0, or an n below 2, is an evaluation error:

from stats import *
[1, 2, 3] | quantile(1.5)
execute failure[invalid-argument]: `quantile` needs a `q` from 0 to 1, and 1.5 is outside it
 --> <expression>:2:22
  |
2 | [1, 2, 3] | quantile(1.5)
  |                      ^^^

Correlation

covariance and correlation take a collection of pairs: the first column in each pair's first, the second in its second. zip builds the pairs from two lists. covariance is the sample covariance, and correlation Pearson's coefficient, the covariance divided by the product of the two standard deviations, from -1.0 to 1.0:

from stats import *
pairs = zip([1, 2, 3, 4, 5, 6], [2, 1, 4, 3, 7, 5])
[pairs | covariance, pairs | correlation]
[3.2, 0.7917946548886297]

A column of a query

A statistic takes any collection, so it ends a pipeline as sum and size do. The books of the reading-club vault (chapter 21) carry their pages in the header, and a typed view reads them as numbers:

from stats import *
fn pages(c: Card) : Int = match c.metadata.pages { some(Int n) => n, _ => 0 }
books = cards | filter(c => c.metadata.category == "book")
lengths = books | map(pages)
{mean: lengths | mean, median: lengths | median, stdev: lengths | stdev, quartiles: lengths | quantiles(4)}
{mean: 251.0, median: 280.0, quartiles: [169.5, 280.0, 307.0], stdev: 105.56040924513319}

A statistic per group is a map over the groups, as each aggregate of chapter 6 was:

from stats import *
fn pages(c: Card) : Int = match c.metadata.pages { some(Int n) => n, _ => 0 }
cards
| filter(c => c.metadata.category == "book")
| group(by = c => c.metadata.genre)
| map(g => {genre: g.key, books: g.items | size, mean: g.items | map(pages) | mean})
{{books: 2, genre: "essay", mean: 208.0}, {books: 4, genre: "novel", mean: 299.0}, {books: 1, genre: "poetry", mean: 145.0}}

Two columns of one query are a column of pairs. Whether longer books are the later ones is the correlation of the year and the pages:

from stats import *
fn pages(c: Card) : Int = match c.metadata.pages { some(Int n) => n, _ => 0 }
fn year(c: Card) : Int = match c.metadata.year { some(Int n) => n, _ => 0 }
cards
| filter(c => c.metadata.category == "book")
| map(b => Pair(year(b), pages(b)))
| correlation
0.33335874566223495

An empty input

A statistic that is undefined over its input is none, so each statistic but quantiles is an Option. The mean, median and mode of nothing are none. The sample variance and standard deviation need two values, so of one value they are none, while the population variance of one value is 0.0. The correlation is none also where one side has no variance:

from stats import *
nothing = [1, 2, 3] | filter(n => n > 5)
[nothing | mean, nothing | median, [4] | stdev, [4] | pvariance, zip([1, 2, 3], [5, 5, 5]) | correlation]
[none, none, none, 0.0, none]

quantiles of nothing is an empty list. Where a program needs a number whatever the input, unwrap_or gives a default (Option); 0.0f is a Float literal:

from stats import *
nothing = [1, 2, 3] | filter(n => n > 5)
nothing | mean | unwrap_or(0.0f)
0.0

What is next

A statistic describes values a program has. Chapter 29 is about values a program draws: samples, shuffles and random numbers, and the seed that makes them repeatable.