28. Statistics¶
This chapter is about the module stats: descriptive statistics over a
collection of numbers. It covers the centre of the values (the mean, the
median and the mode), their spread, their quantiles, the correlation of two
columns, and what a statistic is over an input too small to define it. Each
statistic applies to a list written by hand and to a column a query reads.
The module¶
stats is not in the prelude. A program writes import stats and calls
stats.mean(xs), or binds them all, from stats import *,
and calls mean(xs). Without an import, a statistic is an unknown name, and
the checker names the import it needs:
[1, 2, 3, 4] | mean
check error[unknown-name]: `mean` is declared in the module `stats`: write `from stats import mean`
--> <expression>:1:16
|
1 | [1, 2, 3, 4] | mean
| ^^^^
The centre¶
mean is the sum divided by the count, median the middle value of the
sorted values, and the mean of the two middle values where the count is
even, and mode the most frequent value:
from stats import *
xs = [2, 4, 4, 4, 5, 5, 7, 9]
{mean: xs | mean, median: xs | median, mode: xs | mode}
{mean: 5.0, median: 4.5, mode: 4}
A statistic of Int or Float values is a Float. A statistic of Int
values is computed exactly and rounded to a Float once. A statistic of
Decimal values is a Decimal, exact except where a quotient does not
terminate, which is rounded half to even to 100 significant digits as
Decimal / is. A literal with a decimal point is a Decimal
(Decimal), so a list of prices keeps its cents:
from stats import *
prices = [1.10, 2.20, 3.30, 4.45, 5.05]
[prices | mean, prices | median]
[3.22, 3.30]
mode takes values of any Orderable type, not only numbers. Among equally
frequent values, the mode is the least by the type's order:
from stats import *
[[3, 1, 3, 1, 2] | mode, ["novel", "essay", "novel", "poetry"] | mode]
[1, "novel"]
Spread¶
variance is the sample variance, the sum of squared deviations from the
mean divided by n - 1, and stdev its square root. pvariance is the
population variance, the same sum divided by n, and pstdev its square
root:
import stats
from stats import *
xs = [2, 4, 4, 4, 5, 5, 7, 9]
[xs | stats.variance, xs | stdev, xs | pvariance, xs | pstdev]
[4.571428571428571, 2.138089935299395, 4.0, 2.0]
The prelude declares a variance too: the population variance of Int and
Float values. With variance imported from stats, a bare variance over
those is ambiguous, and the checker asks for the module's name:
from stats import *
[1, 2, 3, 4] | variance
check error[unknown-name]: `variance` over List<Int> is declared by `collections` and `stats`, so the call is ambiguous; qualify it, `collections.variance(…)`
--> <expression>:2:16
|
2 | [1, 2, 3, 4] | variance
| ^^^^^^^^
stats.variance is the sample variance and collections.variance the population
variance. Over Decimal values only stats declares one, and a bare
variance is the sample variance.
Quantiles¶
quantile(q) is the value below which the fraction q of the sorted values
lies, by linear interpolation between the two nearest values: the method R
and NumPy use by default. quantiles(n) is the n - 1 cut points that divide
the sorted values into n groups of equal probability, so quantiles(4) is
the three quartiles and quantiles(100) the percentiles:
from stats import *
xs = [2, 4, 4, 4, 5, 5, 7, 9]
{p90: xs | quantile(0.9), quartiles: xs | quantiles(4)}
{p90: 7.6, quartiles: [4.0, 4.5, 5.5]}
median is quantile(0.5). Over Int and Decimal values a quantile is
computed exactly, so the 95th percentile of the prices is a Decimal:
from stats import *
[1.10, 2.20, 3.30, 4.45, 5.05] | quantile(0.95)
4.9300
A q outside 0.0 to 1.0, or an n below 2, is an evaluation error:
from stats import *
[1, 2, 3] | quantile(1.5)
execute failure[invalid-argument]: `quantile` needs a `q` from 0 to 1, and 1.5 is outside it
--> <expression>:2:22
|
2 | [1, 2, 3] | quantile(1.5)
| ^^^
Correlation¶
covariance and correlation take a collection of pairs: the first column
in each pair's first, the second in its second. zip builds the pairs
from two lists. covariance is the sample covariance, and correlation
Pearson's coefficient, the covariance divided by the product of the two
standard deviations, from -1.0 to 1.0:
from stats import *
pairs = zip([1, 2, 3, 4, 5, 6], [2, 1, 4, 3, 7, 5])
[pairs | covariance, pairs | correlation]
[3.2, 0.7917946548886297]
A column of a query¶
A statistic takes any collection, so it ends a pipeline as sum and size
do. The books of the reading-club vault (chapter 21) carry
their pages in the header, and a typed view reads them as numbers:
from stats import *
fn pages(c: Card) : Int = match c.metadata.pages { some(Int n) => n, _ => 0 }
books = cards | filter(c => c.metadata.category == "book")
lengths = books | map(pages)
{mean: lengths | mean, median: lengths | median, stdev: lengths | stdev, quartiles: lengths | quantiles(4)}
{mean: 251.0, median: 280.0, quartiles: [169.5, 280.0, 307.0], stdev: 105.56040924513319}
A statistic per group is a map over the groups, as each aggregate of
chapter 6 was:
from stats import *
fn pages(c: Card) : Int = match c.metadata.pages { some(Int n) => n, _ => 0 }
cards
| filter(c => c.metadata.category == "book")
| group(by = c => c.metadata.genre)
| map(g => {genre: g.key, books: g.items | size, mean: g.items | map(pages) | mean})
{{books: 2, genre: "essay", mean: 208.0}, {books: 4, genre: "novel", mean: 299.0}, {books: 1, genre: "poetry", mean: 145.0}}
Two columns of one query are a column of pairs. Whether longer books are the later ones is the correlation of the year and the pages:
from stats import *
fn pages(c: Card) : Int = match c.metadata.pages { some(Int n) => n, _ => 0 }
fn year(c: Card) : Int = match c.metadata.year { some(Int n) => n, _ => 0 }
cards
| filter(c => c.metadata.category == "book")
| map(b => Pair(year(b), pages(b)))
| correlation
0.33335874566223495
An empty input¶
A statistic that is undefined over its input is none, so each statistic
but quantiles is an Option. The mean, median and mode of nothing are
none. The sample variance and standard deviation need two values, so of
one value they are none, while the population variance of one value is
0.0. The correlation is none also where one side has no variance:
from stats import *
nothing = [1, 2, 3] | filter(n => n > 5)
[nothing | mean, nothing | median, [4] | stdev, [4] | pvariance, zip([1, 2, 3], [5, 5, 5]) | correlation]
[none, none, none, 0.0, none]
quantiles of nothing is an empty list. Where a program needs a number
whatever the input, unwrap_or gives a default
(Option); 0.0f is a Float literal:
from stats import *
nothing = [1, 2, 3] | filter(n => n > 5)
nothing | mean | unwrap_or(0.0f)
0.0
What is next¶
A statistic describes values a program has. Chapter 29 is about values a program draws: samples, shuffles and random numbers, and the seed that makes them repeatable.