2. Data types¶
HQL is a statically typed expression and query language. Its central idea is that a type is a contract. A type describes a value: the fields of a record, the type of each field, and any rules the values must follow. HQL checks values against their types at two points.
The first is before the program runs. HQL has no compiler, but hql
type-checks the whole program first, the way a compiler would. It infers the
type of every expression and rejects the program if anything does not fit.
This is what makes HQL statically typed.
The second is when data comes in. When a program reads a file, an event or a
row as some type, hql validates it against that type and rejects anything
that does not match. Values are never converted implicitly. This is what
makes HQL strongly typed.
A query language needs both. Static checking finds mistakes in the program, but it cannot see the data, which does not exist yet. Validation finds bad data, but on its own it reports a mistake in the program only when some data happens to trigger it.
Types as data contracts¶
On a data platform, a data contract is an agreement between the team that produces a table and the teams that consume it. It lists the columns, their types, and rules such as "an amount is never negative". It usually lives outside the code, in a schema registry or a validation job.
In HQL, the contract is the type:
type BookData : DataContract = {
title: String
pages: Int
require pages > 0
}
The checker uses BookData to check the query, and the same declaration
validates the rows when the program runs.
Checking the program¶
Here the program asks for page, but the field is called pages:
type BookData : DataContract = {
title: String
pages: Int
}
raw : List<Data> = [{title: "Dune", pages: 412}]
raw | BookData | map(b => b.page / 2)
check error[unknown-field]: BookData has no field `page`
--> <expression>:7:27
|
7 | raw | BookData | map(b => b.page / 2)
| ^^^^^^
This is a check error. The checker found it before anything ran, so it does
not matter how many books there are.
Validating the data¶
Data from outside arrives as Data, a tree of records, lists and scalars,
like JSON or YAML. Data is a type, but it does not say which fields are
there. Reading it as a contract does:
type BookData : DataContract = {
title: String
pages: Int
require pages > 0
}
raw : List<Data> = [{title: "Dune", pages: 412}, {title: "Emma", pages: "many"}]
raw | BookData | map(b => b.pages)
execute error[contract-violation]: this does not read as BookData: field `pages`: "many" is not Int
--> <expression>:9:1
|
9 | raw | BookData | map(b => b.pages)
| ^^^^^^^^^^^^^^
= expected: BookData
This time the program is fine and the data is wrong, so the error is an
execute error. The bad record is rejected as it comes in, before any step
uses it. Every value after that point is a valid BookData, as the checker
assumed.
Open and closed types¶
A closed type, declared as a TypeContract, describes the whole record. A
record with any other field is rejected:
type Book : TypeContract = {
title: String
pages: Int
}
raw : List<Data> = [{title: "Dune", pages: 412, shelf: "sf"}]
raw | Book
execute error[contract-violation]: this does not read as Book: field `shelf`: Book declares no such field
--> <expression>:7:1
|
7 | raw | Book
| ^^^^^^^^^^
= expected: Book
An open contract, declared as a DataContract, lists only the fields the
program needs. Other fields are allowed and kept:
type BookData : DataContract = {
title: String
pages: Int
}
raw : List<Data> = [{title: "Dune", pages: 412, shelf: "sf"}]
raw | BookData | map(b => b.pages)
[412]
Use open contracts for data from other systems, which often adds fields you do not need. Use closed types for values your own program builds, where you decide the exact shape. Data contracts covers rules, defaults and validation in detail.
Contracts are built from simpler types: numbers, text, dates and times, and
Data. The rest of this chapter covers those, starting in the REPL.
Trying values in the REPL¶
hql repl reads one input at a time, evaluates it and prints the value. A
binding is kept for the inputs after it, and :type asks for a type without
evaluating anything:
$ hql repl
hql 0.5.0. `:help` for commands, `:quit` to leave.
hql> pages = 412
()
hql> pages / 3
137
hql> pages % 3
1
hql> :type pages
Int
hql> title = "Dune"
()
hql> title.upper()
"DUNE"
hql> :type title
String
hql> fmt("{} has {} pages", title, pages)
"Dune has 412 pages"
hql> :quit
412 is an Int, so / divides whole numbers and drops the remainder, and
% gives the remainder. "Dune" is a String, and title.upper() calls a
function of String on it. fmt writes values into text, one {} each.
The same programs run with hql eval, as every program in this book does:
pages = 412
title = "Dune"
fmt("{} has {} pages", title, pages)
"Dune has 412 pages"
Type checking¶
Each operation says which types it takes. An Int and a String do not add,
and the checker rejects the program before it runs:
412 + "pages"
check error[type-mismatch]: addition requires two Int, two Float or two Decimal operands
--> <expression>:1:7
|
1 | 412 + "pages"
| ^^^^^^^
There is no implicit conversion between types: a function that takes a
Decimal does not accept an Int. Arithmetic and comparison take an Int
and a Decimal together, and arithmetic on them gives a Decimal; that is
a rule of those operators, not a conversion. A conversion is written: str
writes a value as text, and float turns an Int into a Float:
412.str()
"412"
hql check prints a program's type without evaluating it:
$ hql check - <<< '412 / 3'
Int
The types, one page each¶
| Type | What it holds | Page |
|---|---|---|
Int |
a whole number, 64 bits | Int |
Float |
a binary floating-point number, written 2.5f |
Float |
Decimal |
an exact decimal number, written 2.50 |
Decimal |
String |
text | String |
Bool |
true or false |
Bool |
Bytes |
a sequence of octets | Bytes |
Date, DateTime, Instant, Duration |
a date, a wall-clock reading, a point in time, a length of time | Date and time |
Data |
a tree of records, lists and scalars | Data |
Option<T> |
a T, or nothing |
Option |
Braces write three different things — a set, a Data record and, where one
is declared, a map — and sets, Data and maps tells
them apart. Lists, sets and maps as collections, and the steps they share, are
chapter 3.