Skip to content

2. Data types

HQL is a statically typed expression and query language. Its central idea is that a type is a contract. A type describes a value: the fields of a record, the type of each field, and any rules the values must follow. HQL checks values against their types at two points.

The first is before the program runs. HQL has no compiler, but hql type-checks the whole program first, the way a compiler would. It infers the type of every expression and rejects the program if anything does not fit. This is what makes HQL statically typed.

The second is when data comes in. When a program reads a file, an event or a row as some type, hql validates it against that type and rejects anything that does not match. Values are never converted implicitly. This is what makes HQL strongly typed.

A query language needs both. Static checking finds mistakes in the program, but it cannot see the data, which does not exist yet. Validation finds bad data, but on its own it reports a mistake in the program only when some data happens to trigger it.

Types as data contracts

On a data platform, a data contract is an agreement between the team that produces a table and the teams that consume it. It lists the columns, their types, and rules such as "an amount is never negative". It usually lives outside the code, in a schema registry or a validation job.

In HQL, the contract is the type:

type BookData : DataContract = {
    title: String
    pages: Int

    require pages > 0
}

The checker uses BookData to check the query, and the same declaration validates the rows when the program runs.

Checking the program

Here the program asks for page, but the field is called pages:

type BookData : DataContract = {
    title: String
    pages: Int
}

raw : List<Data> = [{title: "Dune", pages: 412}]
raw | BookData | map(b => b.page / 2)
check error[unknown-field]: BookData has no field `page`
 --> <expression>:7:27
  |
7 | raw | BookData | map(b => b.page / 2)
  |                           ^^^^^^

This is a check error. The checker found it before anything ran, so it does not matter how many books there are.

Validating the data

Data from outside arrives as Data, a tree of records, lists and scalars, like JSON or YAML. Data is a type, but it does not say which fields are there. Reading it as a contract does:

type BookData : DataContract = {
    title: String
    pages: Int

    require pages > 0
}

raw : List<Data> = [{title: "Dune", pages: 412}, {title: "Emma", pages: "many"}]
raw | BookData | map(b => b.pages)
execute error[contract-violation]: this does not read as BookData: field `pages`: "many" is not Int
 --> <expression>:9:1
  |
9 | raw | BookData | map(b => b.pages)
  | ^^^^^^^^^^^^^^
  = expected: BookData

This time the program is fine and the data is wrong, so the error is an execute error. The bad record is rejected as it comes in, before any step uses it. Every value after that point is a valid BookData, as the checker assumed.

Open and closed types

A closed type, declared as a TypeContract, describes the whole record. A record with any other field is rejected:

type Book : TypeContract = {
    title: String
    pages: Int
}

raw : List<Data> = [{title: "Dune", pages: 412, shelf: "sf"}]
raw | Book
execute error[contract-violation]: this does not read as Book: field `shelf`: Book declares no such field
 --> <expression>:7:1
  |
7 | raw | Book
  | ^^^^^^^^^^
  = expected: Book

An open contract, declared as a DataContract, lists only the fields the program needs. Other fields are allowed and kept:

type BookData : DataContract = {
    title: String
    pages: Int
}

raw : List<Data> = [{title: "Dune", pages: 412, shelf: "sf"}]
raw | BookData | map(b => b.pages)
[412]

Use open contracts for data from other systems, which often adds fields you do not need. Use closed types for values your own program builds, where you decide the exact shape. Data contracts covers rules, defaults and validation in detail.

Contracts are built from simpler types: numbers, text, dates and times, and Data. The rest of this chapter covers those, starting in the REPL.

Trying values in the REPL

hql repl reads one input at a time, evaluates it and prints the value. A binding is kept for the inputs after it, and :type asks for a type without evaluating anything:

$ hql repl
hql 0.5.0. `:help` for commands, `:quit` to leave.
hql> pages = 412
()
hql> pages / 3
137
hql> pages % 3
1
hql> :type pages
Int
hql> title = "Dune"
()
hql> title.upper()
"DUNE"
hql> :type title
String
hql> fmt("{} has {} pages", title, pages)
"Dune has 412 pages"
hql> :quit

412 is an Int, so / divides whole numbers and drops the remainder, and % gives the remainder. "Dune" is a String, and title.upper() calls a function of String on it. fmt writes values into text, one {} each.

The same programs run with hql eval, as every program in this book does:

pages = 412
title = "Dune"
fmt("{} has {} pages", title, pages)
"Dune has 412 pages"

Type checking

Each operation says which types it takes. An Int and a String do not add, and the checker rejects the program before it runs:

412 + "pages"
check error[type-mismatch]: addition requires two Int, two Float or two Decimal operands
 --> <expression>:1:7
  |
1 | 412 + "pages"
  |       ^^^^^^^

There is no implicit conversion between types: a function that takes a Decimal does not accept an Int. Arithmetic and comparison take an Int and a Decimal together, and arithmetic on them gives a Decimal; that is a rule of those operators, not a conversion. A conversion is written: str writes a value as text, and float turns an Int into a Float:

412.str()
"412"

hql check prints a program's type without evaluating it:

$ hql check - <<< '412 / 3'
Int

The types, one page each

Type What it holds Page
Int a whole number, 64 bits Int
Float a binary floating-point number, written 2.5f Float
Decimal an exact decimal number, written 2.50 Decimal
String text String
Bool true or false Bool
Bytes a sequence of octets Bytes
Date, DateTime, Instant, Duration a date, a wall-clock reading, a point in time, a length of time Date and time
Data a tree of records, lists and scalars Data
Option<T> a T, or nothing Option

Braces write three different things — a set, a Data record and, where one is declared, a map — and sets, Data and maps tells them apart. Lists, sets and maps as collections, and the steps they share, are chapter 3.