Skip to content

17. Tests

Taught in Testing.

test "name", its body below and end, declares one test. Tests may be declared in any .nomi file. They are checked with the program but run only under nomi test; an ordinary run ignores them. How nomi test finds, selects, times and reports tests is in the tooling reference.

test "new user"
    user = users.new("Alice", 30)
    assert user.name == "Alice"
end
  • A test is declared at the top level of a file or in a group. One inside a body (a function’s, a lambda’s, a test’s or a once initializer’s) is the error a `test` is declared at the top level of a file or in a `group`; it cannot be nested inside a body.
  • A test must be able to fail, so its body holds at least one assert, refute or subtests statement. One without is the error test block must contain at least one assert or refute.
  • A test’s name is not empty or only whitespace: a test name must not be empty; name the test for what it checks, such as `test "parses a date"`. Names are unique within a file by their full path, group included. nomi check accepts a duplicate; nomi test reports it as a failed case, duplicate test name "x" at line 5; first declared at line 1.
  • return in a test body ends the test: its pending deferred calls run and it passes. return value makes value the body’s final value (Assertions below).
  • A with App.field = value line in a test body replaces an application field for the rest of the body (§15 Scoped application fields).

Taught in Testing.

group "name", its lines and tests below and end, holds tests that share a clock, a boot or a setup. It holds at most one clock line, at most one boot line, at most one setup, and its test blocks, or a subtests block that holds them all and runs them once per row (§17 Subtests in a group). Each part is optional and may stand anywhere among the tests; they run in a fixed order whatever their position.

group "defaults"
    setup {name: "No Name"}

    test "default user", {name:}
        user = users.default()
        assert user.name == name
    end
end
ViolationError
A group inside a bodya `group` is declared at the top level of a file; it cannot be nested inside a body
A group inside a groupa `group` cannot contain another `group`; write a sibling group instead
A group with no namea `group` needs a name; name the group for what its tests share, such as `group "dates"`
An empty namea `group` name must not be empty; name the group for what its tests share, such as `group "dates"`
A second clock, boot or setupa `group` has at most one `setup` line (and the same for clock and boot)
fn boot in a groupa `group` does not define `fn boot`; it names an entry's boot on a `boot` line, such as `boot server.boot(startup)`
Anything else in a group bodya group body may contain only clock, boot, setup, subtests and test declarations
A boot line that calls anything but an entry’s boot`boot` in a `group` calls an entry's boot, the `fn boot` of a file that defines `fn main`, such as `boot server.boot(startup)`; `helpers.make_app` is not one

group is a keyword only at the top level of a file, before a name string or alone on its line. Anywhere else it is an ordinary name, so group = Iter.group_by(...) works.

Each test starts fresh. For each test of a group: its clock is selected, the boot line’s call runs and publishes the application value, setup runs, then the test’s body. Nothing carries from one test to the next. A defer in the boot or the setup runs when that test finishes, after its body and its work.

clock. clock Clock.Virtual or clock Clock.System (the default), with Clock from std::testing. Under a virtual clock, time moves only when every task in the test is blocked, so a test that sleeps or waits on a deadline costs no time and takes the same path every run. Work that can never finish is reported as a deadlock, and so is a wait on something outside the program (reading input, a network call), since that never lets a virtual clock move.

boot. The boot line calls an entry’s boot (§15 Scoped application fields) as an ordinary call checked against the boot’s signature: boot server.boot() calls a boot that takes no parameter or whose parameter has a default, and boot server.boot(startup) passes a Startup. The test file imports the entry file. A test outside a group, or in a group with no boot line, runs with no boot and cannot read application fields. A boot that returns Err fails each test it serves before the body runs, with the report of an early ?: the boot line, and the Err it returned.

import
    server
    server.App
end

group "configured sender"
    boot server.boot(Startup{env: {"SENDER" => "ada"}})

    test "reads the booted sender"
        assert App.sender == "ada"
    end
end

Calling boot in a test. A test may also call an entry’s boot as an ordinary function, to check what it returns: the Err that stops the program, or the fields of the app in Ok. The call may stand in a test body, a group’s setup or an attached test, lambdas inside them included; anywhere else, and as a function value anywhere, it is the error in §15 Scoped application fields. The call runs boot as the program does, so boot may create supervisors, but it publishes nothing: the test reads the app’s fields from the value it got, and App.field still reads only what the group’s boot line published. What the boot starts belongs to the test that called it: its supervised work is stopped and the calls it registered with defer run when that test finishes, after its body, whether boot returned Ok or Err.

import server

test "postgres needs a database URL"
    startup = Startup{env: {"STORE" => "postgres"}}
    assert server.boot(startup) is Err("STORE=postgres needs DATABASE_URL")
end

setup. setup expr, or setup begin with statements before its final value and end, produces the value the group’s tests receive. A setup whose last line is a statement, such as a with line, produces Unit; one whose last line is assert x produces what ok = assert x binds. return value in a setup ends the setup with that value, and the setup’s type is the type its final value and every return agree on. A with line in setup stays in force for the body of the test it serves, so setup is where a group installs fakes. A ? or a failed assertion in setup fails the test it serves with the same report it would give in the body, and the body does not run.

A test binds the setup’s value with a pattern after its name: a name (test "loads rows", db), a tuple ((left, right)) or a record ({name:}). A test with no pattern ignores the value. The pattern must match every value of the setup’s type: a test's pattern must match every setup value; bind it with a name or an irrefutable pattern and match it in the body. Bindings inside setup do not reach the tests.

group "store"
    setup begin
        db = sqlite.open("tasks.tsv")?
        defer sqlite.close(db)
        db
    end

    test "loads rows", db
        assert Store.count(db) == 0
    end
end

A test’s standard input and output are captured with io.capture and io.replay (std::io), and its files with std::fs::testing.

Taught in Testing.

An attached test is a run of //! lines immediately before a declaration. It runs under nomi test like a test, and its lines are checked in the scope of the declaration that follows. Its lines hold what a test body holds, subtests included, and it too needs an assertion: attached test must contain at least one assert or refute.

attached-test      = attached-test-line+
attached-test-line = "//!" [statement]

Contiguous //! lines are one attached test; a blank line starts another. A //! starts its line, after nothing but indentation: one after code is the error a `//!` attached test must start its own line, above the declaration it tests; write `//` for a comment after code.

//! assert answer() == 42
//
fn answer() -> Int
    42
end

Taught in Testing.

assert expr and refute expr are language forms, not functions. A failure leaves the nearest enclosing test, or Result-returning function, with Err(AssertionFailure); in main, declared -> Result<Unit, AssertionFailure>, it ends the program with the report. In a function with another result, an assertion is the error `assert` can return AssertionFailure, but the enclosing function returns Int. An assertion in a lambda is an error, since it would only be the lambda’s value.

What each subject requires, and what the assertion’s value is when it holds:

Subjectassert requiresrefute requiresValue
BoolTrueFalsethe Bool
Result<T, E>Ok(_)Err(_)the subject
Maybe<T>Some(_)Nonethe subject
an AssertableAssertable.failure returns Noneit returns Some(details)the subject

assert pattern = value asserts that the value matches the pattern and binds the pattern’s names. A mismatch is an assertion failure.

assert [left, right] = pair

assert Err(Error.InvalidFormat(reason)) = Date.parse("not-a-date")

Where an assertion stands. An assertion is a statement, or a binding’s value (ok = assert x). Its value is the same wherever it stands: ok = begin above assert x and end, an if or case arm ending in assert x whose value is bound, ok: Bool = assert x and Some(n) = assert m else return all bind what ok = assert x binds, and _ = assert x checks x and discards it. A case arm’s body may be an assertion. It is not an operand: not assert x is the error `assert` cannot be an operand of `not`, whose help suggests refute x or assert not x, and so is an assertion beside any binary operator but |>. Anywhere else it is the error `assert` cannot be used as a value here: an assertion stands as its own statement or as a binding's value: as another assertion’s subject, an argument, a collection element, inside parentheses, a condition or scrutinee, a returned value, or the head of a pipe. To assert a pipeline, put the keyword before the whole of it:

assert "Ada Lovelace"
    |> String.contains("Ada")

testing.check. testing.check(expr), from std::testing, takes the subjects assert takes and returns Result<Subject, AssertionFailure> instead of leaving: Ok of the assertion’s value, or Err of its failure.

A final Err(AssertionFailure) fails the test. A test body or a setup whose final value is Err(AssertionFailure) fails with that failure’s report. The rule reaches through a final if, case or block to the arm that ran, and applies to return value. A body whose final statement is itself an assertion is judged by the assertion. The body still needs an assertion of its own (Tests above).

import std::testing

test "total"
    total = 10
    assert total > 0
    testing.check(total == 10)
end

When a ? exits a test early, the report shows the ?’s source and the value it returned.

A failed assertion’s report shows the assertion and, under values:, each operand it compared with its value. A value reads structurally: a String is quoted, a struct’s fields are sorted by name, and a type’s impl Display is not used. A type with a hand-written impl Debug reads as that impl renders it, at any depth, as dbg prints it; a derived or universal Debug changes nothing. A literal operand has no row, nor does a function named by its own name. A name bound to a function keeps its row (g = <func: double>).

Line diffs. When assert a == b fails and both operands are Strings, at least one with a line break, the report shows a line diff in place of their two rows, the left operand as the actual value and the right as the expected. The tooling reference shows its layout.

Taught in Testing.

subtests rows, pattern, its body below and end, runs the body once per row of a table, in order, with pattern bound to the row. rows is anything Iter accepts: a List, Vector, Set, Range or Iter, or a Map, whose rows are its (key, value) pairs.

test "determines palindromes"
    cases = [("abba", True), ("abc", False), ("x", True)]

    subtests cases, (line, known_palindrome)
        assert is_palindrome(line) == known_palindrome
    end
end

The rows may also be piped into the statement: rows |> subtests pattern, name is subtests rows, pattern, name, each with the same body and end. The |> subtests stage has no value, so it must be the last stage of a pipeline that stands as a statement of its own. A stage after it is the error `|> subtests` must be the last stage of its pipeline: it has no value to pass on, and a pipeline used as a value is a `|> subtests` stage has no value, so its pipeline must be a statement of its own.

test "doubles"
    [(1, 2), (2, 4), (3, 6)]
    |> subtests (n, doubled)
        assert n * 2 == doubled
    end
end

The rules:

  • The pattern must match every row: a `subtests` pattern must match every row; bind it with a name or an irrefutable pattern and match it in the body. Rows that are not iterable are `subtests` runs its body once per element of what it is given, which must be something `Iter` accepts, such as a List, a Range or an Iter; got Int.
  • The body must be able to fail a row: it holds an assert or refute of its own (outside a lambda or a nested fn), or its final value, in every arm of a final if or case, is a Result<_, AssertionFailure>. Otherwise it is the error `subtests` body must contain an assert or refute, or end each row with a Result<_, AssertionFailure>. A row whose final value is Err(AssertionFailure) fails, so a row may end with a helper that asserts.
  • A row is not a function, and every row runs to its end, so return, break, continue and ? directly in the body are errors, such as `return` cannot be used in a `subtests` body: a row is not a function, and each row runs to its end and `?` cannot be used in a `subtests` body: a row is not a function, so there is nothing to return the Err or None to. Inside a lambda or a nested fn in the body they leave that, as anywhere.
  • A failed assertion ends its row only, and the next row runs. A trap in a row is not caught: it fails the test, and the rows after it do not run.
  • subtests is a statement whose value is Unit, and it fails as an assertion does. In a function that returns Result<_, AssertionFailure>, a failure returns Err(failure), whose keyword is "subtests", whose reason is the count line (2 of 3 subtests failed), and whose subtests field lists each failed row as a SubtestFailure with its row and its failure. Where an assertion cannot stand, neither can subtests: in a function with another result (`subtests` can return AssertionFailure, but the enclosing function returns Int), in a lambda, in a once initializer (`subtests` cannot be used in the initializer of `once v`: a `once` has no function to return from) or in a default. It cannot be written in a setup (`subtests` cannot be used in a `setup`: a setup builds the value each test binds) or inside another subtests body (`subtests` cannot be written inside another `subtests` body: each row of the outer statement is one case).
  • subtests is a keyword only where a statement starts with it, an expression starts on its line, and a comma follows that expression, or after |>. Anywhere else it is an ordinary name: subtests = rows and a function named subtests work.

Row names. subtests rows, pattern, name names each row. name is an expression of type String, written after the pattern and a comma, and evaluated once per row after the pattern binds and before the body runs, in the scope of the pattern’s bindings. A binding only the name reads counts as used. The name ends with the header’s line, and a struct literal may stand in it (Point{x:, y:} |> Point.describe()). A _ or _-prefixed name in the pattern discards a column either way.

  • A name of another type is the name of a `subtests` row must be a String, got Int.
  • The name is evaluated on its own, so return, break, continue, ? and an assertion directly in it are errors, such as `?` cannot be used in the name of a `subtests` row: the name is evaluated on its own, before the row runs.
  • A name that traps fails its row, and the row’s body does not run (below).
test "leap years"
    cases = [
        ("year divisible by 400 is leap year",                         2000, True),
        ("year divisible by 100, not divisible by 400 in common year", 1900, False),
    ]

    subtests cases, (description, year, expected), description
        assert is_leap_year(year) == expected
    end
end

Reports. After the last row, when any row failed, the statement fails with a report that lists each failed row: the row rendered as a values: row renders a value, or its name as a String literal, then that row’s assertion report, then the count. Passing rows are not listed. Two rows may share a name.

FAIL palindrome_test.nomi :: determines palindromes
  subtest ("abc", True): line 5: assertion failed
    assert is_palindrome(line) == known_palindrome
    values:
      line
        = "abc"
      is_palindrome(line)
        = False
      known_palindrome
        = True
  1 of 3 subtests failed

A row whose name trapped is listed under the rendered row with its name trapped: and the trap’s text (without its own line N: when the trap is on the name’s line), then the name as written:

  subtest 0: line 12: its name trapped: `NonZeroInt.try_from(n)` found nothing
    "${10 / NonZeroInt.try_from(n)!}"

Cases. Each row a test runs is a case in the run’s totals. A test that runs subtests counts as many cases as rows it ran, summed over its statements, and a test with none counts one. A failed row is a failed case and a passing row a passed one, so 9 rows of which 1 failed add 8 passed, 1 failed. A failure outside every row (an assertion after the rows, or a trap in a row) is one more failed case. A passing test prints one ok line with its count, ok palindrome_test.nomi :: determines palindromes (3 cases).

Taught in Testing.

subtests rows, pattern, written in a group with its items below and end, holds the group’s clock, boot and setup lines and its tests, and runs all of them once per row, with pattern bound to the row in the boot line, the setup and every test. Each row’s tests are cases of their own, and each starts fresh as any test of a group does: for each row and each test, the clock is selected, the boot line runs with that row, then the setup, then the test’s body. The boot is the entry’s fn boot, run once for each case.

import
    server
    server.App
end

group "todo api"
    subtests ["memory", "sqlite"], store
        boot server.boot(Startup{env: {"STORE" => store}})

        test "boots the row's store"
            assert App.store == store
        end
    end
end

A case is named with its row after the group’s name, in brackets: todo api [memory] / boots the row's store, then todo api [sqlite] / boots the row's store, rows in order and, within a row, the tests in order. A row that is a String is named by its text. subtests rows, pattern, name names a row by a column: name is a name the pattern binds to a String in every row, as in subtests [("memory", 1), ("sqlite", 4)], (store, workers), store. Any other row is named as a values: row renders it: [2], [("memory", 1)]. A test inside may run a subtests statement of its own, whose rows count as that case’s cases (Cases above), so the group’s cases are its rows times its tests, each counted as its own statements’ rows. nomi test --line at the subtests line, as at the group’s, runs the whole group, and at a test, that test for every row. nomi test --run matches a case by this name, so a pattern naming a row, --run '\[sqlite\]', runs that row alone.

The rows are read when the tests are collected, before anything runs, because each row’s cases are named and counted then. So:

  • The rows are a list literal: the rows of a `subtests` block in a `group` are a list literal, such as `["memory", "sqlite"]`: each row's tests are named and counted when the tests are collected, before anything runs. A test that needs computed rows runs a subtests statement in its body.
  • Each row is a literal value, a String without interpolation, an Int or Float with or without a -, True or False, or a tuple of them: a row of a `subtests` block in a `group` is a literal value, read when the tests are collected: a String, a number, True or False, or a tuple of them. A ? in a row is `?` cannot be used in a row of a `subtests` block in a `group`: the rows are literal values, read when the tests are collected.
  • The pattern must match every row, as a statement’s must: a `subtests` pattern must match every row; bind it with a name or an irrefutable pattern and match it in the body.
  • A name is a String column of the pattern: the name of a row of a `subtests` block in a `group` is a String column its pattern binds, such as `subtests [("memory", 1)], (store, workers), store`, and a column of another type is the name of a `subtests` row must be a String, got Int.
  • The block is the group’s only item, so every item runs once per row and none runs without one: a `group` with a `subtests` block holds nothing else: its clock, boot, setup and tests go inside the block, which runs them once per row.
  • Rows have two dimensions at most: a group’s row, and a statement’s row in one of its tests. A block inside the block is `subtests` in a group cannot hold another `subtests`: each row of the outer one boots the group once; write the rows as tuples instead, as in subtests [("memory", 1), ("sqlite", 2)], (store, workers).
  • In a group, subtests is a keyword where an item starts with it and an expression follows on its line.