Skip to content

std::supervisors

Supervisors: long-lived tasks that outlive the code that started them, restarted by policy when they stop. Supervisor.new makes one, and only during boot; Supervisor.spawn and Supervisor.spawn_all start work under it with a Restart disposition and a Backoff schedule, and GiveUp says what happens when the schedule runs out. Supervisor.flush waits for its tasks to finish. Where supervisors fit in a program’s life and how they shut down are language rules (Concurrency).

Import with import std::supervisors.

enum Restart
    Temporary
    Transient
    Permanent
end

Which endings make a supervisor run a task again.

Restart says whether a stopped task runs again. Backoff says when, and how many times. The three values match the BEAM’s :temporary, :transient and :permanent.

A restart calls the function the spawn was given again, so whatever state that function sets up is built again.

A cancelled task is never restarted, under any value. Shutdown cancels tasks, and a restart then would keep the program from exiting.

Variants

  • Temporary

    Never restarted. This is the default, since most work spawned under a supervisor is meant to finish.

    A failure is still reported on standard error.

  • Transient

    Restarted if it fails, which is a trap in its body. A task that returned is done.

    For work with an end, such as sending an email or writing an audit row: it is retried when it breaks and left alone once it finishes.

  • Permanent

    Restarted whenever it stops, whether it failed or returned.

    For work with no end, such as a worker draining a queue or a server reading an inbox. Such a task returning on its own is a bug: its loop took a branch it should not have, or the channel it read was closed. Under Permanent that is reported and the task restarted. Under the other two values the worker would stop with no report.

impl Debug for Restart

fn inspect(value: Restart) -> String

impl Debug.inspect

enum Backoff
    Exponential { max_restarts: Int = 10, max_elapsed: Duration = Duration.minutes(15) }
end

When a supervisor tries a failed task again, and how many times before it gives up.

Variants

  • Exponential

    Exponential delays with jitter. The first delay is one second, each later one is double the last, and none is longer than a minute. The supervisor gives up after max_restarts restarts or after max_elapsed spent trying, whichever comes first.

    Both fields have defaults, so .Exponential{} is the schedule a supervisor has when none is named. Naming one field keeps the other’s default: .Exponential{max_restarts: 3} still gives up after fifteen minutes.

    max_restarts counts restarts, not runs, so max_restarts: 3 runs the body at most four times.

    Jitter is always applied. Without it, a supervisor restarting fifty identical tasks would retry them all at the same instant.

    A run longer than a minute, the longest delay, counts as recovered: the next failure starts again from the first delay with the whole count and the whole time. So a worker that fails once a month never uses up max_restarts.

impl Debug for Backoff

fn inspect(value: Backoff) -> String

impl Debug.inspect

enum GiveUp
    Report
    Exit
end

What a supervisor does when it decides a task will not run again: under Restart.Temporary at its first failure, and under Backoff.Exponential once max_restarts or max_elapsed runs out.

Neither choice restarts the program. Exit ends the process, and whatever runs the process decides whether it starts again.

Variants

  • Report

    The default: report the failure on standard error and carry on. The supervisor’s other tasks keep running.

  • Exit

    End the process: every supervisor is drained, then the program exits with a non-zero status.

    Nomi does not start the program again. A service manager or an orchestrator does, and the new process runs boot again, so it starts with new connections, new state and new supervisors.

    It is a poor fit for a long-running process with nothing above it to restart it, such as a desktop application or an embedded runtime. There, use Report and your own monitoring.

impl Debug for GiveUp

fn inspect(value: GiveUp) -> String

impl Debug.inspect

enum Wait
    Forever
    UpTo Duration
end

How long Supervisor.flush may block.

Variants

  • Forever

    Block until the supervisor’s work finishes, however long that takes.

  • UpTo

    Block until the work finishes or this much time passes, whichever comes first. It returns as soon as the work is done. A bound past about 292 years, the longest wait the clock counts, waits that long, as Forever does.

impl Debug for Wait

fn inspect(value: Wait) -> String

impl Debug.inspect

enum FlushOutcome
    Flushed
    TimedOut
end

How a Supervisor.flush ended.

A deadline on the Context in force stops the work that runs past it. A bound given to flush stops nothing and returns TimedOut, so the caller decides what a slow flush means.

Variants

  • Flushed

    Every task outstanding when flush was called has finished.

  • TimedOut

    The bound ran out with work still outstanding. Nothing was cancelled: the work carries on, and shutdown still bounds it.

impl Debug for FlushOutcome

fn inspect(value: FlushOutcome) -> String

impl Debug.inspect

type Supervisor

An owner for tasks that outlive the call that started them.

A concurrent block does not produce its value until every task in it has finished, which does not suit a signup handler that responds now and sends mail later. A function that spawns under a Supervisor returns while the work keeps running, and the supervisor decides how long shutdown waits for it. Every task has one of these two owners.

A supervisor is a set of tasks that share a policy. Its tasks share one max_running limit and wait on each other for a slot, Supervisor.flush waits for all of them, and shutdown drains all of them within one shutdown_timeout. Tasks under different supervisors do not affect each other. Put work under one supervisor when it should be treated the same way at shutdown and on failure, not because it serves the same feature.

Supervisors are fields of the application struct and are created in boot, one per policy the program needs:

struct App
    context: Context
    audit:   Supervisor
    notify:  Supervisor
end

fn boot() -> Result<App, String>
    Ok(App{
        context: Context.root(),
        audit:   Supervisor.new(shutdown_timeout: Duration.seconds(30), max_running: 4),
        notify:  Supervisor.new(shutdown_timeout: Duration.milliseconds(200), max_running: 50),
    })
end

Supervised work lives in the process and is not durable: a crash after the response is sent and before the email is sent loses the email. That is acceptable for a welcome message and not for a receipt.

fn new(max_running: PositiveInt, shutdown_timeout: Duration = Duration.seconds(5), restart: Restart = .Temporary, backoff: Backoff = .Exponential{}, on_give_up: GiveUp = .Report) -> Supervisor

type function on Supervisor

Creates a supervisor.

max_running is how many of its tasks run at once, and it has no default. The limit protects whatever the tasks call, such as a connection pool or a rate-limited service, and only the program knows that number. Work reaches a supervisor from call sites all over the program, so no limit would mean no limit for the whole run. A supervisor that holds one long-lived worker has max_running: 1.

shutdown_timeout is how long shutdown lets this supervisor’s work run before cancelling it. Work that finishes within it is never cancelled. With a zero or negative timeout, shutdown cancels the work at once. Supervisors drain at the same time, each within its own timeout, so a 30-second one and a 200-millisecond one do not wait for each other. When the timeout runs out, the runtime cancels what is left, waits up to one second for it to unwind and run its deferred calls, and then abandons it, so a task that never reaches a safe point cannot hang shutdown. Boot’s deferred calls run after the drain, so they close what the work used only once the work has stopped. A timeout past about 292 years, the longest wait the clock counts, lets the work run until it finishes.

restart, backoff and on_give_up say what happens when a task stops: see Restart, Backoff and GiveUp.

A supervisor can be created only during boot: in boot itself or in a constructor boot calls, which is how a server type owns its supervisor and its restart policy. A call to Supervisor.new that boot does not reach is a compile error. Every supervisor lives as long as the program and has its own max_running, so one created per request would give each request its own limit: max_running: 8 under a thousand requests at once would be eight thousand tasks.

fn spawn(supervisor: Supervisor, body: () -> Unit) -> Task<Unit>

type function on Supervisor

Starts body as a task under supervisor and returns at once.

The call never blocks. When max_running tasks are already running, the new task waits for a slot on its own.

It returns a Task, as Task.spawn does, and this one need not be awaited, because the supervisor drains it at shutdown. Use the handle to cancel the task or to read its Task.outcome, or discard it with _ =.

The body must return Unit. Nobody has to await supervised work, so a returned value would have nowhere to go, and a returned Err would be dropped without a report. Wrap the call in a function that handles the failure and reports it.

fn spawn_all<T>(supervisor: Supervisor, source: Iter<T>, f: (T) -> Unit) -> List<Task<Unit>>

type function on Supervisor

Starts one task per item of source under supervisor, each running f on its item.

It takes no max_running: the supervisor’s limit applies to these tasks and to every other task spawned under it.

f returns Unit, as all supervised work does, so there are no values to collect. Use the handles to cancel tasks, or pass them to Task.await_all to wait for this batch and not for everything under the supervisor. It consumes the whole source, so a source that is Iter.is_infinite traps before any task is spawned.

fn flush(supervisor: Supervisor, bound: Wait = Wait.Forever) -> FlushOutcome

type function on Supervisor

Blocks until every task outstanding under supervisor has finished.

It cancels nothing. Shutdown cancels what is left when shutdown_timeout runs out; flush waits and reports. Call it where queued work must have gone out before the program reports success, or in a test, so the background work a call started finishes before the test asserts on it.

A task is outstanding if it was spawned before the call: spawn adds its task before it returns, so work started on an earlier line is always included, and work a flushed task spawns is waited for too. Work spawned after the call is not waited for.

With no bound it waits as long as the work takes. Pass Wait.UpTo(d) to limit the wait; when the time runs out it returns FlushOutcome.TimedOut and the work carries on:

case Supervisor.flush(App.notify, Wait.UpTo(Duration.seconds(5)))
    FlushOutcome.Flushed  -> report_sent()
    FlushOutcome.TimedOut -> carry_on()
end

impl Debug for Supervisor

fn inspect(value: Supervisor) -> String

impl Debug.inspect