Skip to content

std::storage

Object storage: files kept whole under keys, in a directory, an S3 bucket or memory. Code that stores files calls the Store interface, and each backend implements it, so one function serves the local directory in development, a bucket in production and a test’s std::storage::testing.Memory:

fn save_avatar(user_id: Int, data: Bytes) -> Result<Url, storage.Error>
    key = "avatars/${user_id}.png"
    Store.put(App.store, key, data, "image/png")?
    Store.url(App.store, key)
end

storage.local(root) keeps objects as plain files under a directory, and storage.handler(store) serves any store’s objects through the app. lib/storage/s3 is a backend for any server that speaks the S3 protocol: DigitalOcean Spaces, MinIO, Cloudflare R2, Backblaze B2 and AWS. Boot code picks one:

store = if production then
    s3.store(config)?
else
    storage.local("tmp/files", public_base: Some(Url"/files/"))
end

std::storage::testing has Memory, a store kept in memory for tests, Failing, a store whose every call fails, and conformance, the suite every backend passes.

Import with import std::storage.

Every backend enforces one key grammar (storage.check_key), so a key that works on the local store works on S3:

  • 1 to 1,024 bytes of UTF-8, in segments separated by /, none of them empty;
  • each segment at most 255 bytes, and none starting with .;
  • no control character and no \.

Anything else is allowed: spaces, punctuation such as (, !, %, # and ?, and letters of any script, so a store can address the objects a bucket already holds. ../etc/passwd, /abs and a\b are refused before any backend builds a path or a URL from them.

Store.list returns one page of the keys under a prefix, in order, with each object’s size and time. With a delimiter, keys that go on past it are folded into the common prefix they share, as a directory listing shows a folder once; a page’s next is passed back as continuation for the page after it. storage.list_all follows the pages to the end:

page = Store.list(App.store, ListOptions{prefix: "avatars/", delimiter: "/"})?

everything = storage.list_all(App.store, prefix: "avatars/")?

Objects move whole, as Bytes; there is no streaming or multipart upload.

enum Error
    NotFound { key: String }
    InvalidKey { key: String, reason: String }
    InvalidContentType { content_type: String }
    NoPublicUrl
    Denied { message: String }
    Unavailable { message: String }
    Backend { code: String, message: String }
end

Why a store call failed.

Variants

  • NotFound

    No object has the key. get returns it; info returns Ok(None) instead, and delete returns Ok.

  • InvalidKey

    The key breaks the key rules (check_key). reason says which.

  • InvalidContentType

    The content type is empty, longer than 255 characters, or holds a character outside printable ASCII.

  • NoPublicUrl

    The store was made with no public URL for its objects.

  • Denied

    The backend refused the credentials or the permission.

  • Unavailable

    The backend could not be reached, timed out, or said it is overloaded. The same call may succeed later.

  • Backend

    Anything else, with the backend’s own code: an S3 error code such as NoSuchBucket, or an io.Error variant’s name for the local store.

assert "${Error.NotFound{key: "a.txt"}}" == "no object has the key a.txt"

impl Display for Error

fn to_string(error: Error) -> String

impl Display.to_string

impl Equatable for Error

fn equal(a: Error, b: Error) -> Bool

impl Equatable.equal

impl Debug for Error

fn inspect(value: Error) -> String

impl Debug.inspect

struct Info
    size: Int
    content_type: String
    modified: Instant
end

What a store knows about an object without reading it.

impl Equatable for Info

fn equal(a: Info, b: Info) -> Bool

impl Equatable.equal

impl Debug for Info

fn inspect(value: Info) -> String

impl Debug.inspect

struct Object
    data: Bytes
    info: Info
end

An object read whole: its bytes and its Info.

impl Equatable for Object

fn equal(a: Object, b: Object) -> Bool

impl Equatable.equal

impl Debug for Object

fn inspect(value: Object) -> String

impl Debug.inspect

struct Entry
    key: String
    size: Int
    modified: Instant
end

An object as a listing shows it: its key, its size in bytes, and when it was last stored.

impl Equatable for Entry

fn equal(a: Entry, b: Entry) -> Bool

impl Equatable.equal

impl Debug for Entry

fn inspect(value: Entry) -> String

impl Debug.inspect

struct ListOptions
    prefix: String = ""
    delimiter: String = ""
    continuation: Maybe<String> = None
    limit: Int = 1000
end

Which keys Store.list lists, and which page of them.

impl Equatable for ListOptions

fn equal(a: ListOptions, b: ListOptions) -> Bool

impl Equatable.equal

impl Debug for ListOptions

fn inspect(value: ListOptions) -> String

impl Debug.inspect

struct Listing
    objects: List<Entry> = []
    prefixes: List<String> = []
    next: Maybe<String> = None
end

One page of a listing: the objects and the common prefixes on it, each in key order, and the continuation for the page after it, or None on the last page.

impl Equatable for Listing

fn equal(a: Listing, b: Listing) -> Bool

impl Equatable.equal

impl Debug for Listing

fn inspect(value: Listing) -> String

impl Debug.inspect

interface Store
    fn put(store: self, key: String, data: Bytes, content_type: String) -> Result<Unit, Error>
    fn get(store: self, key: String) -> Result<Object, Error>
    fn info(store: self, key: String) -> Result<Maybe<Info>, Error>
    fn delete(store: self, key: String) -> Result<Unit, Error>
    fn url(store: self, key: String) -> Result<Url, Error>
    fn list(store: self, options: ListOptions) -> Result<Listing, Error>
    fn exists(store: self, key: String) -> Result<Bool, Error>
end

A place objects live, each under a key. storage.local, lib/storage/s3 and std::storage::testing.Memory implement it.

Every backend follows these rules, and std::storage::testing’s conformance suite checks each:

  • Every function checks its key with check_key first, and returns its Err(.InvalidKey) before doing anything else. put then checks the content type with check_content_type.
  • put replaces any object under the key. A reader sees the old object or the new one, never part of one.
  • get and info return the bytes and content type the last put stored, and its size.
  • get of a missing key is Err(.NotFound); info of one is Ok(None); delete of one is Ok.
  • url does no I/O. It returns a URL whether or not the object exists.
  • list returns the keys under its prefix in byte order, its prefix checked with check_prefix first, folds keys that go on past the delimiter into their common prefix, and pages as ListOptions says; page is that rule over a list of entries.
import std::storage::testing.Memory

store: Store = Memory.new()
assert Store.put(store, "notes/a.txt", String.to_bytes("hi"), "text/plain") == Ok(Unit)
assert Store.get(store, "notes/a.txt")!.data == String.to_bytes("hi")
assert Store.exists(store, "notes/b.txt") == Ok(False)
fn put(store: self, key: String, data: Bytes, content_type: String) -> Result<Unit, Error>

interface Store

Stores data under key with content_type, replacing any object there.

fn get(store: self, key: String) -> Result<Object, Error>

interface Store

The object under key, read whole, or Err(.NotFound).

fn info(store: self, key: String) -> Result<Maybe<Info>, Error>

interface Store

What the store knows about the object under key, or None when there is none.

fn delete(store: self, key: String) -> Result<Unit, Error>

interface Store

Removes the object under key. A missing key is Ok.

fn url(store: self, key: String) -> Result<Url, Error>

interface Store

Where a reader fetches the object under key, with no I/O, or Err(.NoPublicUrl) from a store made without one.

fn list(store: self, options: ListOptions) -> Result<Listing, Error>

interface Store

One page of the objects whose keys start with options.prefix.

fn exists(store: self, key: String) -> Result<Bool, Error>

interface Store

Whether an object has the key.

fn check_key(key: String) -> Result<Unit, Error>

Ok when key is a valid key, else Err(.InvalidKey) saying why. A key is 1 to 1,024 bytes of segments separated by /. Each segment is 1 to 255 bytes, does not start with ., and holds no control character (U+0000 to U+001F, U+007F to U+009F) and no \. Any other character is allowed, as S3 allows it.

assert check_key("posts/2024-05-01/a_1.jpg") == Ok(Unit)
assert check_key("posts/Noah_-_3010(1).jpg") == Ok(Unit)
assert check_key("100% café/a b#1?.txt") == Ok(Unit)

assert check_key("../etc/passwd") == Err(Error.InvalidKey{
        key:    "../etc/passwd",
        reason: "segment .. starts with a dot",
    })

assert check_key("a//b") is Err(.InvalidKey{})
assert check_key("a\\b") is Err(.InvalidKey{})
assert check_key("a\nb") is Err(.InvalidKey{})
assert check_key("") is Err(.InvalidKey{})
fn check_prefix(prefix: String) -> Result<Unit, Error>

Ok when prefix can start a key, as list takes it: at most 1,024 bytes, with no control character and no \. It may be empty, end with / or stop inside a segment.

assert check_prefix("") == Ok(Unit)
assert check_prefix("avatars/") == Ok(Unit)
assert check_prefix("a\\b") is Err(.InvalidKey{})
fn page(entries: Iter<Entry>, options: ListOptions) -> Listing

The page of entries that options asks for, as every backend lists: the entries whose keys start with the prefix, in key order, each key that holds the delimiter after the prefix folded into its common prefix, after the continuation, at most limit of them. The continuation is the last key or prefix of the page before, so a backend that holds its keys can list with this alone.

at = Instant.from_seconds(0)
entry = fn key: String -> Entry{key:, size: 1, modified: at}
entries = [entry("p/b/2"), entry("p/a"), entry("p/b/1"), entry("q"), entry("p/c")]

assert page(entries, ListOptions{prefix: "p/"}).objects
|> Iter.map(fn e -> e.key)
|> Iter.to_list() == ["p/a", "p/b/1", "p/b/2", "p/c"]

assert page(entries, ListOptions{prefix: "p/", delimiter: "/", limit: 2}) == Listing{
    objects:  [entry("p/a")],
    prefixes: ["p/b/"],
    next:     Some("p/b/"),
}

assert page(
    entries,
    ListOptions{prefix: "p/", delimiter: "/", limit: 2, continuation: Some("p/b/")},
) == Listing{objects: [entry("p/c")]}
fn list_all(store: Store, prefix: String = "", delimiter: String = "", page_size: Int = 1000) -> Result<Listing, Error>

Every object and common prefix under prefix, following each page’s next to the last page.

import std::storage::testing.Memory

store = Memory.new()
_ = Store.put(store, "a/1", String.to_bytes("x"), "text/plain")
_ = Store.put(store, "a/2/z", String.to_bytes("y"), "text/plain")
all = list_all(store, prefix: "a/", delimiter: "/", page_size: 1)!
assert all.objects |> Iter.map(fn e -> e.key) |> Iter.to_list() == ["a/1"]
assert all.prefixes == ["a/2/"]
assert all.next == None
fn join_key(base: Url, key: String) -> Url

base with the segments of key joined to its path, each escaped, so that a %, ? or # in a key stays part of the path. A backend’s url returns it for its public base. key is not checked.

u = join_key(Url"https://cdn.example.com/files/", "a b/50%(1)#2.jpg")
assert u.path == "/files/a b/50%(1)#2.jpg"
assert "${u}" == "https://cdn.example.com/files/a%20b/50%25%281%29%232.jpg"
fn check_content_type(content_type: String) -> Result<Unit, Error>

Ok when content_type can be stored: 1 to 255 characters of printable ASCII, space to ~. It refuses CR and LF, which could otherwise add a header, but does not parse the media type: image/jpg is stored as written.

assert check_content_type("image/jpeg") == Ok(Unit)
assert check_content_type("text/html; charset=utf-8") == Ok(Unit)
assert check_content_type("") is Err(.InvalidContentType{})
assert check_content_type("text/plain\r\nX-Evil: 1") is Err(.InvalidContentType{})
fn local(root: String, public_base: Maybe<Url> = None) -> Store

A store over the directory root. Each object is a plain file at <root>/<key>, so the directory can be backed up, inspected or served by anything; its content type is kept beside it in <root>/.nomi-storage/types/<key>, which no key can name. root is created on the first put.

A few characters of a key are written on disk as % and two hexadecimal digits, because some file system cannot hold them in a name: % itself, <, >, :, ", |, ? and *; a . or space that ends a segment, which Windows drops; and the first letter of a segment Windows keeps for a device (CON, nul.txt, COM1). So the key a:b.txt is the file a%3Ab.txt, and a file copied in by hand is found under the key its name spells only when its name holds none of these.

public_base is where the app serves the directory, as Some(Url"/files/") with a route to storage.handler; url joins the key to it. With None, url returns Err(.NoPublicUrl).

A put writes the type file, then the object, each to a temporary file renamed over the old one, so a reader sees the old object or the new one. A file someone copied in with no type file is read with the type http.detect_content_type gives its bytes. Errors from the file system map to NotFound, Denied for PermissionDenied, and Backend with the io.Error variant’s name for the rest.

It differs from S3 in three ways:

  • Keys a and a/b cannot both exist, since a cannot be a file and a directory. The second put is Err(.Backend).
  • Two concurrent puts to one key may pair one put’s bytes with the other’s type. Keys that never collide, as ones built from a timestamp or a UUID, are safe.
  • On a case-insensitive file system, as macOS and Windows have by default, keys that differ only in case name one file, and on macOS keys that differ only in Unicode normalization (é as one code point or two) do too.

The calls go through std::fs, so std::fs::testing.run sandboxes the store in tests, and fs.within applies.

import std::fs::testing

ran = testing.run(testing.Sandbox{}, fn ->
    store = local("files", public_base: Some(Url"/files/"))
    Store.put(store, "a/b.txt", String.to_bytes("hi"), "text/plain")?
    Ok((Store.get(store, "a/b.txt")?.info.content_type, "${Store.url(store, "a/b.txt")?}"))
end)

assert ran.value == Ok(("text/plain", "/files/a/b.txt"))
assert testing.text(ran, "/work/files/a/b.txt") == Some("hi")
assert testing.text(ran, "/work/files/.nomi-storage/types/a/b.txt") == Some("text/plain")
fn handler(store: Store, param: String = "key") -> (Request) -> Response

A handler that serves GET and HEAD with the object whose key is the request’s path value param, with its stored content type. It handles Range and conditional requests as Response.content does. A missing object or an invalid key is a 404, Denied a 403, Unavailable a 503, and any other error a 500.

files = storage.handler(App.store)

routes = [Route.to_with("GET /files/{key...}", fn r, _ -> files(r))]

It sets no Cache-Control: an app whose keys never change wraps the handler and adds one. It reads the whole object into memory first, which suits images and documents, not video.

import std::http.{Method, Status}
import std::http::testing
import std::storage::testing.Memory

store = Memory.new()
_ = Store.put(store, "a/b.css", String.to_bytes("p{}"), "text/css")
serve = handler(store)

found = serve(
    Request.with_path_value(testing.request(Method.Get, "/files/a/b.css"), "key", "a/b.css")
)

assert found.status == Status.ok
assert Headers.get(found.headers, "Content-Type") == Some("text/css")
assert Response.text(found) == Ok("p{}")

missing = serve(Request.with_path_value(testing.request(Method.Get, "/files/x"), "key", "x"))
assert missing.status == Status.not_found