iyi

Blog0.22.0

iyi 0.22.0: closing what a peer could reach

iyi 0.22.0 keys a string's hash per process, holds std/http to its caller's limits, answers a refused file as a value and runs tests at once.

iyi 0.22.0 is out. 0.21.0 made tasks run in parallel; this release is about what a program faces once it is serving: a peer that sends keys built to collide, a request head past the server's limit, a disk that fills, a dependency that changes under a build. It also makes iyi test a runner a project can lean on. Here is the release in its own words:

A string's hash is keyed afresh in every process, std/http holds a request to the limits its caller sets, and the runtime 0.21.0 made parallel holds under load. String#hash is SipHash-1-3 keyed from the system's entropy once a process, so keys built offline to collide no longer make a Hash or JSON.parse quadratic (#212). Server.serve and Server.handle take max_head and max_body, and a head past the limit whose end came in the same read is refused (#213). std/tls, std/x509, std/ocsp, std/cookie, std/password, std/otp, std/rsa and std/big close what a peer could reach, and an iyi.sum hash is SHA-256 and says where each file ends. File.open?, File.read?, File.write? and a FileIO's write?, flush? and close? answer a refusal as an IOError (#214), and compile-time macros in .iyi files run no shell command. In the runtime a pool short of workers fills, a finished thread leaves nothing on the scheduler's registry, threads that panic at once print whole lines, and a size class keeps warm what it took the epoch before, so #145's app holds 127 MB at steady load on this Linux machine against 0.21.0's 170, and 134 against 179 on macOS. A build that runs a command from a macro is no longer taken for up to date, the compiler's cache stays bounded, a regex literal a macro writes and a lib global read by a generic that travels compile (#215, #204), and iyi test runs tests at once. heap22, a heap per pool unit that each unit collects on its own thread, ships as an experiment behind -Dheap_partitions, kept or dropped on its measurements.

CHANGELOG.md at v0.22.0, lines 5 to 26

A hash a client cannot aim#

String#hash was FNV-1a from a fixed start, and keys that all share one hash are cheap to build offline. Anywhere a program filed client-chosen strings in a table, a request body's JSON or std/http's headers among them, such keys made a Hash or JSON.parse quadratic. It is SipHash-1-3 now, the hash Rust's and Python's tables use, keyed once a process from the system's entropy before the program's first line runs.

A table's order is still the order things went in, so nothing a program prints moves. What does move is the hash itself: it differs from one run to the next, so a program must not keep or print one as if it stayed.

Limits the server holds#

Server.serve and Server.handle take max_head and max_body. The defaults are Server::MAX_HEAD and Server::MAX_BODY, as before, and they are now held to while the request is read: a head past max_head is a 431 and a body past max_body is a 413, over HTTP/1.1 and HTTP/2, on the calling thread and on the pool, streamed or not. A server with smaller limits of its own, as iyi-web is, used to be able to check them only after std/http had read up to its own ceiling. A head past the limit whose end arrived in the same read as the bytes that passed it used to be served; it is refused now wherever its end lies.

A refused file is a value#

iyi has no rescue, and File.open, a FileIO's write and the flush in close panicked when the system said no. A server could not answer 507 or 500 and carry on. std/file now has File.open?, File.read?, File.write?, and a FileIO's write?, flush? and close?, in std's ? convention. Each answers an IOError carrying the operation, the path and the platform's error number, with reason in words and not_found?, denied? and full? to branch on. The panicking forms are unchanged.

Closing what a peer could reach#

Compile-time macros in .iyi files no longer run shell commands: system and backticks are refused at their line, read_file stays inside its own package and the standard library, and only the root package reads the build environment with env. A run helper has to live in the package that defines the macro, though it still runs without a sandbox.

The standard library's network and crypto modules got the same scrutiny. std/tls, std/x509, std/ocsp, std/cookie, std/password, std/otp, std/rsa and std/big close what a peer could reach. A signature that fails its own check is now an error the caller answers beside the signature, where it used to end the process.

Packages are pinned harder too. An iyi.sum hash is SHA-256 and says where each file ends, and iyi.sum now records every manifest the version selection reads, outbid versions included, so a requirement cannot slip in through a manifest nobody checked. install.sh refuses a tarball when the release's SHA256SUMS cannot be fetched and fetches over HTTPS only, and a release now attests where it was built, which gh attestation verify reads.

Tests at once#

iyi test runs half as many tests at once as the machine has cores, -p N says how many, and the report stays in file order. test "name" do ... end is a named case: --json reports each one's verdict and its panic's message, --run TEXT runs the cases whose name holds the text (and fails a run in which none does, rather than passing having tested nothing), and skip: "why" leaves one out and says why. --release and -D FLAG build every test that way, and -v shows what a passing test printed.

In a directory, iyi test no longer walks into lib, a dot directory or another module's directory, so it runs your tests and not your dependencies'. A test killed at its deadline takes the processes it started with it, and its output comes back in the order it wrote it.

Underneath#

The runtime that 0.21.0 made parallel holds up under load. A pool short of workers fills instead of letting its tasks take turns on too few, a finished thread leaves nothing behind on the scheduler's registry, and threads that panic at once print whole lines. Each size class keeps warm only what it took in the last epoch, and an idle pool hands its warm pages back, which is where the steady memory figures above come from.

The build is steadier as well. A build that runs a command from a macro is no longer taken for up to date by the next, a build no longer links objects another build's cache rotation has just deleted, and the compiler's cache stays bounded. iyi lsp no longer recompiles a large module for every question. A regex literal that a macro writes compiles (#215), and so does a lib global read by a generic body that travels across a module boundary (#204). iyi doc --json prints a package's exported API as data.

heap22, a heap per pool unit that each unit collects on its own thread, ships as an experiment behind -Dheap_partitions. A build without the flag is today's, apart from two std/tls accessors that now answer copies, and whether heap22 stays is decided by its measurements.

What you have to change#

What a user must change. A macro in a .iyi file that called system or backticks is refused at its line, read_file reads only within its package and std, and only the root package reads env. The signing calls of std/rsa, std/tls, std/x509 and std/acme answer an error beside the signature, Bcrypt.hash panics for a password with a NUL or past 72 bytes, std/otp refuses a secret under 16 bytes unless min_secret: says less, and Cookie#to_set_cookie panics for a path, domain or SameSite a cookie cannot carry. iyi.sum lines are h2:, which 0.21.0 does not read, so a team that shares one upgrades together, and iyi.sum now records the manifests MVS reads: run iyi mod tidy and commit what it writes before iyi mod tidy --check passes again. A string's hash differs from one run to the next, so nothing may keep or print one as if it stayed. The Changed entry "What the fixes below change in a program's source" has the details.

CHANGELOG.md at v0.22.0, lines 28 to 40

In practice: move any macro that shelled out to a step outside the build, run iyi mod tidy and commit the iyi.sum it writes, and upgrade together if your team shares one, since 0.21.0 does not read the new h2: lines. Where a program signs with std/rsa, std/tls, std/x509 or std/acme, handle the error the call now answers. Anything that saved or compared a string's hash across runs has to stop.

Upgrade#

The installer fetches the latest release:

sh
curl -fsSL https://raw.githubusercontent.com/iyilang/iyi/master/install.sh | sh
~/.local/bin/iyi run ~/.local/share/iyi/samples/hello.iyi
PowerShell
irm https://raw.githubusercontent.com/iyilang/iyi/master/install.ps1 | iex
& "$env:LOCALAPPDATA\Programs\iyi\bin\iyi.exe" run "$env:LOCALAPPDATA\Programs\iyi\share\iyi\samples\hello.iyi"