iyi

Standard library, 0.18.0

std/compress

DEFLATE (RFC 1951), and the two envelopes it travels in: zlib (RFC 1950) and gzip (RFC 1952). Written in iyi; no zlib is linked, which is what SPEC.md III.10 asks for ("own it. DEFLATE is a known quantity and arrives with HTTP").

import std/compress to use it. Source: src/std/compress.iyi at v0.18.0.

Deflate.compress is a DEFLATE compressor of zlib's shape at zlib's levels: LZ77 by hash chains over the 32 KiB window, greedy at levels 1 to 3 and lazy at 4 to 9 with zlib's own table of how hard each level looks, the symbols gathered into blocks of 16,383, and each block written stored, with the fixed code, or with Huffman codes built for it (held to 15 bits, the code-length code to 7), whichever costs the fewest bits. On the exercise's corpora it writes within 0.3% of Python's zlib at every level, smaller at most, at zlib's speed at level 1 and three quarters of it at 9. Deflate.decompress reads every block type the format has, so anything zlib wrote reads back. Zlib adds the two-byte header and the Adler-32 trailer, Gzip the ten-byte header, the optional fields, and the CRC-32 and length trailer. Each envelope checks what it carries and refuses by name: a header that is not the format's, a preset dictionary, a checksum or a length that does not match, a stream that ends early.

Those take and answer whole strings, and a stream that is not what it says is a panic: the caller handed over bytes it vouched for. A stream that arrives a piece at a time - an HTTP body - is Inflater's: it pulls the compressed bytes from a block as it needs them, answers the decompressed ones as they are asked for, keeps 32 KiB of what it wrote for back-references and nothing more, and answers a bad stream as a CompressError, a value, because bytes from a peer are not a bug in the program reading them. The whole-string calls are that same decoder run over one piece with an output that only grows; there is one decoder. Deflater is the other direction a piece at a time: each piece is blocks the reader can decode at once (zlib's sync flush), the window kept from piece to piece, and finish ends the stream; it can be held to a window smaller than 32 KiB. Inflater.flushed is the reader for such pieces taken one at a time, each decoded whole and the window kept between them, a decoded size past a limit refused as it is reached: what RFC 7692's permessage-deflate (std/websocket) sends.

Left out: zlib's strategies (filtered, Huffman-only, RLE) and a preset dictionary in either direction (zlib's FDICT is refused by name).

Strings are the byte type here, as everywhere in src/std: a compressed stream is a String whose bytes are the stream.

CompressErrorstruct

What Inflater#read answers for a stream it cannot read: one that is corrupt, whose checksum or length does not match, or that ends early. message is the sentence the whole-string calls panic with.

std/compressiyi doc src/std/compress.iyi
pub struct CompressError
  def initialize(message : String)
  def to_s : String
end

Deflatestruct

std/compressiyi doc src/std/compress.iyi
pub struct Deflate
  # *data* as a DEFLATE stream, at *level*: 1 is fastest, 9 smallest, 6
  # (the default) zlib's own balance, 0 stored blocks only; anything else
  # is a panic. The same level writes what zlib's would size for size,
  # within the margins the exercise records.
  def self.compress(data : String, level : Int32 = 6) : String
  def self.decompress(data : String) : String
  # The bytes, and how many of the stream's bytes they took: an envelope
  # reads its trailer from there.
  #
  # *expected* is how long the caller thinks the result is, so that the
  # result is one allocation rather than a guess grown by doubling; -1,
  # the default, is no idea, and the guess is three times the stream's
  # bytes. Held to what the stream could expand to, so a length that lies
  # is not an allocation it asked for.
  def self.inflate(data : String, start : Int32, expected : Int64 = -1_i64) : ::Tuple(String, Int32)
  def initialize
end

Deflaterclass

A compressed stream written a piece at a time. Each compress codes its piece in blocks as Deflate.compress does and ends in zlib's sync flush: an empty stored block, so everything written so far ends on a byte boundary and decodes without waiting for more. The window is kept from piece to piece - the next piece's matches reach back into the last 32 KiB of the ones before it, as zlib's do across a sync flush - so a stream written in pieces costs little more than the same bytes written whole. finish writes the final block, empty, and the envelope's trailer.

std/compressiyi doc src/std/compress.iyi
pub class Deflater
  # The compressed bytes for *data*: the envelope's header on first use,
  # then *data* as complete non-final blocks and the sync flush. "" for
  # empty data after the first call. After `finish`, a panic.
  def compress(data : String) : String
  # The final block and the envelope's trailer. Once only: a second call
  # is a panic.
  def finish : String
  # *level* is `Deflate.compress`'s: 0 stores, 1 is fastest, 9 smallest,
  # 6 the default. *window_bits* is the base-2 logarithm of how far back a
  # match may reach, 8 to 15: 15 is DEFLATE's whole 32 KiB, and a smaller
  # one is what a decoder that keeps less can read - RFC 7692's
  # `max_window_bits`. Anything else for either is a panic.
  def initialize(envelope : Envelope, level : Int32 = 6, window_bits : Int32 = 15)
end

Envelopeenum

What a DEFLATE stream travels in: nothing (RFC 1951), zlib's two-byte header and Adler-32 (RFC 1950), or gzip's header and CRC-32 and length (RFC 1952).

std/compressiyi doc src/std/compress.iyi
pub enum Envelope : Int32
  Raw = 0
  Zlib = 1
  Gzip = 2
end

Gzipstruct

std/compressiyi doc src/std/compress.iyi
pub struct Gzip
  # The ten-byte header with nothing optional set, an mtime of zero, the
  # level's XFL and an unknown OS; then the CRC-32 and the length modulo
  # 2^32, little-endian. *level* as `Deflate.compress`.
  def self.compress(data : String, level : Int32 = 6) : String
  # Every member of the file, one after another; bytes after the last
  # that do not start another are left as zlib leaves them.
  #
  # The last member's trailer names its length, which for a file of one
  # member is the length of the answer, so that is the size the output
  # starts at; a file of several grows past it by doubling. Every member
  # had been sized on its own, by the last trailer or by three times the
  # bytes after it, so each allocated what the rest of the file could
  # hold, and a hundred two-byte members ahead of one of 64 MB took 2.5 s
  # to read where Python's gzip took 0.15.
  def self.decompress(data : String) : String
  def initialize
end

Inflaterclass

A compressed stream read a piece at a time: the compressed bytes come from the block, the decompressed ones go out through read, and memory stays at the 32 KiB window and what is waiting to be read.

body = Inflater.new(Envelope::Gzip) { socket_read }
while (piece = body.read(65536)!) != ""
  ...
end

Inflater.flushed is the same decoder for a raw stream that comes as pieces each ending between blocks - zlib's sync flush, what each Deflater#compress ends with - and is decoded a piece at a time, the window kept from one to the next: RFC 7692's permessage-deflate, whose messages are such pieces with the flush's last four bytes left off.

std/compressiyi doc src/std/compress.iyi
pub class Inflater
  # Whether the stream has ended, its trailer checked, and every byte been
  # read: the next `read` answers "".
  def finished? : Bool
  # A raw stream decoded by `inflate`, a piece at a time.
  def self.flushed : Inflater
  # The next piece of a `flushed` stream, decoded whole: the blocks it
  # holds, which may refer back to the 32 KiB before it, and may be final
  # (RFC 7692 §7.2.1 lets a final block be followed by more). A piece
  # that does not end between blocks, or is not DEFLATE, is a
  # `CompressError`, and so is every piece after it. More than *limit*
  # bytes out is nil, found as they come - a piece that would decode to
  # gigabytes stops 64 KiB past *limit* - and fails the stream the same
  # way. On another `Inflater`, a panic.
  def inflate(piece : String, limit : Int32 = 2147483647) : String | CompressError | Nil
  # *source* answers the next bytes of the compressed stream, nil (or "")
  # when there are no more. It is called only when the decoder has no
  # bytes in hand for what it must read next, so it is never asked for
  # anything past the end of the stream - except that a gzip stream ends
  # only where no member follows, which takes a look at the next two
  # bytes.
  def initialize(envelope : Envelope, &source : (-> String | ::Nil))
  def initialize(core : Inflate)
  # Bytes the source gave beyond the end of the stream (gzip: after the
  # last member), for a caller that frames other data after it; "" before
  # the stream has ended.
  def leftover : String
  # Up to *max* decompressed bytes; "" once the stream and its trailer
  # (CRC-32 and length, or Adler-32) have ended and been checked. A read
  # answers as soon as it has bytes and going on would call the source,
  # so it can answer fewer than *max* before the end; and never more than
  # 64 KiB less a match at once, the room ahead of the window.
  #
  # A corrupt stream, a bad checksum, or input that ends early is a
  # `CompressError`, after which every read answers the same error. What
  # the same read decoded before the error is not handed out. A *max*
  # below 1 is a panic.
  def read(max : Int32 = 65536) : String | CompressError
end

Zlibstruct

std/compressiyi doc src/std/compress.iyi
pub struct Zlib
  # CMF 0x78 (deflate, a 32 KiB window), FLG with the level's band and the
  # check bits that make the pair a multiple of 31 (no dictionary), then
  # the Adler-32 of the bytes, big-endian. *level* as `Deflate.compress`.
  def self.compress(data : String, level : Int32 = 6) : String
  def self.decompress(data : String) : String
  def self.hex(w : UInt32) : String
  def initialize
end

Error for CompressErrorimpl

std/compressiyi doc src/std/compress.iyi
impl Error for CompressError
  def message : String
end