iyi

Standard library, 0.18.0

std/text

Standard text processing library for iyi.

import std/text to use it. Source: src/std/text.iyi at v0.18.0.

Answers the two architectural questions:

1. UTF-8 and byte indexing convention:

 iyi's `String#size` counts UTF-8 code points (decoded in
 `src/iyi/string.iyi`, a byte that begins none counted as one U+FFFD),
 while `bytesize` counts raw bytes.
 Following the prelude's established convention, all string indexing
 (`[]`, `[start, count]`), slice operations, and search offsets
 (`index`, `rindex`, `sub`, `gsub`, `split`) are strictly
 BYTE-INDEXED.
 For example, with `s = "héllo"`, `s.size` is 5, `s.bytesize` is 6,
 `s[1]` is byte 195 (0xC3), `s[2]` is byte 169 (0xA9), and `s.index('l')`
 returns byte offset 3. This guarantees O(1) indexing and byte-slice
 alignment without hidden multi-byte traversals.
 Multi-byte UTF-8 awareness is preserved where character boundaries
 matter: `Char#to_s` encodes multi-byte Unicode code points into 1 to 4
 valid UTF-8 bytes rather than truncating to a single byte; `chars`
 and `each_char` decode full Unicode code points so `"héllo".chars.size`
 equals `"héllo".size` (5); and `reverse` reverses decoded characters
 so `"héllo".reverse` produces `"olléh"` rather than invalid byte sequences.

2. Garbage collector and allocation safety:

 Every string constructed in this library allocates through
 `String.new(bytesize) { |buffer| ... }`. This routes directly to
 `String.memory`, ensuring `IyiHeap.new_object` is called with the
 canonical `String` type ID, headed=true, atomic=true, and that the
 object header words (byte length and scanned character count) are
 properly initialized. No headless buffers, raw pointer arithmetic, or
 out-of-band allocations are used.

3. Regular expressions:

 `sub` and `gsub` here take exact string and character patterns. A
 pattern is `std/regex`'s `Regex`, iyi's own linear-time engine, which
 adds `match`, `=~`, `index`, `sub`, `gsub` and `scan` taking a `Regex`
 to `String`; import both and each call takes the one its argument names.

4. Scope and omissions from Crystal's string.cr (5,922 lines):

 Normalization (NFC, NFD, NFKC, NFKD), case folding, the Unicode
 categories and grapheme clusters are `std/unicode`'s, and regex
 matching is `std/regex`'s. What neither module nor this one carries:
 iconv encoding conversions, C string null-byte guards, and decades of
 legacy Ruby compatibility aliases. This library provides the core operations
 that real programs reach for: whitespace stripping, splitting, joining,
 indexing, substitutions, transformations (tr, squeeze, delete, count,
 center), character inspection, and robust conversions with nilable
 failure paths.

Written against iyi's own prelude, and only that one. Its index, split and sub are byte-indexed by the convention above, which is not Crystal's, and the names it gives ::String and ::Char are names Crystal has — so import std/text in a --crystal program is refused rather than replacing the library's methods under the whole program (SPEC.md III.6).

Array(T)class, reopened

std/textiyi doc src/std/text.iyi
class ::Array(T)
  def join(separator : Char) : String
end

Charstruct, reopened

std/textiyi doc src/std/text.iyi
struct ::Char
  def ascii_alphanumeric? : Bool
  def ascii_letter? : Bool
  def ascii_lowercase? : Bool
  def ascii_number? : Bool
  def ascii_uppercase? : Bool
  def ascii_whitespace? : Bool
  def downcase : Char
  def to_i(base : Int32) : Int32
  def to_i : Int32
  def to_i?(base : Int32) : Int32 | ::Nil
  def to_i? : Int32 | ::Nil
  def upcase : Char
end

Stringclass, reopened

std/textiyi doc src/std/text.iyi
class ::String
  # `s[1..3]`, `s[-3..-1]`: the bytes a range names, its ends counted from
  # the back when negative, as `[](start, count)` counts them.
  def [](range : Range(Int32, Int32)) : String
  # The whitespace `blank?`, `lstrip` and `rstrip` look through is the
  # prelude's `Char#whitespace?`, decoded, so `strip` with no argument agrees
  # with these and with the other library: `"\u3000".blank?` was false.
  def blank? : Bool
  def bytes : Array(UInt8)
  def capitalize : String
  # The width is in characters, because `size` is and because a centred
  # column is a visual claim: padded by bytes, `"héllo".center(9, '.')` came
  # back seven characters wide on one side and eight long in all. The pad
  # travels as its own bytes, so a multi-byte one is a character rather than
  # its first byte three times.
  def center(width : Int32, pad : Char = ' ') : String
  def center(width : Int32, pad : String) : String
  def chars : Array(Char)
  # `"\n"` is every line ending, as `chomp` with no argument reads it and as
  # the other library answers: `"a\r\n".chomp("\n")` is `"a"`.
  def chomp(suffix : String) : String
  def chomp : String
  def chop : String
  def count(char : Char) : Int32
  # The characters in every one of *sets*, a set read as the other
  # library's `Char#in_set?` reads one: `"hello world".count("a-z")` is 10
  # and `"hello".count("^l")` is 3. This counted *sets* as a substring, so
  # `"banana".count("an")` was 2 here and 5 there, and a range or a `^`
  # was text: `count("a-z")` was 0, and `delete` and `squeeze` took the
  # set as a list of the characters written.
  def count(*sets : String) : Int32
  def delete(char : Char) : String
  # The characters in every one of *sets*, as `count` reads them. Characters,
  # not bytes: a 256-entry byte mask cannot hold a character above ASCII,
  # and deleting one byte of one leaves the rest behind.
  def delete(*sets : String) : String
  def each_byte(& : (UInt8 -> Nil)) : Nil
  # Scanned, a line at a time: `lines` first built the whole array - 2.4
  # million strings for a 143 MB text - and `chomp` a second string for each.
  # The same lines, a final newline closing the last rather than opening an
  # empty one, and a `\r` before a newline left off.
  def each_line(& : (String -> Nil)) : Nil
  def ends_with?(char : Char) : Bool
  # An empty search matches before every character and at the end, as
  # `sub` matches it at the start: `"abc".gsub("", "-")` is `"-a-b-c-"`, the
  # other library's answer, where it was `"abc"`. A byte that begins no
  # character is kept, a unit of its own.
  def gsub(search : String, replacement : String) : String
  def gsub(search : Char, replacement : Char) : String
  def gsub(search : Char, replacement : String) : String
  def gsub(search : String, & : (String -> String)) : String
  def gsub(search : Char, & : (Char -> String)) : String
  def index(char : Char, offset : Int32) : Int32 | ::Nil
  def index(search : String, offset : Int32 = 0) : Int32 | ::Nil
  def self.join(items : Array(String), separator : String = "") : String
  # Into one builder, as `Enumerable#join` is and for its reason.
  def self.join(items, separator : String = "") : String
  # A `Char` above ASCII is more than one byte, and this family narrowed it
  # to one with a checked convert: `"a☃b".index('☃', 0)` panicked with
  # "arithmetic overflow", and so did nine of its siblings. Each asks the
  # character's own bytes now, which is what the string holds.
  def lstrip(char : Char) : String
  # Characters of *chars*, not bytes: a byte mask built from `"©"` (C2 A9)
  # took the last byte of `é` (C3 A9) with it and answered a lone C3, which
  # is not a string. ASCII is the mask; a wider character is looked up whole.
  def lstrip(chars : String) : String
  def lstrip : String
  def partition(separator : String) : Tuple(String, String, String)
  def partition(separator : Char) : Tuple(String, String, String)
  def presence : String | ::Nil
  # The characters in reverse, found from the back by `each_char`'s rule,
  # so each character's bytes are copied as they stand and a byte that
  # begins no character is one of its own: taking every `10xxxxxx` byte
  # with the lead before it kept `C0 80` and a stray byte after `é` whole.
  # It decoded the whole string into an array of `Char`s first - four
  # bytes each, 520 MB to reverse 130 MB of ASCII.
  def reverse : String
  def rindex(char : Char, offset : Int32) : Int32 | ::Nil
  def rindex(search : String, offset : Int32) : Int32 | ::Nil
  def rindex(char : Char) : Int32 | ::Nil
  def rindex(search : String) : Int32 | ::Nil
  def rpartition(separator : String) : Tuple(String, String, String)
  def rpartition(separator : Char) : Tuple(String, String, String)
  def rstrip(char : Char) : String
  def rstrip(chars : String) : String
  def rstrip : String
  def split(separator : Char, limit : Int32 | ::Nil) : Array(String)
  # No limit is `nil`, and one of 1 or less is the string whole, as the
  # other library answers both. A negative limit split everything, and 0
  # stood for none: `"a,b".split(",", -1)` was `["a", "b"]` here and
  # `["a,b"]` there. An empty string is one empty piece whatever the
  # limit, as it is without one: `"".split(",", -1)` was `[]`.
  def split(separator : String, limit : Int32 | ::Nil = nil) : Array(String)
  def split(limit : Int32 | ::Nil) : Array(String)
  def split : Array(String)
  def squeeze(char : Char) : String
  # A run of one character becomes one of it, a character being what
  # `each_char` yields, so a run of bytes that begin none is one U+FFFD, as
  # in the other library; grouped by their `10xxxxxx` bytes, C3 41 came back
  # as it was. A byte at a time, `"耀".squeeze` (E8 80 80) dropped a
  # continuation byte and answered two bytes that are not a string, and
  # `"éé"` stayed two.
  def squeeze : String
  # A run of the same character in every one of *sets* becomes one of it.
  # Compared per character, so a multi-byte one is a run rather than three
  # bytes that never match.
  def squeeze(*sets : String) : String
  def strip(char : Char) : String
  def strip(chars : String) : String
  def sub(search : String, replacement : String) : String
  def sub(search : Char, replacement : Char) : String
  def sub(search : Char, replacement : String) : String
  def sub(search : String, & : (String -> String)) : String
  # The character's own bytes, as the non-block form asks: narrowed to one
  # byte, `'☃'` panicked with "arithmetic overflow" and `'é'` matched nothing.
  def sub(search : Char, & : (Char -> String)) : String
  # The other library's reading, whitespace around it aside: a sign, then
  # `inf`, `infinity` or `nan` in any case (a `nan` may close with a `(...)`
  # of letters, digits and `_`), or decimal digits. Text whose value leaves
  # the range is nil, as that reader's ERANGE makes it: nonzero digits that
  # round to zero, or digits that round past the largest double. Here
  # `"1e400".to_f?` was Infinity, `"1e-400".to_f?` 0.0, and only `Infinity`
  # and `NaN` spelled so were words: `"inf".to_f?` was nil.
  def to_f? : Float64 | ::Nil
  def to_i(base : Int32) : Int32
  def to_i64(base : Int32) : Int64
  def to_i64 : Int64
  def to_i64?(base : Int32) : Int64 | ::Nil
  def to_i64? : Int64 | ::Nil
  # Whitespace before and after the number is not part of it, as the other
  # library reads one (`Char#whitespace?`, so a no-break space too); inside
  # it is still refused. `" 42".to_i64?`, `"ff ".to_i?(16)` and a line
  # read with its `\n` were nil here, and 42 and 255 there.
  def to_i?(base : Int32) : Int32 | ::Nil
  def to_u64(base : Int32) : UInt64
  # The unsigned 64-bit reading, for the checksums and hashes a port keeps
  # in `UInt64`: digits in *base*, an optional `+`, and nil for a sign,
  # anything that is not a digit, or a value past 18446744073709551615.
  def to_u64 : UInt64
  def to_u64?(base : Int32) : UInt64 | ::Nil
  def to_u64? : UInt64 | ::Nil
  # Mapped per character. The 256-entry byte table this used could not hold
  # a character above ASCII: `"a☃".tr("☃", "x")` answered `"axxx"`, one `x`
  # per byte of the character it was asked to replace once. Both sets name
  # runs: `a-z` is the twenty-six letters, and a `-` at either end is
  # itself, as the other library's `delete` sets read them. Its `tr` pairs
  # the characters positionally, a `-` among them, so
  # `"bab".tr("a-c", "A-C")` is `"bAb"` there and `"BAB"` here.
  #
  # Both sets in ASCII - the common case, and a DNA complement's - are one
  # 256-entry byte table built once and one pass over the bytes, which a
  # character above ASCII passes through untouched because none of its
  # bytes is under 0x80. A byte that begins no character is not one of
  # those: the table kept it as it stood where the per-character map
  # writes U+FFFD, so text holding one takes that map too. Otherwise the
  # map is per character. A `to` shorter than `from` repeats its last
  # character, which is the rule the other library follows; an empty one
  # deletes the character. Each character was looked up in an array of
  # one-character strings and the answer appended to a string rebuilt every
  # time, and a 130 MB `tr` did not finish.
  def tr(from : String, to : String) : String
end