iyi

Standard library, 0.18.0

std/regex

Regular expressions, RE2's contract, written in iyi.

import std/regex to use it. Source: src/std/regex.iyi at v0.18.0.

Linear time: a Thompson NFA run as a Pike VM, one pass over the text with a thread list in priority order, so a search costs the text's length times the pattern's and never more. Each thread carries its own capture positions, copied as it splits, which is how RE2 reports groups without ever backtracking. Lookaround, backreferences, recursion, possessive quantifiers, atomic groups and (?#..) comments are refused with a sentence: the first five are not regular, and RE2 has no comments.

Matches are leftmost-first, as RE2's and Perl's are: a|ab on ab answers a, a greedy * takes what it can and a lazy *? what it must, and a group reports what the winning thread passed through.

Code points, not bytes: the text is read as UTF-8, one character a step, so ., a negated class and a range like [α-ω] take a whole é or 中. A byte that begins no character is one U+FFFD, as String#each_char reads it. Positions stay byte offsets, as everywhere in iyi: find, =~, index and MatchData#byte_begin answer bytes, and MatchData#begin the character index the other library answers.

The surface is RE2's: ., ^ $ \A \z, \b \B, \d \w \s and their negations (ASCII, as in RE2: \w is [0-9A-Za-z_] and \b sits between one of those and anything else), classes [a-z] [^\s] over any code point, RE2's ASCII [[:alpha:]] [[:^digit:]], Unicode's \p{L} \pN \p{Greek} \P{Lu} \p{^Han} (every general category, its one-letter class, Any, and every script), \xHH \x{10FFFF} \0oo, \Q..\E, ? * + {n} {n,} {n,m} and their lazy forms, groups ( ) (?: ) (?<name> ) (?P<name> ), and the flags i (Unicode simple case folding, k matching the Kelvin sign), m (^ and $ at each line), s (. takes a newline), U (greedy and lazy swap) and x (whitespace and # comments are not the pattern), inline as (?i) and (?i-s) or scoped as (?i:..). Options gives a whole pattern its flags; MULTILINE is the other library's, m and s at once, and MULTILINE_ONLY and DOTALL are each half.

match? is the whole string; find the first match; match its groups; match_at the match that begins at a given byte, which is what std/string_scanner reads tokens with; scan every non-overlapping match; split and replace read those, and an empty match costs no text. A literal, /a+b/i, is Regex.compile("a+b", Options::IGNORE_CASE) compiled once for the program. There is no $~: a match is a value match answers, never a variable a call sets behind the program's back.

Regexstruct

std/regexiyi doc src/std/regex.iyi
pub struct Regex
  # One match: the text it was found in and where each group began and
  # ended, in bytes. A group that took no part in the match - the second
  # of `(a)|(b)` when `a` matched, `(x)?` when `x` did not - has no span,
  # and `[]?` answers nil for it.
  class MatchData
    # The text group *n* matched. A group the pattern does not have, or one
    # that took no part in this match, is a mistake the program made, and
    # it panics with the reason.
    def [](n : Int32) : String
    # The text the group called *name* matched; a name the pattern does not
    # have, or a group that took no part, panics with the reason.
    def [](name : String) : String
    # The text group *n* matched, or nil for a group that took no part or
    # that the pattern does not have.
    def []?(n : Int32) : String | ::Nil
    # The text the group called *name* matched, or nil for a group that
    # took no part or a name the pattern does not have.
    def []?(name : String) : String | ::Nil
    # Where group *n* begins, in characters, as the other library counts:
    # `"クリスタル".match(/リ(ス)/).begin(0)` is 1 where `byte_begin` is 3.
    def begin(n : Int32 = 0) : Int32
    # Where group *n* begins, in bytes: what `String#[]` and `index` count
    # in. A negative *n* counts from the last group.
    def byte_begin(n : Int32 = 0) : Int32
    # Where group *n* ends, in bytes.
    def byte_end(n : Int32 = 0) : Int32
    # What each unnamed group matched, in order, nil for one that took no
    # part; the named ones are `named_captures`, as in the other library.
    def captures : Array(String | ::Nil)
    # Where group *n* ends, in characters.
    def end(n : Int32 = 0) : Int32
    # *replacement* with its references filled from this match: `\0` to
    # `\9` are groups, `\k<name>` a named one, `\\` one backslash, and any
    # other `\` is itself - how the other library reads the replacement
    # `String#sub` and `gsub` are given. A group that took no part is
    # empty; a name the pattern does not have panics.
    def expand(replacement : String) : String
    # How many capture groups the pattern has, the whole match not counted.
    def group_size : Int32
    def initialize(regex : Regex, string : String, slots : Array(Int32))
    # `Regex::MatchData("ab" 1:"a" name:"b")`, as the other library writes it.
    def inspect : String
    # What each named group matched, nil for one that took no part.
    def named_captures : Hash(String, String | ::Nil)
    # The text after the match.
    def post_match : String
    # The text before the match.
    def pre_match : String
    # The pattern that matched.
    def regex : Regex
    # How many groups there are, the whole match counted: `(a)(b)` is 3.
    def size : Int32
    # The text the match was found in.
    def string : String
    # Every group, the whole match first.
    def to_a : Array(String | ::Nil)
    # The text the whole pattern matched.
    def to_s : String
  end
  # The flags a whole pattern is compiled with, combined with `|`. The
  # numbers are the other library's, which is what lets a literal's flags
  # (`/a/i`, `/a/mx`) reach this type unchanged.
  enum Options : Int32
    IGNORE_CASE = 1
    MULTILINE = 6
    DOTALL = 2
    MULTILINE_ONLY = 4
    EXTENDED = 8
  end
  # The same pattern with the same flags.
  def ==(other : Regex) : Bool
  # Whether the pattern occurs anywhere in *text*: what `case text when
  # /x/` asks.
  def ===(text : String) : Bool
  # Where the first match begins, in bytes, or nil. It sets nothing: the
  # other library's `=~` also fills `$~`, which iyi does not have, and a
  # program that wants the groups asks `match`.
  def =~(text : String) : Int32 | ::Nil
  # Where the first match at or after byte *offset* begins, in bytes, as
  # `String#byte_index` answers for a string.
  def byte_index(text : String, offset : Int32 = 0) : Int32 | ::Nil
  # How many capture groups the pattern has.
  def capture_count : Int32
  # The machine is compiled first either way: it is what refuses a pattern
  # this engine does not take, and the fast paths only ever see one it did.
  # They read the pattern's text as written, so a pattern with flags, which
  # change what that text means, is the machine's alone. So is one holding
  # U+FFFD: the machine reads a byte that begins no character as that
  # character, and comparing bytes would find only its three-byte spelling.
  def self.compile(pattern : String, options : Options = Options::None) : Regex
  # Every non-overlapping match with its groups, leftmost first, as `scan`
  # finds them.
  def each_match(text : String, & : (MatchData -> Nil)) : Nil
  # *text* with every character a pattern would read as an operator
  # escaped, so `Regex.compile(Regex.escape(text))` matches *text* and
  # nothing else - with `x` too, whose whitespace and `#` are escaped.
  def self.escape(text : String) : String
  def find(text : String) : String | ::Nil
  # The number of the group called *name*, or nil.
  def group_index(name : String) : Int32 | ::Nil
  # The name of group *n*, or nil for an unnamed group or one the pattern
  # does not have.
  def group_name(n : Int32) : String | ::Nil
  def initialize(nfa : RxNfa, literal : String | ::Nil, runs : RxRuns | ::Nil, source : String, options : Options, names : Array(String | ::Nil))
  # The pattern as a literal would write it, `/a\/b/ix`.
  def inspect : String
  # The first match at or after byte *pos*, with its groups; a negative
  # *pos* counts from the end. `^` and `\A` still mean the start of the
  # text, not *pos*.
  def match(text : String, pos : Int32 = 0) : MatchData | ::Nil
  # Whether the pattern spans the whole of *text*. The other library's
  # `match?` asks whether it occurs anywhere, which is `===` and
  # `String#matches?` here.
  def match?(text : String) : Bool
  # The match that begins exactly at byte *pos*, with its groups, or nil:
  # the other library's `match_at_byte_index` with `MatchOptions::ANCHORED`,
  # Python's `pattern.match(text, pos)`, what a scanner asks at its
  # offset. As there, the text before *pos* is still the text: `^` and
  # `\A` mean its start, not *pos*, and `\b` sees the byte before *pos*.
  # A negative *pos* counts from the end, as `match`'s does.
  def match_at(text : String, pos : Int32) : MatchData | ::Nil
  # Each named group's number and name.
  def name_table : Hash(Int32, String)
  # The other library's name for `compile`.
  def self.new(pattern : String, options : Options = Options::None) : Regex
  # The flags the pattern was compiled with; inline ones are the
  # pattern's own text and are not here.
  def options : Options
  # Every match replaced, into one buffer: `out = out + ...` copied all of
  # the text so far at each match, and eleven substitutions over a 60 MB
  # sequence with a million matches each did not finish. The replacement
  # is taken as written; `String#gsub` is the one that reads `\1` in it.
  def replace(text : String, replacement : String) : String
  # Every match replaced by what the block answers for it — the per-match
  # substitution (`gsub(pattern, hash)` in the other library) that a fixed
  # replacement cannot say.
  def replace(text : String, & : (String -> String)) : String
  def scan(text : String) : Array(String)
  # The pattern as written, the other library's `Regex#source`: what a
  # literal was compiled from, for a program that prints its patterns.
  def source : String
  def split(text : String) : Array(String)
  # The pattern with its flags inside it, `(?i-msx:a+)`, which compiles to
  # a pattern that matches as this one does: what `union` joins.
  def to_s : String
  # One pattern matching any of *patterns*: a string as its own text, a
  # `Regex` with its flags kept to itself (`(?i-msx:...)`, its `to_s`).
  # The groups of every pattern after the first are numbered after the
  # groups before them.
  def self.union(patterns : Array(T)) : Regex forall T
  def self.union(*patterns : String | Regex) : Regex
end

Stringclass, reopened

std/regexiyi doc src/std/regex.iyi
class ::String
  def =~(pattern : Std::Regex::Regex) : Int32 | ::Nil
  def gsub(pattern : Std::Regex::Regex, replacement : String) : String
  def gsub(pattern : Std::Regex::Regex, & : (String, Std::Regex::Regex::MatchData -> String)) : String
  # Each match replaced by its entry in *table*; a match the table has no
  # entry for is dropped, as the other library drops it.
  def gsub(pattern : Std::Regex::Regex, table : Hash(String, String)) : String
  def index(pattern : Std::Regex::Regex, offset : Int32 = 0) : Int32 | ::Nil
  def match(pattern : Std::Regex::Regex, pos : Int32 = 0) : Std::Regex::Regex::MatchData | ::Nil
  def matches?(pattern : Std::Regex::Regex) : Bool
  def scan(pattern : Std::Regex::Regex) : Array(Std::Regex::Regex::MatchData)
  def scan(pattern : Std::Regex::Regex, & : (Std::Regex::Regex::MatchData -> Nil)) : String
  def sub(pattern : Std::Regex::Regex, replacement : String) : String
  def sub(pattern : Std::Regex::Regex, & : (String, Std::Regex::Regex::MatchData -> String)) : String
end