iyi

Standard library, 0.18.0

std/xml

XML for iyi: a well-formedness parser, a document tree, a serializer, and XPath 1.0.

import std/xml to use it. Source: src/std/xml.iyi at v0.18.0.

parse reads an XML 1.0 document - declaration, comments, processing instructions, an internal DTD subset with its entity, element, attribute-list and notation declarations, elements, attributes, namespaces, text, CDATA, character and entity references - into a Document of Nodes. parse? answers nil where parse panics. A node can be walked (children, parent, root), searched (find, find_all, and xpath, xpath_nodes, xpath_node, xpath_string, xpath_float, xpath_bool with all of XPath 1.0 - see XPath), located (path), edited (add_child, set_attribute, text=) and written back out (to_xml, inner_xml). escape, escape_attribute and unescape are the character-reference rules on their own.

Bytes are UTF-8 and stay as they are: text is copied by the run, never decoded and re-encoded, so a document with "café" or an emoji reads and writes the same bytes. Line ends are normalised (\r\n and \r become \n) and attribute values have their literal tabs and newlines replaced by spaces, as XML 1.0 says. The serializer writes what a document reads back as: &amp; &lt; &gt; and &#13; in text, &quot; and &#9; &#10; &#13; in attribute values as well, ]]> inside CDATA as the end of one section and the start of the next (and a carriage return in it as &#13; between two), namespace declarations before attributes, an empty element as <a/>, and a newline after the declaration and after every top-level node; a document declared US-ASCII (or ascii) has every other character in its text, attribute values and CDATA written as a decimal reference. A comment or a processing instruction has no such escape, so -- in a comment, a comment ending in -, and ?> in a processing instruction are refused when the node is made or its text set; in a document declared US-ASCII, to_xml refuses a character past U+007F in one of them or in a name. to_xml refuses a name that is not an XML name (and a processing instruction target xml, or one holding a colon) and a character XML cannot hold at all (a C0 control but tab, newline and carriage return; U+FFFE; U+FFFF): written as they were, they made a document no parser reads. A parsed document holds none of them.

What it refuses, each with the line and column:

* an undeclared entity (`&bogus;`), an unterminated one (`&amp` without
  the `;`), and a character reference that is not an XML character
  (surrogates, U+0000 and the other C0 controls, anything past U+10FFFF);
* `--` inside a comment and `]]>` in text; a control character (other than
  tab, newline and carriage return) anywhere; a byte that is not UTF-8, a
  character that is not an XML character (U+FFFE, U+FFFF, an encoded
  surrogate), and in a document declared US-ASCII any byte past 0x7F;
* an XML declaration without its version first, or with `version`,
  `encoding`, `standalone` out of that order or twice, or a version that
  is not digits, a dot and digits;
* a raw `<` in an attribute value, an attribute given twice - by one
  spelling or through two prefixes bound to the same uri - a namespace
  prefix nobody declared, a code point a name cannot hold (XML 1.0, fifth
  edition: `<a×b/>`, `<·a/>`), a name with a colon first, last or twice
  or a local part that cannot start a name, a colon in an entity name or a
  processing instruction target, a declaration of the `xmlns` prefix or
  of `xml` to another uri, the `xml` or `xmlns` uri bound to any other
  prefix or to the default namespace, an unquoted value, a second root
  element, a closing tag that does not match, an unclosed anything;
* an external DTD subset or entity (`SYSTEM`/`PUBLIC`) - there is no
  file or network access here, so there is no XXE;
* more than `max_entity_expansions` entity expansions (1,000 by default),
  an entity that expands more than ten levels deep, or expansions that
  copy more than 100 times the document's size (8 MiB at least), which
  is what a "billion laughs" or a quadratic-blowup document is;
* elements nested deeper than `max_depth` (256 by default);
* an entity whose replacement text holds a `<` - markup where the
  reference is in content, an error where it is in an attribute value -
  or, in content, `]]>`;
* an `<!ELEMENT>`, `<!ATTLIST>` or `<!NOTATION>` declaration that is not
  its production in XML 1.0 (`<!ELEMENT>`, `(a,|b)`, an attribute with
  no default), and an attribute default a written value could not be;
* markup inside an entity's replacement text, a parameter entity
  reference, and an encoding other than UTF-8 or US-ASCII: each is named
  as unsupported rather than read as something else.

unescape follows the same reference rules as the parser: the five predefined entities and valid character references are decoded, everything else is refused with the same sentence. The DTD is read for its entities and its attribute-list declarations: an attribute given a default is added to an element written without it, and a value whose declared type is not CDATA has its spaces trimmed and collapsed. Element and notation declarations are checked and not used, as nothing is validated. The DTD is not kept but for which attributes it declares of type ID (Document#id_attributes, what XPath's id() reads): to_xml writes no DOCTYPE, and entity references come out as the text they stood for.

escapedef

text as character data: &, <, > and \r become references, every other byte is copied as it is. With ascii, a character past U+007F becomes a decimal reference as well, for a document declared US-ASCII.

std/xmliyi doc src/std/xml.iyi
pub def escape(text : String, ascii : Bool = false) : String

escape_attributedef

text for inside a double-quoted attribute value: " and a literal tab or newline become references as well, so the value reads back unchanged. ascii is as for escape.

std/xmliyi doc src/std/xml.iyi
pub def escape_attribute(text : String, ascii : Bool = false) : String

parsedef

std/xmliyi doc src/std/xml.iyi
pub def parse(xml : String, max_depth : Int32 = 256, max_entity_expansions : Int32 = 1000) : Document

parse?def

std/xmliyi doc src/std/xml.iyi
pub def parse?(xml : String, max_depth : Int32 = 256, max_entity_expansions : Int32 = 1000) : Document | ::Nil

unescapedef

std/xmliyi doc src/std/xml.iyi
pub def unescape(text : String) : String

unescape?def

The five predefined entities and character references decoded, the same way and with the same refusals as inside a document; nil where unescape panics.

std/xmliyi doc src/std/xml.iyi
pub def unescape?(text : String) : String | ::Nil

xml_char?def

Whether cp is a character XML 1.0 allows in a document (production [2]).

std/xmliyi doc src/std/xml.iyi
pub def xml_char?(cp : Int32) : Bool

Attributestruct

std/xmliyi doc src/std/xml.iyi
pub struct Attribute
  def initialize(name : String, value : String, namespace_uri : String = "")
  def local_name : String
  def name : String
  # The namespace the prefix was bound to; "" for an unprefixed attribute,
  # which has no namespace, not even the default one.
  def namespace_uri : String
  def prefix : String
  # With *ascii*, as `Namespace#to_s`.
  def to_s(ascii : Bool = false) : String
  def value : String
end

Documentclass

std/xmliyi doc src/std/xml.iyi
pub class Document < Node
  def encoding : String
  def encoding=(value : String) : String
  # The attributes the internal DTD subset declares of type ID, by element
  # name: what XPath's `id()` finds an element by, beside `xml:id`. The
  # rest of the DTD is not kept.
  def id_attributes : Hash(String, Array(String))
  # The attributes the internal DTD subset declares of type ID, by element
  # name: what XPath's `id()` finds an element by, beside `xml:id`. The
  # rest of the DTD is not kept.
  def id_attributes=(value : Hash(String, Array(String))) : Hash(String, Array(String))
  def initialize(version : String = "1.0", encoding : String = "UTF-8", standalone : Bool | ::Nil = nil)
  def root : Node
  def root? : Node | ::Nil
  def standalone : Bool | ::Nil
  def standalone=(value : Bool | ::Nil) : Bool | ::Nil
  # The declaration and the document's own children are checked as the
  # reader checks them: the declaration first, the shape after each node's
  # own refusals. `version` and `encoding` were written between quotes as
  # they were, so an encoding holding a `"` closed the declaration and wrote
  # markup of its own (`Document.new(encoding: "x\" standalone=\"maybe")`),
  # and a version that is not `1.` and digits or an encoding this module
  # does not read (`ISO-8859-1`) made a document its parser refuses. So did
  # a second root element, none, text that is not white space, or a CDATA
  # section beside the root.
  def to_xml : String
  def version : String
  def version=(value : String) : String
end

Namespacestruct

std/xmliyi doc src/std/xml.iyi
pub struct Namespace
  def default? : Bool
  def initialize(prefix : String, uri : String)
  def prefix : String
  # With *ascii*, a character past U+007F in the uri is written as a
  # character reference, for a document declared US-ASCII.
  def to_s(ascii : Bool = false) : String
  def uri : String
end

Nodeclass

std/xmliyi doc src/std/xml.iyi
pub class Node
  # The same node, by identity. It compared `id`s, and a `Document` counts
  # its own: the first `Document.new` and the first `Node.new_element`
  # were both 1, so the two were `==`, and `add_child`'s question "is this
  # node one of my ancestors" would answer yes for a stranger.
  def ==(other : Node) : Bool
  def [](name : String) : String
  def []?(name : String) : String | ::Nil
  # Moves *child* here, as the last child: a node is in one place in a
  # tree. It stayed among its old parent's children as well, so a node
  # moved from `<a>` to `<b>` was written under both and an attribute set
  # on it showed in both places; and a node added under itself or under one
  # of its own descendants made a loop that `to_xml` followed until the
  # stack ran out. That is refused, and so is a document as a child.
  def add_child(child : Node) : Node
  # The parser's own append: the child was just made, under this node, and
  # is in no list yet.
  protected def append_parsed(child : Node) : Node
  # The value of the attribute with exactly this name (`"x:attr"` is not
  # `"attr"`), or nil.
  def attribute(name : String) : String | ::Nil
  def attributes : Array(Attribute)
  def attributes=(value : Array(Attribute)) : Array(Attribute)
  # What `to_xml` refuses to write (see `serialize_node`): a name that is
  # not an XML name, and a character XML cannot hold raw or as a reference.
  protected def self.check_writable_name(name : String, what : String) : Nil
  protected def self.check_writable_text(text : String) : Nil
  def children : Array(Node)
  def children=(value : Array(Node)) : Array(Node)
  protected def collect_find_all(name : String, res : Array(Node)) : Nil
  protected def collect_text_into(parts : Array(String)) : Nil
  # A text, CDATA, comment or PI node's own text; for an element or a
  # document, the text and CDATA of every descendant, joined.
  def content : String
  def element_children : Array(Node)
  # The first descendant element whose name or local name is *name*.
  def find(name : String) : Node | ::Nil
  # Every descendant element whose name or local name is *name*, in document
  # order.
  def find_all(name : String) : Array(Node)
  def first_child : Node | ::Nil
  def first_element_child : Node | ::Nil
  # The first character past U+007F in *text*, decoded, or -1.
  protected def self.first_non_ascii(text : String) : Int32
  def has_attribute?(name : String) : Bool
  def id : Int32
  def initialize(node_type : NodeType, name : String = "", content : String = "")
  def inner_xml : String
  def last_child : Node | ::Nil
  def local_name : String
  def local_name=(value : String) : String
  def name : String
  def name=(value : String) : String
  # The namespace this element's name is in: the one its prefix is bound to,
  # or the default namespace in scope, or nil.
  def namespace : Namespace | ::Nil
  def namespace_uri : String
  def namespaces : Array(Namespace)
  def namespaces=(value : Array(Namespace)) : Array(Namespace)
  def self.new_cdata(content : String) : Node
  def self.new_comment(content : String) : Node
  def self.new_element(name : String) : Node
  def self.new_pi(target : String, content : String) : Node
  def self.new_text(content : String) : Node
  def node_type : NodeType
  def node_type=(value : NodeType) : NodeType
  def parent : Node | ::Nil
  def parent=(value : Node | ::Nil) : Node | ::Nil
  # Where this element is, as `/root/item[2]/name[1]`: each step counts
  # same-named element siblings from 1.
  def path : String
  def prefix : String
  def prefix=(value : String) : String
  # `xmlns=""` undeclares the default namespace, so a declaration with an
  # empty uri answers nil rather than being passed over.
  def resolve_default_namespace : Namespace | ::Nil
  def resolve_namespace_prefix(prefix : String) : Namespace | ::Nil
  # The root element of the tree this node is in.
  def root : Node
  # With *ascii* (a document declared US-ASCII), a character past U+007F in
  # text or an attribute value is written as a character reference: written
  # as its UTF-8 bytes, it made a document no other parser reads. A name, a
  # comment and a processing instruction have no references, so one holding
  # such a character is refused; it was written as UTF-8, and this module's
  # own parser refused the result ("byte 0xC3 is not US-ASCII, the declared
  # encoding").
  #
  # A name that is not an XML name, and a character XML cannot hold at all
  # (a C0 control but tab, newline and carriage return; U+FFFE, U+FFFF; a
  # byte that is not UTF-8), are refused here: both were written as they
  # were, so `set_attribute("k=\"1\" j", "2")` wrote two attributes and a
  # "\u0001" in text a document no parser reads. A parsed tree holds
  # neither.
  protected def serialize_node(buf : Array(String), ascii : Bool = false) : Nil
  # Sets or replaces the attribute; a prefixed name takes the namespace its
  # prefix is bound to here, "" where none is declared.
  def set_attribute(name : String, value : String) : Attribute
  def text : String
  # For a text, CDATA, comment or PI node, its own text; for an element, its
  # children are replaced by one text node with *value*.
  def text=(value : String) : Nil
  def to_s : String
  def to_xml : String
  # Nothing but XML's white space: space, tab, newline, carriage return.
  protected def self.xml_blank?(text : String) : Bool
  # XPath 1.0 (see `XPath`) with this node as the context node: the value
  # the expression is, of whichever type. *namespaces* binds the prefixes
  # the expression uses (a prefix not in it is looked up where it is
  # declared at this node); *variables* gives `$name` its value. Each call
  # compiles *expression*; `XPath.parse` once and `evaluate` many times
  # for a loop.
  def xpath(expression : String, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : XPathValue
  def xpath_bool(expression : String, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : Bool
  def xpath_float(expression : String, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : Float64
  # The first node of that node-set in document order, or nil.
  def xpath_node(expression : String, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : XPathNode | ::Nil
  # The node-set *expression* is, in document order; panics when it is a
  # number, a string or a boolean.
  def xpath_nodes(expression : String, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : Array(XPathNode)
  # The value converted as `string()`, `number()` and `boolean()` convert.
  def xpath_string(expression : String, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : String
end

NodeTypeenum

std/xmliyi doc src/std/xml.iyi
pub enum NodeType : Int32
  Element = 0
  Text = 1
  CData = 2
  Comment = 3
  ProcessingInstruction = 4
  Document = 5
  def cdata? : Bool
  def pi? : Bool
end

Parserclass

The parser keeps the first thing that went wrong in error and, once something has, runs its cursor to the end so every loop above it unwinds: parse? then answers nil and parse panics with that sentence.

std/xmliyi doc src/std/xml.iyi
pub class Parser
  # An entity's replacement text in an attribute value: a tab, newline or
  # carriage return in it is a space, as one written in the value is. One
  # that a character reference in the replacement text names is not.
  protected def self.attribute_spaces(text : String) : String
  # A value whose declared type is not CDATA, normalised further (XML 1.0,
  # 3.3.3): the spaces at either end dropped and each run of them made one.
  # Only U+0020 counts; a tab a character reference wrote stays. Values
  # were kept as written: `x="  p   q  "` of an NMTOKENS `x` read as is.
  protected def self.collapse_spaces(value : String) : String
  # The sentence the last parse failed with, or "" when it did not.
  def error : String
  # Where the first byte at or after *from* that does not begin an XML
  # character is, with the sentence it is refused with; -1 and "" when there
  # is none. Only a C0 control is caught where the text is read, so without
  # this a byte that is not UTF-8, an encoded surrogate, U+FFFE and U+FFFF
  # were read and written back, while their character references were
  # refused. With *ascii*, any byte past 0x7F is one.
  protected def self.first_bad_character(input : String, from : Int32, ascii : Bool) : Tuple(Int32, String)
  protected def self.hex4(value : Int32) : String
  def initialize(input : String, max_depth : Int32 = 256, max_expansions : Int32 = 1000)
  # Production [4a]: digits, `-`, `.`, U+00B7, the combining marks
  # U+0300-036F and U+203F-2040 continue a name without starting one.
  protected def self.name_char?(cp : Int32) : Bool
  # Whether all of *text* is one name.
  protected def self.name_shaped?(text : String) : Bool
  # XML 1.0 (fifth edition), production [4]: the characters that start a
  # name, by code point. Any byte past 0x7F was taken as starting and
  # continuing one, so `<a×b/>` read as an element named "a×b", a no-break
  # space after a name became part of it, and `<·a/>` was an element.
  protected def self.name_start?(cp : Int32) : Bool
  # `\r\n` and a lone `\r` both mean `\n` (XML 1.0, 2.11).
  protected def self.normalize_line_ends(input : String) : String
  def parse : Document
  def parse? : Document | ::Nil
  # Production [13]: space, newline, ASCII letters and digits, and
  # -'()+,./:=?;!*#@$_%
  protected def self.public_id_byte?(b : UInt8) : Bool
  # One or more digits, a dot, then digits: the version numbers the other
  # library reads (`1.0`, `1.1`, `1.`).
  protected def self.version_number?(value : String) : Bool
end

XPathclass

An XPath 1.0 expression, compiled once and evaluated against any node.

All of XPath 1.0 (W3C, 1999): the thirteen axes and the abbreviations (//, ., .., @), name tests with prefixes, the node-type tests, predicates by position and by value, filter expressions, |, the operators with their conversions (section 3.4's comparisons of node-sets included), variables, and the 27 functions of the core library.

A prefix in a name test is looked up in namespaces, and when it is not there, in the declarations in scope at the node the expression is evaluated against; xml is always bound. An unprefixed name is in no namespace even where a default namespace is declared, as section 2.3 says: <feed xmlns="urn:a"> is a:feed with {"a" => "urn:a"}, never feed.

Where libxml2 - the engine under the other library's XPath - departs from the recommendation, this follows the recommendation:

* a number becomes the fewest digits that read back as it, written
  out in full: `string(1 div 3)` is `0.3333333333333333` and
  `string(-0.0000001)` is `-0.0000001`, where libxml2 writes fifteen
  digits and `-1e-07`;
* a number in an expression is digits and one point: `1e0` is refused;
* an attribute's following axis holds its element's descendants,
  which come after the attribute in document order;
* `xmlns=""` leaves no namespace node for the default namespace;
* an empty CDATA section is no text node (section 5.7);
* `id()` splits its argument on white space wherever it is: libxml2
  finds nothing for `id(' a')`;
* `.//.` (or `/.//.`) as the whole expression holds the context node
  itself; libxml2 2.9, which darwin carries, leaves it out.

parse panics on a malformed expression with what was expected and the column; parse? answers nil. Evaluation panics on what only a value can show: a function given a number where it takes a node-set, a variable not given, a prefix bound nowhere.

std/xmliyi doc src/std/xml.iyi
pub class XPath
  # The minimum and maximum argument count of each core function; nil for
  # a name that is not one.
  protected def self.arity(name : String) : Tuple(Int32, Int32) | ::Nil
  def bool(node : Node | XPathNode, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : Bool
  # The value at *node*, which is the context node, at position 1 of 1.
  def evaluate(node : Node, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : XPathValue
  def evaluate(node : XPathNode, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : XPathValue
  def initialize(source : String, expression : XPathExpression, prefixes : Array(String))
  # The node-set the expression is; panics when it is another type.
  def nodes(node : Node | XPathNode, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : Array(XPathNode)
  def number(node : Node | XPathNode, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : Float64
  # A number as a string (section 4.2): NaN, Infinity, an integer without a
  # point, and otherwise the fewest digits that read back as the number,
  # never with an exponent; both zeros are `0`.
  def self.number_string(value : Float64) : String
  def self.parse(source : String) : XPath
  def self.parse?(source : String) : XPath | ::Nil
  # `round()` (section 4.4): the nearest integer, a tie to the one toward
  # positive infinity, with the sign of a zero kept: `round(-0.5)` is -0.
  def self.round(value : Float64) : Float64
  def source : String
  # XPath's white space: space, tab, carriage return, line feed.
  protected def self.space?(b : UInt8) : Bool
  def string(node : Node | XPathNode, namespaces : Hash(String, String) | ::Nil = nil, variables : Hash(String, XPathValue) | ::Nil = nil) : String
  # A string as a number (section 4.4): white space, an optional minus,
  # digits with at most one point, white space; anything else is NaN.
  def self.string_number(text : String) : Float64
  # `boolean()` of a value.
  def self.to_bool(value : XPathValue) : Bool
  # `number()` of a value.
  def self.to_number(value : XPathValue) : Float64
  # `string()` of a value.
  def self.to_s(value : XPathValue) : String
  protected def self.type_name(value : XPathValue) : String
end

XPathKindenum

The seven kinds of node in XPath 1.0's data model (section 5): the document is the root node, and a CDATA section is a text node.

std/xmliyi doc src/std/xml.iyi
pub enum XPathKind : Int32
  Root = 0
  Element = 1
  Attribute = 2
  Namespace = 3
  ProcessingInstruction = 4
  Comment = 5
  Text = 6
end

XPathNodestruct

A node as an XPath expression sees it. A tree node is itself. An attribute or a namespace node is not a Node here - an element keeps its attributes and its declarations as values - so it is the element it belongs to and which of them it is. Two are == when they are the same node, which is what a node-set's union and xpath_node compare.

std/xmliyi doc src/std/xml.iyi
pub struct XPathNode
  def ==(other : XPathNode) : Bool
  # The attribute an attribute node is, or nil.
  def attribute : Attribute | ::Nil
  # An attribute node's place in its element's `attributes`, a namespace
  # node's in the element's namespaces in scope ordered by prefix, a text
  # node's among its parent's children where it is known; else -1.
  def index : Int32
  def initialize(node : Node, kind : XPathKind, index : Int32 = -1, namespace_prefix : String = "", namespace_target : String = "")
  def kind : XPathKind
  # `local-name()`: the name without its prefix.
  def local_name : String
  # `name()`: an element's or attribute's name as written, a processing
  # instruction's target, a namespace node's prefix; "" for the rest.
  def name : String
  # A namespace node's prefix ("" for the default namespace) and uri.
  def namespace_prefix : String
  def namespace_target : String
  # `namespace-uri()`: an element's or attribute's namespace, else "".
  def namespace_uri : String
  # The node itself, or the element an attribute or namespace node is on.
  def node : Node
  # *node* as the tree node it is: a document is the root node, a CDATA
  # section a text node - with the text and CDATA after it, up to the next
  # node of another kind, one text node (section 5.7). *place* is a text
  # node's index among its parent's children, when the caller knows it.
  def self.of(node : Node, place : Int32 = -1) : XPathNode
  # The string-value (XPath 1.0, section 5): every descendant text node's
  # text joined for the root and an element, an attribute's value, a
  # namespace node's uri, a text node's run of text and CDATA, and the
  # text of the rest.
  def text : String
  # A tree node as XML; an attribute as `name="value"`; a namespace node as
  # its declaration.
  def to_s : String
end

XPathValuealias

What an XPath expression evaluates to: a node-set in document order, a number, a string or a boolean.

std/xmliyi doc src/std/xml.iyi
pub alias XPathValue = Array(XPathNode) | Float64 | String | Bool