# Parserino: Fast HTML5 Parser and DOM Manipulation for D

Parserino is a high-performance HTML5 parser and DOM manipulation library for the D programming language, written in pure D (no C dependencies; born as a D port of lexbor, from which the tree construction is derived). Names and behaviour follow the DOM, HTML and Selectors 4 standards; the parser passes the tree construction tests of the web-platform-tests.

## Rules

- **Nodes and elements**: `Node` is any node; `Element` is a node that is an element. An `Element` converts implicitly to `Node`; `node.asElement` gives an `Element` (invalid if the node isn't an element).
- **Navigation filter**: navigation methods take the kinds of nodes to see as a template parameter `Show` (the bits of DOM `NodeFilter.whatToShow`, combine them with `|`). The default is `Show.Element`, and then the result is an `Element` or a range of `Element`s; with other filters the results are `Node`s.
- **Errors**: searches and navigation give an invalid node (`== null`, `isValid == false`) or an empty range when nothing is found; nodes that can't have children (texts, comments) just have none. Calling a method on an invalid node or document, or an impossible mutation (a child of a text, a node inside itself, a node of another document, a sibling of a node without parent), throws `ParserinoException`.
- **Views**: getters that copy a stored string have a `...View` variant returning a slice of the document memory (valid until the value changes or the document is destroyed): `localNameView`, `idView`, `getAttributeView`, `classesView`, `attributesView`, `dataView`. Computed values (`textContent` of an element, `innerHTML`, `title`) have none.
- **Memory**: `Document` is reference counted; every `Node` keeps its document alive; removed nodes stay valid and can be inserted again.

## Core Types

### `Document`
The root of the DOM tree.

- `this(string html, Parsing parsing = Parsing.Eager, size_t chunkSize = DefaultChunkSize)`: Parses the document. With `Parsing.Lazy` it parses incrementally, only as far as queries and accessors need. Results are identical to a full parse. Mutations complete the parsing first.
- `this(string html, ParseOptions options)`: `ParseOptions` has `parsing`, `chunkSize` and `scripting` (with `scripting: true` the content of `<noscript>` is text, as in a browser; by default it's parsed as html).
- `ctDocument!(html, ParseOptions options = ParseOptions.init)`: A document parsed at compile time. Each call returns a new mutable document, rebuilt from the stored tree without parsing (e.g. `ctDocument!(import("page.html"))`).
- `Snapshot snapshot`, `this(Snapshot)`: A compact, immutable copy of the tree; `Document(snap)` makes a new independent document from it much faster than parsing. Use it for a page filled many times (a template, for each request): parse once, then `Document(snap)` each time. Snapshots can be shared between threads; the documents borrow their strings (the snapshot is kept alive as long as needed).
- `Document dup`: A deep copy.
- `void finishParsing()`, `bool isParsing`, `size_t bytesParsed`: Lazy parsing.
- `bool isValid`: True if the document is initialized.
- `Element documentElement`, `Element head`, `Element body` (the first `<body>` or `<frameset>` child of `<html>`, as in the DOM), `Node node` (the document itself as a `Node`).
- `string title` (get/set; the getter strips and collapses whitespace), `string rawTitle`.
- `Element createElement(string tagName)`, `Node createText(string text)`, `Node createComment(string text)`.
- `Node fragment(string html, string context = "body")`: Parses html in the context of an element (as the standard fragment parsing: `"tr"` makes `<td>` valid, `"textarea"` makes everything text, `"svg"`/`"math"` for foreign content) into a `DocumentFragment` node. Inserting the fragment moves its children.
- `string toString()`, `toString(sink)` (a delegate or an output range).
- `string toPrettyString()`: The html indented, to read it: each block element on its own line, two spaces per level; elements with only text and inline elements stay on one line. Whitespace between nodes changes (not in `<pre>`, `<textarea>`, `<script>`, ...): use `toString` to store or send html.
- `Element byId(id)`, `byClass(name)`, `byTagName(name)` (`"*"`: all), `bySelector(css)` / `bySelector(Selector)` / `bySelector!css`, `byComment(text)` (whitespace at the ends ignored), `byCommentExact(text)`, `children!show`, `descendants!show`.

### `Node`
- `bool isValid`, `NodeType nodeType`, `string nodeName` (`"DIV"`, `"#text"`, `"#comment"`, `"#document"`, ...), `bool isElement`, `bool isText`, `bool isComment`, `Element asElement`, `Document ownerDocument`.
- `string textContent`: Get/set. Elements and fragments: the text of all the descendants (setting replaces the children with a text node); texts, comments, processing instructions: the data; document and doctype: `null`.
- `const(char)[] dataView`: The data of a text/comment without copying (`null` for other nodes).
- Navigation, each with a `Show` filter (default `Show.Element`):
  - `parent!show`: the parent if it matches (`html.parent` is invalid, `html.parent!(Show.All)` is the document);
  - `firstChild!show`, `lastChild!show`, `nextSibling!show`, `previousSibling!show`;
  - `children!show`, `descendants!show`: lazy bidirectional ranges (`retro` = reverse document order);
  - `walk!(show, descend)`: a lazy bidirectional range of steps (`s.node`, `s.enter`/`s.leave`) in tree order, entering and leaving each element (texts, comments, ... only enter; the node itself gives no steps). `descend(element)` (default: always true) is called on every element, also the hidden ones, after its enter step: false skips its children, the element still gives both steps. With `retro` a leave comes before its enter and `descend` is called after the leave step: with a predicate that depends on what the loop did, the two directions can give different steps.
- `bool hasChildNodes()`, `bool contains(Node other)` (true also for the node itself, as in the DOM).
- Search: `byId`, `byClass`, `byTagName`, `bySelector`, `byComment`, `byCommentExact` (inside this node).
- Mutations (`what` can be a `Node`/`Element`, a `string` added as text, `"<b>x</b>".asFragment` parsed as html in the context of the parent, a `DocumentFragment` node):
  - `append(what)`, `prepend(what)`: as last / first child;
  - `before(what)`, `after(what)`: as siblings;
  - `replaceWith(what)`;
  - `appendChild(Node)`, `node ~= what`;
  - `bool remove()`: detaches the node (it stays valid).
- `Node dup()` (deep), `Node shallowDup()`, `string toString()` (the html of the node), `string toPrettyString()` (indented, to read it).

### `Element` (a `Node` too)
- `string localName` ("div", "foreignObject"), `string tagName` ("DIV" for html elements), `localNameView`.
- `string id` (get/set), `idView`, `classes` (a range of `string`s), `classesView` (of slices).
- `auto attributes` (a range of `Element.Attribute {name, value}`), `attributesView` (`Element.AttributeView`).
- `string getAttribute(name)` (`null` if missing, `""` if empty), `getAttributeView(name)`, `setAttribute(name, value)`, `hasAttribute(name)`, `removeAttribute(name)`. Names are case-insensitive.
- `string innerHTML` (get/set; for `<template>` the content), `string outerHTML` (get/set, parsed in the context of the parent), `string innerText` (the same as `textContent`), `string startTag` (`<a href="x">`).
- `bool matches(css)`, `bool isEmpty` (as `:empty`: no element and no text children; comments don't count), `bool isBlank` (as `:blank`: whitespace text doesn't count either).
- `copyFrom(Element e)` (tag, attributes and children), `shallowCopyFrom(Element e)` (tag and attributes).
- `Element dup()`, `Element shallowDup()`.
- `el = "<b>html</b>";`: replaces the element with the (single) element parsed from the string.

### Parse errors
- `ParseOptions.collectErrors` (off by default, no cost): `doc.parseErrors` returns `ParseError[]` sorted by position; `ParseError` has `code` (`ParseErrorCode`), `line`, `column` (1-based), `name` (the code of the HTML standard, such as `"eof-in-tag"`; tree construction errors, not named by the standard, are `missing-doctype`, `unexpected-start-tag`, `unexpected-end-tag`, `misnested-tag`, ...) and `toString` (`"3:12: duplicate-attribute"`).
- `ctDocument!(html, options)` with `collectErrors: true` makes any parse error a compile error listing them.

### Encodings (`parserino.encoding`)
- The input of `Document` is UTF-8: a leading BOM is removed, invalid bytes become U+FFFD.
- `string toUtf8(const(ubyte)[] bytes, const(char)[] encoding = null, const(char)[] transportCharset = null)`: converts to UTF-8; without `encoding` it uses `sniffEncoding`.
- `string sniffEncoding(const(ubyte)[] bytes, const(char)[] transportCharset = null)`: BOM, then the HTTP charset, then `<meta charset>` in the first 1024 bytes, else UTF-8 if valid, else windows-1252.
- `string encodingForLabel(label)`: The canonical name of an encoding label (`"latin1"` is `"windows-1252"`), or `null` if not supported.
- Supported: UTF-8, UTF-16LE/BE, windows-1252 (labels latin1, iso-8859-1, us-ascii, ...), ISO-8859-2, windows-1250, windows-1251.

### `Show` (Enum, flags)
`Show.Element`, `Show.Text`, `Show.CDataSection`, `Show.ProcessingInstruction`, `Show.Comment`, `Show.Document`, `Show.DocumentType`, `Show.DocumentFragment`, `Show.All`.

### `Parsing` (Enum)
`Parsing.Eager`, `Parsing.Lazy`.

### `Selector`
A compiled CSS selector, reusable with any document.
- `this(string css)`: Compiles the selector (throws `ParserinoException` on invalid syntax).
- `bool isValid`.
- `ctSelector!css`: Compiled at compile time (syntax errors are compile errors).
- `bool isValidSelector(css)`: Checks the syntax without throwing; it works at compile time too.

```d
auto sel = Selector("div > a[href]");
foreach (doc; docs)
    foreach (a; doc.bySelector(sel))
        writeln(a.getAttribute("href"));
```

### `ParserinoException`
Thrown on invalid operations (invalid nodes, impossible mutations, invalid selectors, empty ranges in `frontOrThrow`).

## Range Helpers
- `frontOrThrow(range)`: Returns the first element or throws `ParserinoException` if empty.
- `frontOr(range, fallback)`: Returns the first element or the `fallback` element.
- `frontOrInit(range)`: Returns the first element or an invalid node (`== null`).

## Examples

### Navigation by node type
```d
import std, parserino;
void main() {
    Document doc = "<div><!-- note --><p>Hello <b>world</b></p>tail</div>";
    auto div = doc.byTagName("div").front;

    assert(div.children.walkLength == 1);                         // elements
    assert(div.children!(Show.All).walkLength == 3);              // comment, p, text
    assert(doc.descendants!(Show.Comment).front.textContent == " note ");
    assert(div.descendants!(Show.Text).map!(t => t.textContent).join == "Hello worldtail");
    assert(div.lastChild!(Show.All).isText);
    assert(div.descendants.retro.front.localName == "b");
}
```

### Walk: enter/leave steps, skipping subtrees
```d
import std, parserino;
void main() {
    Document doc = "<div><ul><li><b>x</b></li></ul></div>";

    // The predicate sees what the loop did at the enter step of the element
    int depth;
    foreach (s; doc.body.walk!(Show.Element, e => depth < 3))
    {
        if (s.enter) { depth++; writeln(' '.repeat(depth * 2), s.node.localName); }
        else depth--;
    }
    // div, ul, li are printed; the children of li are skipped
}
```

### Scraper (Lazy Processing): what's new in the latest D, from dlang.org
```d
import std.stdio;
import std.net.curl : get;
import std.range : take;
import std.string : strip;
import parserino;
void main() {
    // "Latest version: 2.113.0 – Changelog" on the home page
    auto home = Document(get("https://dlang.org/"), Parsing.Lazy);   // parse only what the queries need
    auto link = home.bySelector(".download .smallprint a").front.getAttribute("href");

    auto changelog = Document(get("https://dlang.org/" ~ link), Parsing.Lazy);
    writeln(changelog.title);

    // The first changes of each section (compiler, runtime, library, dub): the rest is never parsed
    foreach (section; changelog.byClass("bugsfixed").take(4))
    {
        writeln(section.byTagName("h4").front.textContent, ":");
        foreach (change; section.bySelector("li a").take(3))
            writeln(" - ", change.textContent.strip);
    }
    writeln(changelog.bytesParsed, " bytes parsed");
}
```

### DOM Manipulation
```d
import std.stdio, parserino;
void main() {
    Document doc = Document("<html><body><h1>Old Title</h1></body></html>");
    auto h1 = doc.byTagName("h1").frontOrThrow;
    h1.textContent = "New Title";

    auto p = doc.createElement("p");
    p.textContent = "A new paragraph.";
    p.setAttribute("class", "description");
    h1.after(p);
    doc.body.append("<footer>end</footer>".asFragment);

    writeln(doc.toString());
}
```

### Templates
```d
import parserino;
Document page() {
    auto snap = Document(`<ul><li class="item"></li></ul>`).snapshot;   // parse once
    auto doc = Document(snap);                                         // fast copies
    auto model = doc.byClass("item").front;
    foreach (name; ["a", "b"]) {
        auto row = model.dup;
        row.textContent = name;     // escaped automatically
        model.before(row);
    }
    model.remove();
    return doc;
}
```

## Notes on Performance
Parserino uses lazy ranges for traversal. This means searching through a large document often doesn't require visiting every node unless the entire range is consumed (e.g., using `.array`).
With `Parsing.Lazy`, even the parsing stops as soon as the range has what it needs: `doc.byTagName("a").take(3)` on a 1 MB page parses a few KB. Content inside tables that are still open is returned only when the table is closed, so table-based layouts parse further.