Streams, grammars, subwords and prefilters

A scan of input that arrives in pieces, token grammars (trex grammar), byte-pair subword encoders (trex bpe) and presence filters (trex prefilter), each an object with methods as well as a set of cmdlets. The examples are on tools.

New-TrexStreamScanner

Begins a scan of input that arrives in pieces: a Trex.StreamScanner whose Push method takes each chunk and gives back the matches no later chunk can change, and whose Finish method gives back the rest.

The matches over the whole stream are the ones Select-TrexMatch finds in it at once, each at its offset into the whole stream in UTF-16 code units, and the scanner holds only the part of the stream a match may still reach. Several patterns scan as one set, each match naming its pattern. The patterns are compiled with the atoms trex ships, since the stream is lexed as it arrives, with no declarations in force.

Alias: New-TxStreamScanner

New-TrexStreamScanner [-Pattern] <string[]> [<CommonParameters>]

Writes Trex.StreamScanner.

New-TrexGrammar

Compiles a token grammar for Invoke-TrexGrammar and as an object with Parse, Test, CountParses, BestProbability, TotalProbability, CountSegmentations and BestSegmentationProbability methods.

A grammar is one rule a line, name := alternatives, alternatives separated by |: <rule> another rule, "lit" a token by its text, number, ident, string, ip, url, email, time or punct a token by its kind, *, + and ? after a symbol its repetition, ( a | b ) a group, and @p after an alternative its weight, 1 when absent. # starts a comment. A rule that begins with itself is left recursive, which is how an operator grammar writes precedence. The first rule is the start unless -Start names another.

Alias: New-TxGrammar

New-TrexGrammar [-Source] <string> [-Start <string>] [<CommonParameters>]
New-TrexGrammar -Path <string> [-Start <string>] [<CommonParameters>]

Writes Trex.Grammar.

Invoke-TrexGrammar

Parses text against a token grammar and writes the parse tree, or with -Count, -Best or -Probability a value over every derivation.

The parse is one ordered-choice parse from the start rule, written as a Trex.ParseNode; text that does not parse in full is an error record. -Count writes how many derivations the grammar has of the text, which says how ambiguous it is; -Best the probability of the most probable one under the grammar’s @p weights, which resolves the ambiguity; and -Probability all of them together. -Segment reads a run-together string over the words of -Dictionary, every way it tiles into them, and writes how many tilings the grammar accepts, or with -Best the probability of the most probable one.

Each input gets one answer, in the order read. The count and the probabilities come from a chart cubic in the token count, so a long text is parsed a sentence at a time.

Alias: Invoke-TxGrammar

Invoke-TrexGrammar [-Grammar] <Object> [-InputObject] <string> [-Start <string>] [-Count] [-Best] [-Probability] [<CommonParameters>]
Invoke-TrexGrammar [-Grammar] <Object> -Path <string[]> [-Start <string>] [-Count] [-Best] [-Probability] [<CommonParameters>]
Invoke-TrexGrammar [-Grammar] <Object> -LiteralPath <string[]> [-Start <string>] [-Count] [-Best] [-Probability] [<CommonParameters>]
Invoke-TrexGrammar [-Grammar] <Object> -Segment <string> -Dictionary <string[]> [-Start <string>] [-Count] [-Best] [<CommonParameters>]

Writes Trex.ParseNode, ulong, double.

New-TrexBpe

Learns a byte-pair encoder from a corpus.

Each word of the corpus is its characters and an end-of-word mark, and every round merges the most frequent adjacent pair of symbols, ties broken by the pair’s text, so the same corpus always learns the same model. The files named, or the strings piped in, are read as one corpus. -MaxBytes trains on the start of a large corpus, cut back to a whole word.

Alias: New-TxBpe

New-TrexBpe [-Path] <string[]> [-Merges <uint>] [-MaxBytes <ulong>] [<CommonParameters>]
New-TrexBpe -LiteralPath <string[]> [-Merges <uint>] [-MaxBytes <ulong>] [<CommonParameters>]
New-TrexBpe -InputObject <string> [-Merges <uint>] [-MaxBytes <ulong>] [<CommonParameters>]

Writes Trex.Bpe.

Import-TrexBpe

Reads a byte-pair encoder from a model file, one Export-TrexBpe or the trex command’s bpe train wrote.

A model line is two symbols separated by a tab, and a line that is not is an error naming its number rather than a merge left out.

Alias: Import-TxBpe

Import-TrexBpe [-Path] <string[]> [<CommonParameters>]

Writes Trex.Bpe.

Export-TrexBpe

Writes a byte-pair encoder’s model to a file, one merge a line with its two symbols separated by a tab, which Import-TrexBpe and the trex command’s bpe encode --model read.

Alias: Export-TxBpe

Export-TrexBpe [-Path] <string> -Bpe <Bpe> [-WhatIf] [-Confirm] [<CommonParameters>]

ConvertTo-TrexBpe

Splits text into a byte-pair encoder’s subwords, writing each subword.

Each word is split on its own, the last subword of a word ending in </w>, so joining a word’s subwords and dropping the mark gives the word back.

Alias: ConvertTo-TxBpe

ConvertTo-TrexBpe [-Bpe] <Bpe> [-InputObject] <string> [<CommonParameters>]
ConvertTo-TrexBpe [-Bpe] <Bpe> -Path <string[]> [<CommonParameters>]
ConvertTo-TrexBpe [-Bpe] <Bpe> -LiteralPath <string[]> [<CommonParameters>]

Writes string.

New-TrexPrefilter

Builds a presence filter over a corpus, as an object whose MightContain method answers whether a literal might occur in it.

The files named, or the strings piped in, are read as one corpus. The filter keeps no copy of it, so the object leaves a yes unsettled, where Test-TrexPrefilter settles each with an exact search.

Alias: New-TxPrefilter

New-TrexPrefilter [-Path] <string[]> [-Filter <FilterKind>] [<CommonParameters>]
New-TrexPrefilter -LiteralPath <string[]> [-Filter <FilterKind>] [<CommonParameters>]
New-TrexPrefilter -InputObject <string> [-Filter <FilterKind>] [<CommonParameters>]

Writes Trex.Prefilter.

Test-TrexPrefilter

Tests literals against a presence filter built over a corpus, or with -Verify every filter’s contract over it.

Each literal gets a Trex.LiteralTest: whether the filter says it might occur, and whether an exact search finds it, so a false positive reads as one. -Verify probes all three filters with substrings of the corpus, which each must report as present, and with byte strings found nowhere in it, the share of which a filter rejects being the scanning it saves; it writes a Trex.FilterCheck for each.

Alias: Test-TxPrefilter

Test-TrexPrefilter [-Path] <string[]> [-Literal <string[]>] [-Filter <FilterKind>] [-Verify] [<CommonParameters>]
Test-TrexPrefilter -LiteralPath <string[]> [-Literal <string[]>] [-Filter <FilterKind>] [-Verify] [<CommonParameters>]
Test-TrexPrefilter -InputObject <string> [-Literal <string[]>] [-Filter <FilterKind>] [-Verify] [<CommonParameters>]

Writes Trex.LiteralTest, Trex.FilterCheck.