Skip to content
Docs
Reference
Pattern files

Pattern files

Pattern files

A pattern file declares the names a pattern reads as \{name}, states what they match, gives a built pattern its fields, and holds the rules a scan reports findings for. One declaration a line; blank lines and # comments are skipped. The command line reads one with --lib FILE (--rules FILE for its rules), Rust with ShapeSet::declare_text or declare_file, Python with lib= on Pattern and infer, and PowerShell with Import-TrexAtom, -Library or -RuleFile.

LineDeclares
shape NAME = BYTES``a token kind from a bounded byte-pattern, tried before the built-in recognizers
shape-after NAME = BYTES``the same, tried only where no built-in recognizer matched
kind NAME = PATTERNa token kind from a pattern: the tokens each match covers fuse into one token of the kind
let NAME = PATTERNa sub-pattern, inlined where \{NAME} appears
test NAME accepts "text"... rejects "text"...what a name matches
fields NAME {mark}...the fields of the sub-pattern NAME
rule NAME ...a named pattern with what a finding of it says

\{name} resolves to a built-in kind, then a name declared, then the shipped library, so a declaration shadows a library entry of its name and nothing may take a built-in’s. A later let of a name shadows an earlier one.

Shapes

A shape’s byte-pattern must have a fixed maximum length ({m,n} rather than +). A shape tried before the built-ins wins an overlap; shape-after leaves a built-in kind alone.

$ trex scan '\{ticket}' --shape 'ticket = `[A-Z]{2,4}-\d{1,4}`' --text 'see AB-12 and XYZ-9 now'
[4..9] "AB-12"
[14..19] "XYZ-9"

In PowerShell Register-TrexAtom -After declares a shape-after, and Get-TrexToken shows the token a shape makes.

Kinds and sub-patterns

A kind from a pattern runs after the lex: the tokens each match covers fuse into one token of the kind, brackets are paired again over what remains, and a later pattern reads the kind as one atom, a scan reporting it whole. A sub-pattern is inlined, so its bindings and quantifiers compose as if written in place.

$ cat defs.trex
let rhs = \N | \Q
kind assign = \W "=" \{rhs}
test assign accepts "x = 1" "name = \"bob\"" rejects "x == 1"
test rhs accepts "42" rejects "forty-two"
test iban accepts "GB82 WEST 1234 5698 7654 32" rejects "GB82WEST12345698765433"

$ trex scan '\{assign}' --lib defs.trex --text 'let x = 1; name = "bob"'
[4..9] "x = 1"
[11..23] "name = \"bob\""

A declared shape or kind decides the token boundaries every scan of a pattern read under it is made on, so a rewrite, a redaction, a table and a build under the same file lex the same way.

Tests

A test line states what a name matches. It accepts a text when the name’s match in the text is the whole of it, from the first significant token to the last, and rejects a text when the name matches nowhere in it. Either keyword takes any number of double-quoted texts, which read \", \\, \n and \t. The name is any declared shape, kind or sub-pattern, or a library entry, tested as the file finally declares it, so a test may stand above the line it checks.

$ cat wrong.trex
let rhs = \N
test rhs accepts "42" "\"bob\"" rejects "4 2"

$ trex lib --test defs.trex
defs.trex: 3 tests passed

$ trex lib --test wrong.trex
wrong.trex:2: rhs accepts "\"bob\"": no match
  tokens: quoted "\"bob\""
wrong.trex:2: rhs rejects "4 2": matched "4" at 0..1
wrong.trex: 1 of 1 test failed

lib --test prints each expectation not met as a FILE:LINE: line and exits 1 when any test fails; given several files, they declare into one set in order, so a later file’s tests may name an earlier file’s declarations. Test-TrexAtom -Quiet writes one boolean, and -Shipped tests the shipped library.

Fields

A fields NAME {mark}... line, below the let declaring NAME, gives that sub-pattern’s fields in order, each written as a ConvertFrom-String mark with the example text left out: {[int]os} casts the field to the type it names, {Name*} begins a record, {Person.Name} is a field inside another, and {host:host} reads the field through an accessor, as ${host:host} does in a template. A field the sub-pattern binds no register under, a field named twice, an accessor that is not one and a [type] no mark knows are refused.

let extract = ^ "GET" (\U):host (\N):code ~<($ .)
fields extract {host:host} {[int]code}

A build writes the line for the pattern it builds, and scan --fields, Python’s Pattern.records, PowerShell’s ConvertFrom-TrexText and Rust’s infer::build::fields_for read \{NAME} through it; saving and reusing a build shows each.

Rules

A rule is a named pattern with what a finding of it says. As a block, rule NAME over indented field = value lines; or as one line, rule NAME [error|warning|note] "message" = PATTERN, with fix NAME = TEMPLATE, meta NAME KEY = VALUE, files NAME = GLOBS, unless NAME = PATTERN, record NAME = UNIT, record-start NAME = PATTERN and record-span NAME = PATTERN lines below it.

FieldHolds
patternthe pattern a finding is a match of
messagea report template rendered at each finding: the match’s registers and their typed slices, ${path}, ${line}, ${col}, ${rule} and ${severity}
severityerror, warning or note; a warning where it says nothing
fixa rewrite template rendered in the match’s place
filesglobs the inputs it reads must pass, as -g reads them; a rule with globs reads no unnamed input
unlessa pattern the record must not hold
record, record-start, record-spanwhat a record is, as --record and its two companions take it
meta.KEYfree metadata a pipeline reads; meta.tags is split into tags

A rule naming unless or a record fires on a record holding its pattern and none of the unless patterns, a line where it names no record; its finding covers the record, and the message and fix read the rule’s first match in it. The rule is also a sub-pattern under its name, so a later pattern reads it as \{name} and a test line checks it.

$ cat rules.trex
# what a config file may not hold
rule private_ip
  pattern = \{ip_private}:addr
  message = private address ${addr} in ${path:name}
  severity = warning
  fix = ${addr:octet1-2}.x.x
  meta.cwe = CWE-200
  meta.tags = network, config

rule cardnum error "card number ending ${card:last4}" = \{card}:card
fix cardnum = ****
meta cardnum tags = pii
rule todo note "a TODO left in ${path:name}" = "TODO"
test private_ip accepts "10.0.0.5" rejects "8.8.8.8"
$ cat app.conf
host = 10.0.0.5
pay 4111 1111 1111 1111 now
# TODO rotate

$ trex scan --rules rules.trex app.conf
app.conf:1:8: warning: private address 10.0.0.5 in app.conf [private_ip]
  fix: "10.0.x.x"
app.conf:2:5: error: card number ending 1111 [cardnum]
  fix: "****"
app.conf:3:3: note: a TODO left in app.conf [todo]

$ trex scan --rules rules.trex --format '${severity} ${rule} ${line}:${col} ${message}' app.conf
warning private_ip 1:8 private address 10.0.0.5 in app.conf
error cardnum 2:5 card number ending 1111
note todo 3:3 a TODO left in app.conf

A finding of an error rule fails the run. --rules DIR scans every .trex file under a directory; the lint with rules how-to shows the reports a CI system reads, --sarif and --github, and the fixes. The rules that fire on each match scan as one set per list of files; a rule on records scans on its own.

$ cat vault.trex
rule unguarded
  pattern = "password"
  unless = "vault"
  record = paragraph
  message = a password outside the vault
  severity = error
$ cat notes.txt
password = x
vault: ok

password = y
plain

$ trex scan --rules vault.trex notes.txt
notes.txt:4:1: error: a password outside the vault [unguarded]

Libraries

In PowerShell the session holds the atoms Register-TrexAtom and Import-TrexAtom declare, in $TrexSession; a Trex.Library from New-TrexLibrary holds a set of its own, from pattern files, declaration lines or a copy of the session’s, and a cmdlet given -Library reads it in their place. In Rust a ShapeSet is that set, handed to the parser and the scan; in Python a Pattern reads the files lib= names.

let mut lib = trex::ShapeSet::new();
lib.declare_text("let big = \\N{>1000}\n").expect("a pattern file");
let pat = trex::parser::parse_with_shapes(r"\{big}", &lib).expect("valid pattern");
let text = b"sizes 5 and 5000";
assert_eq!(trex::scan_with_shapes(&pat, text, &lib).iter().map(|s| &text[s.range()]).collect::<Vec<_>>(), [&b"5000"[..]]);

The shipped library

The shipped library is seventy-five entries every pattern reads with no declaration. Twenty-one are kinds with a bounded shape, twelve of them checked as they are lexed: eleven by their standard’s checksum, \{iban} (mod 97-10 and the registry’s length per country), \{isbn} (ISBN-10 mod 11 or ISBN-13), \{vin}, \{isin}, \{ean13}, \{upca}, \{ean8}, \{imei}, \{ethaddr} (EIP-55 case over Keccak-256), \{btcaddr} (base58check, bech32 and bech32m) and \{github_token} (the CRC-32 its last six characters carry over the thirty before them, for the gh?_ forms; a fine-grained github_pat_ token passes on its shape), and \{k8s_name} by its length, at most sixty-three characters; by shape alone \{awskey}, \{slack_token}, \{google_key}, \{stripe_key}, \{twilio_key}, \{cve}, \{mime}, \{docker_image} and \{git_sha}. Sixteen are sub-patterns over tokens: \{private_key} (a PEM block from BEGIN to END), \{ip_private}, \{ip_loopback} and \{ip_linklocal} through \I{in:...}, \{log_level}, \{http_method}, \{http_status} and \{http_1xx} to \{http_5xx} through \N{a..b}, \{currency}, \{country}, \{weekday} and \{month}. Sixteen are the unit kinds a quantity reads from the context: \{kelvin}, \{inch}, \{meter}, \{second}, \{hour}, \{gram}, \{tonne}, \{ampere}, \{volt}, \{watt}, \{newton}, \{joule}, \{calorie}, \{liter}, \{gallon} and \{bar}. The last twenty-two are the cues those read, a word list and a sub-pattern for each of eleven families, from \{temperature_word} and \{temperature_cue} to \{pressure_word} and \{pressure_cue}. A library kind is lexed only for a pattern that names it: thirteen digits with a valid check are a number to \N and an EAN-13 to \{ean13}.

$ trex lib | head -4
name           form     guard    what
iban           kind     checked  an IBAN, compact or in groups of four, with its country's length and the mod 97-10 check
isbn           kind     checked  an ISBN-10 (mod 11, X as ten) or ISBN-13 (978 or 979, EAN check), hyphens or spaces allowed
vin            kind     checked  a vehicle identification number: seventeen characters without I, O or Q and the check digit ninth

$ trex scan '\{iban}' --text 'pay GB82 WEST 1234 5698 7654 32 or DE89370400440532013000; not GB82WEST12345698765433'
[4..31] "GB82 WEST 1234 5698 7654 32"
[35..57] "DE89370400440532013000"
Building patternsTools