The spectral axis

A reading of the input at every byte position: the byte-class mix at five timescales, the entropy, the byte period and the novelty of the neighbourhood, and the offsets where that reading changes.

Source: src/spectral.rs.

The reader is one causal forward pass: the reading at byte t depends only on the bytes up to t, so a change in the input shows in the reading a few bytes after the bytes that cause it.

Reading the axis

$ trex spectral --text 'the quick brown fox jumps over the lazy dog'
trex spectral: 43 bytes, 3 frames (hop 16), 0 change-points
  entropy   min 0.62  mean 0.68  max 0.73  (normalized bits/byte)
  period    none detected (no strong byte-periodicity)
  texture timeline (merged regions, cut at change-points):
    [       0..43      ] prose  H=0.68 per=0 nov=1.00

use trex::spectral::{Texture, analyze, regions};

let field = analyze(b"the quick brown fox jumps over the lazy dog");
assert_eq!((field.len, field.hop, field.frames.len()), (43, 16, 3));
assert!(field.boundaries.is_empty());
assert_eq!(regions(&field), [(0, 43, Texture::Prose)]);

PS> Measure-TrexSpectral 'the quick brown fox jumps over the lazy dog' | Select-Object Length, Hop, ChangePoints, Timeline | Format-List

Length       : 43
Hop          : 16
ChangePoints : {}
Timeline     : {Prose at 0, 43 long}

In PowerShell -Classify adds the regions --code-classify prints, and -Detail adds every frame.

In a pattern

\F{pred} is a token atom: a token matches when the predicate holds of the pooled reading over its span.

A pattern naming the axis runs on the set engine, which builds the field once a scan with only the readings its atoms take: entropy the entropy, period the period, texture the class mix and the entropy, and onset those two and the change-points.

$ trex scan '\F{texture:code}+' --text 'the quick brown fox jumps; fn add(a,b){let c=a+b;return c*2;}'
[33..61] "(a,b){let c=a+b;return c*2;}"

let pat = trex::parse(r"\F{texture:code}+").expect("valid pattern");
let text = "the quick brown fox jumps; fn add(a,b){let c=a+b;return c*2;}";
let found: Vec<&str> = trex::scan(&pat, text.as_bytes()).iter().map(|s| &text[s.range()]).collect();
assert_eq!(found, ["(a,b){let c=a+b;return c*2;}"]);

>>> import trex
>>> [m.text for m in trex.Pattern(r"\F{texture:code}+").scan("the quick brown fox jumps; fn add(a,b){let c=a+b;return c*2;}")]
['(a,b){let c=a+b;return c*2;}']

PS> Select-TrexMatch '\F{texture:code}+' -InputObject 'the quick brown fox jumps; fn add(a,b){let c=a+b;return c*2;}' -Raw
(a,b){let c=a+b;return c*2;}

fn add is read as prose: the medium clock still holds the sentence before it.

In the lexer

The lexer collapses each whitespace-free run whose rolling entropy holds at or above 0.85 for at least 48 bytes (BLOB_ENTROPY_PCT, BLOB_MIN_LEN) into one opaque token, so base64, hex and packed data are not split into word and number tokens:

$ trex scan . --text 'key = aGVsbG8gd29ybGQgdGhpcyBpcyBiYXNlNjQgZGF0YSB0aGF0IGtlZXBzIGdvaW5nIGZvciBhIHdoaWxl end'
[0..3] "key"
[4..5] "="
[6..86] "aGVsbG8gd29ybGQgdGhpcyBpcyBiYXNlNjQgZGF0YSB0aGF0IGtlZXBzIGdvaW5nIGZvciBhIHdoaWxl"
[87..90] "end"

Prose and code stay below the gate. spectral::high_entropy_runs returns the runs.

Data model

SpectralFrame, one position’s reading:

SpectralField, keyed by byte offset:

analyze(input) computes every reading with the default configuration, analyze_with(input, cfg) with a SpectralConfig, and analyze_needing(input, cfg, needs) only the readings needs names. texture_of(&frame) classifies a frame as Prose, Code, Math, Data or Mixed, regions(&field) cuts the input at its change-points and merges adjacent spans of one texture, and code_regions(input) labels each span Code, Blob, Prose, Numeric or Mixed from a fresh pass over that span’s bytes.

Algorithms

Filterbank

Five leaky integrators per byte class, x[t] = a*x[t-1] + (1-a)*u[t], with u[t] one for the byte’s class and zero for the others. The decays a_d = exp(-1/tau_d) follow a power-of-two ladder of time constants:

Each band is a leaky average in [0,1]. The texture is read from the medium clock, d = 2.

Entropy, period and novelty

• Entropy: a 256-bin histogram over the sliding window, keeping s = sum c_i*log2(c_i) incrementally, so each byte is O(1); H = log2(n) - s/n, normalized by log2(min(W, 256)).
• Period: centered autocorrelation over the trailing window at lags 2..max_lag, the dominant period the arg-max with prominence, recomputed every period_hop bytes; the hop rises with the input so the total work stays under a fixed operation budget.
• Novelty: a rolling hash of the last k bytes with a count map over the window and a ring buffer for eviction; novelty = 1/(1 + count_before_increment).

Change-points

Per byte, d[t] is the distance between the two fastest clocks’ class mix and entropy at t and at t-1. An EWMA mean and mean absolute deviation of d give the threshold: a boundary is recorded where d[t] > mean + max(k*mad, f*mean) and at least cp_min_gap bytes have passed since the last.

The floor f acts where d is constant rather than noisy: a run of one byte class drives mad to zero and puts the threshold on the signal. Over six files of prose and source in both line-ending conventions (69,118 boundaries as CRLF, 74,841 as LF) and three padded binaries with constant runs of 5,140, 4,169 and 9,812 bytes, a floor anywhere from 0.03 to 0.10 moves no boundary. In those three binaries the run lies 26 kB, 49 kB and 380 kB in, and the reader marks its two ends and nothing between.

The floor acts where the input opens with the constant run. mean then converges onto the run’s own residual divergence, the spread collapses, and without the floor one ULP clears the threshold at byte 2218, for letters and zeros alike and at run lengths from 3,735 to 14,864; the hysteresis then holds through the run’s real end, so that boundary is lost too. In a corpus of 14,017 binaries three files open this way, all sparse instruction-set decode tables:

Each boundary the floor gains is the leading run’s length, where the zeros stop. The knee is 0.04 on all three files and on a synthetic constant run; the default is 0.05.

Cost