The magnitude axis

The order of magnitude of each token: a number’s value, any other token’s byte length, on a log scale, with the change from the token before and the energy of the window behind it.

Source: src/magnitude.rs.

What it reads that nothing else does

timeout = 30 ; retries = 5 ; max_bytes = 5000000000 ; port = 8080

Every value here is a number. Shape reads one silhouette for all four, spectral one digit texture, and orbit folds none together. Magnitude reads them as m = 1.48, 0.70, 9.70, 3.91: max_bytes is six orders above its neighbours. The same reading finds an address among indices, a magic constant among small literals, a size field among flags, a price among quantities.

The readings

A jump is a token whose gradient reaches the jump threshold, three orders by default; an outlier is a token more than outlier_sigma standard deviations from the stream’s mean magnitude.

Reading the axis

$ trex magnitude --text 'retries = 3 ; max_bytes = 5000000000'
trex magnitude: 36 bytes, 7 tokens, total energy 112.2, 3 scale-jump(s)
  peak magnitude: 9.70 at '5000000000'

let field = trex::magnitude::analyze_bytes(b"retries = 3 ; max_bytes = 5000000000");
let peak = field.frames.iter().map(|f| f.magnitude).fold(0.0f32, f32::max);
assert!((peak - 9.70).abs() < 0.01);
assert_eq!((field.n_tokens, field.jumps.len()), (7, 3));

PS> Measure-TrexMagnitude 'retries = 3 ; max_bytes = 5000000000' | Select-Object Tokens, TotalEnergy, Peak, Jumps | Format-List

Tokens      : 7
TotalEnergy : 112.2273
Peak        : 5000000000 at 26
Jumps       : {max_bytes at 14, = at 24, 5000000000 at 26}

--field prints the per-token magnitude, gradient and energy, --jumps the scale discontinuities, --energy the heaviest points by local energy, and --outliers the tokens whose magnitude is a stream outlier. In PowerShell each frame carries Offset, Text, Magnitude, Gradient and Energy, and -Detail adds a frame at every token.

In a pattern

\M{>k} and \M{<k} hold where a token’s magnitude crosses k; \N{mag>k} is a number atom with a magnitude predicate, and the relative forms \N{>+1} read it against a context.

$ trex scan '\M{>5}' --text 'retries = 3 ; max_bytes = 5000000000'
[26..36] "5000000000"

let pat = trex::parse(r"\M{>5}").expect("valid pattern");
let text = b"retries = 3 ; max_bytes = 5000000000";
assert_eq!(trex::scan(&pat, text).iter().map(|s| &text[s.range()]).collect::<Vec<_>>(), [&b"5000000000"[..]]);

>>> import trex
>>> [m.text for m in trex.Pattern(r"\M{>5}").scan("retries = 3 ; max_bytes = 5000000000")]
['5000000000']

PS> Select-TrexMatch '\M{>5}' -InputObject 'retries = 3 ; max_bytes = 5000000000' -Raw
5000000000

Data model

MagnitudeFrame, one token’s reading:

MagnitudeField, keyed by byte offset:

Query API: magnitude_at(byte), gradient_at(byte), energy_of(start, end), outliers().

Algorithm

One pass over the token stream:

1. magnitude: a number’s value parsed (decimal, hex, scientific, underscores stripped) to log10(|value|), else log2(byte_len);
2. statistics: the stream’s mean and standard deviation of m, and total_energy, in the same scan;
3. gradient: m[i] - m[i-1], the first token 0;
4. jump: |gradient| >= jump_threshold records a discontinuity;
5. energy: local sum(m^2) over the trailing energy_window.

Cost