Scan input that arrives in pieces
Scan input that arrives in pieces
Scan a stream chunk by chunk and take each match as soon as no later chunk can change it, ending with the matches one scan over the whole stream would give. A chunk that ends inside a token holds that match until the token ends (streams).
Feed the chunks
The address split across the first two chunks, ann@y. and org, is reported once the second
chunk completes it:
$ printf 'mail bob@x.com\nand ann@y.org now\nthen ted@z.net\n' | trex scan '\E'
[5..14] "bob@x.com"
[19..28] "ann@y.org"
[38..47] "ted@z.net"
The command line reads its standard input this way, printing each match the moment it commits,
so tail -f app.log | trex scan '\E' reports an address as soon as its line arrives.
Check against one scan of the whole
The matches over the stream are those of one scan over all of it, whatever the chunk size:
let text = b"mail bob@x.com\nand ann@y.org now\nthen ted@z.net\n";
let pat = trex::parse(r"\E").expect("valid pattern");
for size in [1, 3, 7, 64] {
assert_eq!(trex::scan_chunked(&pat, text.chunks(size)), trex::scan(&pat, text));
}A pattern that cannot settle before the stream ends, such as one with a lookahead or a
whole-input axis, is scanned when the stream ends. A pattern set streams through one scanner,
StreamScanner(set) in Python and over_set in Rust.