Can a small source sample choose typed columns, raw columns or the dictionary-free source route without encoding all full-file candidates? Each trial includes sample scanning, structural projection, typed-column preparation, pilot coding and exactly one full-file production route. Every resulting archive is decoded with source checksum verification and compared byte-for-byte. The selector uses no application/field names, input filenames or corpus labels. A pilot margin is a heuristic, not a bound on final compressed size.
V1 inspects a line-aligned 10% prefix capped at 16 KiB (whole sources below that cap). Native estimates use 4-way LZ plus actual flat/segregated Huffman frame costs; Hybrid uses Zstd L9. Alternatives require 3% sample savings. V2 uses 20% capped at 64 KiB, Native 8-way, Hybrid L1 and a 5% margin. Both fail on ordinary public logs.
A post-hoc threshold sweep on the development panel motivated a 10% margin and disabling Native sampling entirely. This calibration is explicitly not an independent validation result. The policy was frozen before evaluating 22 disjoint byte windows from the same sources. Those windows were already used for reference-dictionary training and are not new production systems or a fully untouched corpus. No further threshold tuning follows their results.
Three warmup rounds and five measured rounds per case, alternating control/selector first order. Aggregate times are sums of per-file medians. Exact-byte controls match the real 4.2.1 CLI archive fingerprints on all 30 original inputs × two engines. Timing includes the pilot; source I/O, inverse verification and fingerprint checks are outside encoding timers. No confidence interval or external-codec drift comparison is claimed.
| Policy / data | Engine | Files | Archive byte change | Encoding time change | Files that grow |
|---|---|---|---|---|---|
| V1: 16 KiB /10%, 3% margin | Native | 30 | +7.49% | +6.17% | 8/30 |
| V1: 16 KiB /10%, 3% margin | HybridZstd | 30 | +2.80% | +19.38% | 9/30 |
| V2: 64 KiB /20%, 5% margin | Native | 30 | +6.76% | +28.01% | 4/30 |
| V2: 64 KiB /20%, 5% margin | HybridZstd | 30 | -0.11% | -1.06% | 2/30 |
| Calibrated: Hybrid only, 10% margin | Native | 30 | +0.00% | +1.90% | 0/30 |
| Calibrated: Hybrid only, 10% margin | HybridZstd | 30 | -1.28% | +12.91% | 0/30 |
| Disjoint within-source windows | Native | 22 | +0.00% | -3.03% | 0/22 |
| Disjoint within-source windows | HybridZstd | 22 | -1.35% | +7.63% | 1/22 |
Native is unchanged by the calibrated policy; its small observed timing shifts are ordinary measurement variation, not a speed improvement. Hybrid calibrated routing saves about 1.3% aggregate bytes at roughly 8–13% more encoding time. One new within-source window still grows. This does not justify replacing the current production policy.
| Policy / data | Input | Engine | Chosen route | Control bytes | Selected bytes | Increase |
|---|---|---|---|---|---|---|
| V1: 16 KiB /10%, 3% margin | 02_loghub_hadoop.log | Native | Direct | 17,013 | 18,297 | +7.55% |
| V1: 16 KiB /10%, 3% margin | 04_loghub_zookeeper.log | HybridZstd | Untyped | 8,049 | 8,727 | +8.42% |
| V1: 16 KiB /10%, 3% margin | 05_loghub_openstack.log | HybridZstd | Direct | 39,096 | 39,960 | +2.21% |
| V1: 16 KiB /10%, 3% margin | 06_loghub_bgl.log | HybridZstd | Direct | 17,416 | 20,685 | +18.77% |
| V1: 16 KiB /10%, 3% margin | 07_loghub_hpc.log | HybridZstd | Direct | 19,351 | 28,766 | +48.65% |
| V1: 16 KiB /10%, 3% margin | 08_loghub_thunderbird.log | HybridZstd | Direct | 35,294 | 40,177 | +13.84% |
| V1: 16 KiB /10%, 3% margin | 09_loghub_windows.log | Native | Direct | 17,470 | 22,494 | +28.76% |
| V1: 16 KiB /10%, 3% margin | 10_loghub_linux.log | Native | Untyped | 31,830 | 37,911 | +19.10% |
| V1: 16 KiB /10%, 3% margin | 11_loghub_mac.log | HybridZstd | Direct | 20,605 | 27,263 | +32.31% |
| V1: 16 KiB /10%, 3% margin | 12_loghub_android.log | Native | Direct | 40,960 | 85,902 | +109.72% |
| V1: 16 KiB /10%, 3% margin | 12_loghub_android.log | HybridZstd | Direct | 38,974 | 45,722 | +17.31% |
| V1: 16 KiB /10%, 3% margin | 14_loghub_apache.log | Native | Untyped | 25,470 | 27,132 | +6.53% |
| V1: 16 KiB /10%, 3% margin | 14_loghub_apache.log | HybridZstd | Untyped | 23,973 | 24,080 | +0.45% |
| V1: 16 KiB /10%, 3% margin | 15_loghub_openssh.log | Native | Untyped | 16,327 | 19,304 | +18.23% |
| V1: 16 KiB /10%, 3% margin | 16_loghub_proxifier.log | Native | Direct | 25,976 | 28,558 | +9.94% |
| V1: 16 KiB /10%, 3% margin | 16_loghub_proxifier.log | HybridZstd | Direct | 24,182 | 24,953 | +3.19% |
| V1: 16 KiB /10%, 3% margin | 19_beats_postgresql_log.log | Native | Direct | 3,027 | 3,181 | +5.09% |
| V2: 64 KiB /20%, 5% margin | 07_loghub_hpc.log | HybridZstd | Direct | 19,351 | 28,766 | +48.65% |
| V2: 64 KiB /20%, 5% margin | 09_loghub_windows.log | Native | Untyped | 17,470 | 20,050 | +14.77% |
| V2: 64 KiB /20%, 5% margin | 10_loghub_linux.log | Native | Untyped | 31,830 | 37,911 | +19.10% |
| V2: 64 KiB /20%, 5% margin | 12_loghub_android.log | Native | Direct | 40,960 | 85,902 | +109.72% |
| V2: 64 KiB /20%, 5% margin | 12_loghub_android.log | HybridZstd | Untyped | 38,974 | 39,476 | +1.29% |
| V2: 64 KiB /20%, 5% margin | 15_loghub_openssh.log | Native | Untyped | 16,327 | 19,304 | +18.23% |
| Disjoint within-source windows | 04_loghub_zookeeper.log | HybridZstd | Untyped | 1,962 | 2,069 | +5.45% |
Android Native demonstrates the risk: both original pilots choose the source route, expanding 40,960 to 85,902 bytes. A sample may mis-rank model overhead, dictionary reuse and backend match opportunities that change with the full input length. This is an empirical limitation of the tested pilots; it is not a proof that every sampling method must fail. The calibrated margin hides the known failures but still selects a worse representation for the new ZooKeeper window (1,962->2,069B). A constant source byte entropy is not a guarantee of stable model rankings.
The accepted panels contain 3,584 exact/checksummed full-archive round trips, excluding extra preflights. Each source/configuration also retains stable archive identities across rounds. Every original-panel control fingerprint matches the real saved 4.2.1 CLI fingerprint. New within-source windows have pinned hashes and a verified production-control preflight; no previous archive fingerprint is claimed for those previously unevaluated byte ranges. No production source, archive grammar or Cargo version changed.
The first research executable was linked against experimental-selection artifacts from a same-name/version frozen build sharing the target directory. Its controls incorrectly matched the previous unconditional experiment, so those timings are rejected. Rebuilding the root library and relinking the example restored the correct fingerprints. The frozen preparation helper now gives experimental packages a distinct identity while sharing dependency caching. Discarded runs and the rejection reason remain beside the accepted results; they are not mixed into totals. Published release CLI/worker executables were not rebuilt by the example-only experiment.
Reject both original pilots and retain the calibrated policy as research only. Its mean density benefit is small, it still costs CPU, and the extra within-source check exposes a wrong decision. A future selector should use information already collected during full column-model preparation, account for model/dictionary amortization, or reuse prepared backend work; recompressing full candidates or trusting an unqualified short-prefix ranking is not a free improvement. These are next hypotheses, not implemented fixes. The current 4.2.1 decoder improvements remain the latest validated production change.
cargo build --release --offline --locked --example sampled_selector_probe
python3 tools/run_sampled_selector.py --output target/selector-reproduction
python3 tools/run_sampled_selector.py --output target/selector-window-reproduction --windows
python3 tools/render_sampled_selector.pyThe current example implements the final calibrated policy. probe_v1.rs, probe_v2.rs and probe_calibrated.rs freeze each source. initial.json, v2.json, calibrated_panel/results.json and disjoint_windows/results.json retain all accepted observations. Phase manifests identify binaries/source and checks; the window plan records the calibration boundary and provenance. discarded_cache_* and discarded_not_relinked.json are explicitly rejected. The frozen baseline includes expected actual CLI archive fingerprints, so a wrong cached control fails validation.