This patch integrates the measured dictionary and integer arenas into the production decoder. Mode9 dictionaries reconstruct prefix-coded entries in one checked buffer plus ranges and existing row IDs. If expanded dictionary storage exceeds the ordinary output budget because of unused entries, the previous owned dictionary path remains available for archive compatibility. All dictionary prefixes, suffix separators, indices and referenced output lengths are validated.
Mode6 numeric columns reuse one arena for existing integer-delta and integer-FoR streams. Signed canonical decimal bytes are generated in a stack buffer; wrapping deltas/strides, canonical varints, exact input consumption and record/output budgets remain enforced. Timestamp, prefix and other Novel models retain their prior paths. The archive writer, routing policies, entropy coders and checksum choices are unchanged.
Typical reconstruction costs O(dictionary decoded bytes + records) time/storage for dictionaries and O(decimal output bytes + records) for integers, apart from existing FoR unpacking. Memory usage has not been benchmarked; fewer allocations do not establish a lower peak RSS. No zero-copy or universal speed claim is made.
Three alternating frozen 4.2.0 /4.2.1 panel pairs, each with three warmups and three rotated measured rounds over 30 inputs and 18 codec configurations. Aggregate times are medians of the three per-panel sums. These are full verified decode timings, including source checksum and production archive validation. Inputs are the same bounded development excerpts used previously, not held out. No confidence intervals are claimed.
| Configuration | Archive bytes (both versions) | Encode time change | Verified decode ms old → new | Decode time change |
|---|---|---|---|---|
| PIZD_hybrid_default | 997,105 | -0.21% | 13.456 → 12.698 | -5.63% |
| PIZD_hybrid_fast | 973,161 | -2.18% | 16.504 → 16.266 | -1.44% |
| PIZD_hybrid_thin_L9 | 756,241 | +0.97% | 33.259 → 25.138 | -24.42% |
| PIZD_native_default | 1,353,762 | -2.18% | 22.677 → 21.143 | -6.76% |
| PIZD_native_fast | 1,374,345 | -0.06% | 25.543 → 25.492 | -0.20% |
| PIZD_native_thin_L0 | 826,197 | +2.42% | 38.286 → 30.723 | -19.76% |
Encoding was not changed; its observed shifts are retained as timing context, not intentional compression-speed gains. Reference-codec shifts disclose session drift:
| Reference | Encode time change | Verified decode time change |
|---|---|---|
| Zstd_L3 | +1.68% | -4.32% |
| Zstd_L9 | +0.31% | -8.31% |
| Gzip_L6 | -0.57% | -1.01% |
| Brotli_Q9 | +0.07% | -0.94% |
Archive bytes do not change. Negative time changes indicate faster decoding. All inputs, including weak gains and slower observations, are retained.
| Input | Engine | Old verified ms | New verified ms | Time change |
|---|---|---|---|---|
| 01_loghub_hdfs.log | PIZD_hybrid_thin_L9 | 2.565 | 1.428 | -44.34% |
| 01_loghub_hdfs.log | PIZD_native_thin_L0 | 2.919 | 1.735 | -40.57% |
| 02_loghub_hadoop.log | PIZD_hybrid_thin_L9 | 0.722 | 0.591 | -18.16% |
| 02_loghub_hadoop.log | PIZD_native_thin_L0 | 0.868 | 0.634 | -26.92% |
| 03_loghub_spark.log | PIZD_hybrid_thin_L9 | 1.758 | 1.307 | -25.62% |
| 03_loghub_spark.log | PIZD_native_thin_L0 | 1.983 | 1.494 | -24.64% |
| 04_loghub_zookeeper.log | PIZD_hybrid_thin_L9 | 1.226 | 1.053 | -14.11% |
| 04_loghub_zookeeper.log | PIZD_native_thin_L0 | 1.350 | 1.175 | -12.91% |
| 05_loghub_openstack.log | PIZD_hybrid_thin_L9 | 1.089 | 0.816 | -25.00% |
| 05_loghub_openstack.log | PIZD_native_thin_L0 | 1.379 | 1.298 | -5.87% |
| 06_loghub_bgl.log | PIZD_hybrid_thin_L9 | 1.582 | 1.103 | -30.28% |
| 06_loghub_bgl.log | PIZD_native_thin_L0 | 1.479 | 1.019 | -31.06% |
| 07_loghub_hpc.log | PIZD_hybrid_thin_L9 | 2.055 | 1.551 | -24.53% |
| 07_loghub_hpc.log | PIZD_native_thin_L0 | 2.146 | 1.711 | -20.28% |
| 08_loghub_thunderbird.log | PIZD_hybrid_thin_L9 | 1.729 | 1.379 | -20.29% |
| 08_loghub_thunderbird.log | PIZD_native_thin_L0 | 2.111 | 1.474 | -30.16% |
| 09_loghub_windows.log | PIZD_hybrid_thin_L9 | 1.506 | 1.183 | -21.47% |
| 09_loghub_windows.log | PIZD_native_thin_L0 | 1.583 | 1.253 | -20.87% |
| 10_loghub_linux.log | PIZD_hybrid_thin_L9 | 1.679 | 0.934 | -44.35% |
| 10_loghub_linux.log | PIZD_native_thin_L0 | 1.953 | 1.173 | -39.95% |
| 11_loghub_mac.log | PIZD_hybrid_thin_L9 | 1.334 | 1.016 | -23.82% |
| 11_loghub_mac.log | PIZD_native_thin_L0 | 1.534 | 1.208 | -21.25% |
| 12_loghub_android.log | PIZD_hybrid_thin_L9 | 1.827 | 1.545 | -15.43% |
| 12_loghub_android.log | PIZD_native_thin_L0 | 2.154 | 1.879 | -12.75% |
| 13_loghub_healthapp.log | PIZD_hybrid_thin_L9 | 0.599 | 0.610 | +1.84% |
| 13_loghub_healthapp.log | PIZD_native_thin_L0 | 0.963 | 0.976 | +1.33% |
| 14_loghub_apache.log | PIZD_hybrid_thin_L9 | 2.426 | 1.692 | -30.26% |
| 14_loghub_apache.log | PIZD_native_thin_L0 | 2.565 | 2.152 | -16.08% |
| 15_loghub_openssh.log | PIZD_hybrid_thin_L9 | 1.195 | 0.855 | -28.47% |
| 15_loghub_openssh.log | PIZD_native_thin_L0 | 1.421 | 1.041 | -26.73% |
| 16_loghub_proxifier.log | PIZD_hybrid_thin_L9 | 1.392 | 0.968 | -30.47% |
| 16_loghub_proxifier.log | PIZD_native_thin_L0 | 1.652 | 1.177 | -28.73% |
| 17_beats_nginx_access.log | PIZD_hybrid_thin_L9 | 0.027 | 0.024 | -11.18% |
| 17_beats_nginx_access.log | PIZD_native_thin_L0 | 0.030 | 0.028 | -8.72% |
| 18_beats_nginx_error.log | PIZD_hybrid_thin_L9 | 0.009 | 0.008 | -10.20% |
| 18_beats_nginx_error.log | PIZD_native_thin_L0 | 0.013 | 0.013 | +0.08% |
| 19_beats_postgresql_log.log | PIZD_hybrid_thin_L9 | 0.188 | 0.129 | -31.30% |
| 19_beats_postgresql_log.log | PIZD_native_thin_L0 | 0.178 | 0.119 | -33.25% |
| 20_beats_mysql_slowlog.log | PIZD_hybrid_thin_L9 | 0.060 | 0.059 | -2.25% |
| 20_beats_mysql_slowlog.log | PIZD_native_thin_L0 | 0.083 | 0.082 | -0.29% |
| 21_beats_mongodb_log.log | PIZD_hybrid_thin_L9 | 0.028 | 0.028 | +1.15% |
| 21_beats_mongodb_log.log | PIZD_native_thin_L0 | 0.055 | 0.052 | -5.90% |
| 22_beats_redis_log.log | PIZD_hybrid_thin_L9 | 2.458 | 1.641 | -33.26% |
| 22_beats_redis_log.log | PIZD_native_thin_L0 | 2.455 | 1.655 | -32.60% |
| 23_beats_elasticsearch_audit.log | PIZD_hybrid_thin_L9 | 0.016 | 0.015 | -2.25% |
| 23_beats_elasticsearch_audit.log | PIZD_native_thin_L0 | 0.047 | 0.044 | -6.46% |
| 24_beats_traefik_access.log | PIZD_hybrid_thin_L9 | 0.010 | 0.009 | -5.12% |
| 24_beats_traefik_access.log | PIZD_native_thin_L0 | 0.016 | 0.017 | +3.86% |
| 25_nasa_http_jul95.log | PIZD_hybrid_thin_L9 | 1.601 | 1.343 | -16.08% |
| 25_nasa_http_jul95.log | PIZD_native_thin_L0 | 1.844 | 1.511 | -18.03% |
| 26_nasa_http_aug95.log | PIZD_hybrid_thin_L9 | 1.499 | 1.242 | -17.10% |
| 26_nasa_http_aug95.log | PIZD_native_thin_L0 | 1.777 | 1.395 | -21.50% |
| 27_github_events_20150101_00.jsonl | PIZD_hybrid_thin_L9 | 0.649 | 0.618 | -4.87% |
| 27_github_events_20150101_00.jsonl | PIZD_native_thin_L0 | 1.197 | 1.188 | -0.80% |
| 28_github_events_20150101_06.jsonl | PIZD_hybrid_thin_L9 | 0.569 | 0.609 | +6.94% |
| 28_github_events_20150101_06.jsonl | PIZD_native_thin_L0 | 1.163 | 1.168 | +0.43% |
| 29_github_events_20150101_12.jsonl | PIZD_hybrid_thin_L9 | 0.623 | 0.568 | -8.82% |
| 29_github_events_20150101_12.jsonl | PIZD_native_thin_L0 | 1.048 | 1.005 | -4.12% |
| 30_github_events_20150101_18.jsonl | PIZD_hybrid_thin_L9 | 0.724 | 0.878 | +21.28% |
| 30_github_events_20150101_18.jsonl | PIZD_native_thin_L0 | 1.144 | 1.156 | +1.01% |
All 251 existing tests and five new arena tests pass (256 total). The new cases cover signed extremes and wrapping, zero-/64-bit FoR, precise output budgets, malformed/truncated/noncanonical streams, strict capability enforcement, dictionary indices/prefixes/separators, empty values, unused oversized entries and dependent-column access. The paired measurement retains 19,440 archive emissions, 38,880 exact restores and 19,440 independent SHA-256 checks. All 540 source/configuration archive identities agree across binaries, including all 180 PIZD configurations. The actual CLI generates 180 archives byte-identical to saved 4.2.0 output; each is checked with the previous reader, and every old archive is restored with the new reader. This compatibility result is separate from the benchmark worker.
This is a decoder-speed correction, not an improvement in compression density. The retained Spark/MongoDB size regressions and four GH Archive JSON losses from 4.2.0 remain. Zstandard can still decode faster; use the complete reference tables rather than inferring a universal ranking. Row assembly and noninteger column models remain CPU costs. The small parser fixtures and historical HTTP windows are labeled separately from LogHub and JSON inputs.
cargo test --offline --locked --lib --tests
cargo build --release --offline --locked --bin pizd --example public_log_panel
python3 tools/run_public_log_panel.py --workspace-build
python3 tools/benchmark_decoder_arenas.py
python3 tools/validate_decoder_arenas.py
python3 tools/render_decoder_arenas.pyThe public runner requires the pinned corpus manifest and cached verified inputs. The paired script requires the retained 4.2.0 control worker. paired_manifest.json, paired_summary.json, paired_per_file.json, paired_validation.json, six raw CSVs and compressed per-round logs retain the comparison. cli_validation.json records real archive/source identities. The current public report retains the source snapshot and build/input hashes. Frozen4.2.0 reports and research prototypes remain separately labeled.