# Adaptive modeling on actual rc8 Native channels

This investigation starts from the selected complete rc8 Native archive, not a weaker standalone LZ77 or Huffman baseline. It focuses on logs, JSON/NDJSON and structured controls. All framing, tables and source checksums are charged. Source SHA-256 and exact byte restores are mandatory.

## What is measured

| Stage | Public30 total bytes | Reduction versus actual rc8 | Status |
|---|---:|---:|---|
| Frozen rc8 Native thin + template factorization | 687,535 | Baseline | Existing production reader |
| Sparse copy-source retargeting / short-copy removal | 684,875 | 0.387% | Existing-format .pizd; research search policy |
| Sparse + literal and instruction-channel PPM portfolio | 656,103 | 4.572% | Research wrappers; direct integration is separately measured below |

The research portfolio improves 21/30 inputs. Its full model competition is paid in CPU; the incumbent is retained whenever no complete candidate shrinks. The policy frozen before the later-window run improves 18/22 additional windows: 204,816 -> 194,363 bytes (5.104%). These later windows are disjoint in bytes but come from the same families and were previously available for dictionary studies. They are not project-blind validation.

## Where the gain comes from

1. Recover the real selected token stream, including keyword substitution, four literal contexts, repeat distances and logarithmic slots. Canonical reserialization must reproduce the actual archive byte for byte before any search.
2. At existing copy boundaries only, search equivalent references with a small beam and optionally literalize copies of at most16 bytes. Exact source equality, including overlapping copies, validates every replacement. This avoids the expensive all-byte candidate graph, but the measured size gain is modest.
3. Apply PPM to the actual literal sequence and independently to run-length/distance/length instruction channels. The extra-bit reservoirs often fail to repay another model. Preserve their incumbent bytes rather than assuming every channel benefits.
4. Compare complete costs. Per-channel headers, arithmetic termination, model metadata and nested numeric framing count against the candidate. PPM order1/2 generally works better for literals here than indiscriminately raising the order.

The source model and LZ77 grammar remain unchanged. This is model placement and cost arbitration engineering, not a claim to have invented PPM or arithmetic coding.

## Large input examples (research portfolio)

| Input | Actual rc8 bytes | Research bytes | Reduction |
|---|---:|---:|---:|
| nasa_large.log | 237,207 | 212,431 | 10.445% |
| hdfs_large.log | 177,553 | 167,367 | 5.737% |
| github_large.jsonl | 468,503 | 432,325 | 7.722% |

## Instruction-channel autopsy: large GitHub NDJSON

These are body costs in the selected literal research parent. They do not include a hypothetical future transform. The complete research archive totals above include their actual wrapper costs. Packed extra-bit reservoirs retain their incumbents when all tested models lose.

| Channel | Raw bytes | Old body bytes | Best PPM body bytes | Model order / escape |
|---|---:|---:|---:|---|
| Literal-run lengths | 103,045 | 28,497 | 26,191 | 1 / 0 |
| Distance slots | 103,045 | 78,081 | 72,015 | 2 / 1 |
| Distance extra bits | 113,150 | 113,150 | 116,676 | 1 / 1 |
| Match-length slots | 103,045 | 55,543 | 43,803 | 4 / 1 |
| Match-length extra bits | 18,208 | 18,208 | 18,919 | 2 / 1 |

## Direct isolated CLI candidate

The candidate is built in `target/adaptive-runtime-candidate/`; the main runtime remains rc8 until peer review and merge. This candidate does not include sparse parsing. It recodes the final selected zero-pattern Native payload once, including one nested numeric wrapper and the separate composed-template reader, only for standalone Native thin inputs from 512 bytes through 8 MiB without a local/remote dictionary or explicit rANS4. The adaptive postpass leaves other routes untouched; the keyword tie fix applies wherever keyword extraction is already used.

The candidate also fixes existing keyword-score tie nondeterminism by adding a lexical tie break. The frozen stock rc8 retains its original randomized equal-score ordering. Every CLI trial records its size and archive hash; public sizes are medians across three trials. Consequently this CLI comparison includes the keyword-order fix as well as adaptive modeling. The exact-parent experiment below isolates adaptive modeling alone.

The payload capability is explicitly `PZSG\x03`; adaptive channels use mode3 and self-describing `PZPPM001` frames. Old readers reject a new capability. The candidate reader also reads old archives. Original BLAKE3/SHA-256 source checksums remain unchanged.

```sh
./target/adaptive-build/release/pizd compress INPUT --offline --standalone --thin --template-factorization --engine native -o OUTPUT.pizd
./target/adaptive-build/release/pizd decompress OUTPUT.pizd --offline -o RESTORED
```

Use the candidate reader for its new archives; the main rc8 binary cannot read the new payload capability. The archive is standalone and contains all model parameters.

| Cohort | Sources | rc8 bytes | Candidate bytes | Reduction | Wins |
|---|---:|---:|---:|---:|---:|
| Public excerpts | 30 | 687,537 | 657,356 | 4.390% | 21 |
| Size/type controls | 24 | 7,056,286 | 6,983,023 | 1.038% | 13 |
| Later disjoint windows | 22 | 204,816 | 193,571 | 5.490% | 18 |

Public30 uses three alternating measured trials per build; remaining inputs use one diagnostic trial. Timings are process wall times including startup and file I/O, not pure codec throughput. Sum of per-input public median compression times: 28793.56 -> 32408.01 ms (1.126x). Decompression: 533.76 -> 999.13 ms (1.872x). These aggregate costs must accompany any density claim. No concurrent compilation is allowed during this benchmark.

Recorded 90 public default/fast/hybrid control comparisons; each must be byte-identical and source-exact. All 76 direct cases check the new reader on old archives and the old reader on candidate archives. See `direct_results.json` for every input, archive hash, source hash, trial and compatibility outcome.

Stock rc8 varied across measured trials on 4 inputs: 27_github_events_20150101_00.jsonl, 28_github_events_20150101_06.jsonl, 29_github_events_20150101_12.jsonl, 30_github_events_20150101_18.jsonl. The interrupted initial run that exposed this issue remains in `initial-cli-nondeterminism.log` and `initial-cli-partial-results.json`; absence of variation in a later short run would not invalidate the identified unstable tie ordering.

## Adaptive model only: exact saved parent archives

`adaptive_recode.rs` applies the candidate postpass to each exact saved rc8 archive without extracting keywords, searching matches or changing routes. This removes randomized keyword selection as a confounder. Every output must be no larger than its own exact parent, and an external candidate CLI restore must match the original source SHA-256. The complete `.pizd` bytes, including checksums and all model headers, count.

| Cohort | Sources | Exact parent bytes | Adaptive bytes | Reduction | Wins |
|---|---:|---:|---:|---:|---:|
| Public excerpts | 30 | 687,537 | 657,356 | 4.390% | 21 |
| Size/type controls | 24 | 7,056,286 | 6,983,023 | 1.038% | 13 |
| Later disjoint windows | 22 | 204,816 | 193,572 | 5.490% | 18 |

| Large input | Exact parent bytes | Adaptive bytes | Reduction |
|---|---:|---:|---:|
| nasa_large.log | 237,207 | 212,491 | 10.420% |
| hdfs_large.log | 177,553 | 167,635 | 5.586% |
| github_large.jsonl | 468,503 | 436,393 | 6.854% |

## Verified decoding without process startup

Both archive variants use the same candidate library reader. Input/archive files are preloaded; one warmup and five alternating measured trials include decoding, allocation and source-checksum verification, excluding file I/O and the final equality assertion. This isolates the cost of the new payload. On public30, summed per-input median decode time is 69.095 -> 399.217 ms (5.778x slower). This cost must remain visible alongside density improvements; CLI process startup can hide a substantial part of it. See `inprocess_decode_results.json` for all five measurements per input.

## Process memory diagnostics

Single `/usr/bin/time -l` runs on macOS; RSS includes all ordinary thin route searches and the adaptive postpass, not only PPM state.

| Input | Compression peak MiB | Decompression peak MiB |
|---|---:|---:|
| nasa_large.log | 107.34 | 10.92 |
| hdfs_large.log | 186.16 | 9.40 |
| github_large.jsonl | 337.17 | 23.73 |

## Regression test outcome

311 tests pass across 67 successful targets; 0 failed and 0 ignored. Exact invocation and profile scope are in `test_summary.json`. The final correctness suite uses release optimization with LTO disabled and 16 codegen units to avoid repeated whole-library LTO for every integration binary; the measured production CLI is separately rebuilt with the project's normal release profile. Initial failures and their corrected dispatch path remain in the audit logs. The added keyword reproducibility test repeats extraction with 64 independently randomized counting maps.

## Further encoder optimization, measured separately

The shared-learning prototype produces the same two escape-method streams while learning their common context counts once. Its 240 output controls match the original encoder byte for byte and cross-decode exactly. On 102 actual numeric stream/order paths, median paired/separate speed ratio is 1.730x; summed per-path median times are 2506.45 -> 1378.08 ms. This is a two-candidate model-search optimization, not a full-codec speedup. It remains outside the isolated runtime candidate, ready for the peer's integration review.

## Speed and resource limits

The bit-identical PPM implementation uses direct low-order context lookup and cached totals when no exclusions apply. The 360 controls cover empty/single-byte, constant/rescaling, full alphabet, noise, short tails, model resets, orders0/1/2/4/8/12 and three escape methods. Old and optimized encoders produce identical bytes and peak/reset counters; both decoders restore each stream.

One warmup plus three alternating trials on 123 actual stream/order paths measures isolated model costs. Median order1 speed ratios are 1.36x encoding and 1.28x decoding; this is not a whole-PIZD speed claim. The direct reader avoids the research scaffold's inverse Huffman re-encoding. PPM remains more CPU-intensive than independent-symbol Huffman. Model context count is capped at1,000,000, each node has its own variable symbol vector, and this is not a constant-byte RSS guarantee. Output length is validated before model decoding and accumulated channel representation is bounded.

## Limits and unfavorable evidence

- This cohort was used during development. The 22 later windows address byte overlap, not independence of projects or source families.
- Some production routes are unsupported by this postpass, notably non-Native and nonzero-pattern archives and structural payloads that do not expose root segregated channels. Their incumbent is retained and included in totals.
- Higher-order models and many packed numeric channels lose; all losing trials remain in the raw ledgers. No universal irreducibility claim follows.
- Historical same-input size references from the rc1 public comparison use their recorded versions/settings. The research total still exceeds historical Kanzi9's roughly 538.6 KB. These improvements do not establish superiority to every compressor, nor replace a fresh matched release panel.
- Sparse parsing's roughly0.39% public saving is compatible with old readers; the larger adaptive gain requires a new payload capability. Keep those compatibility claims separate.

## Evidence and reproduction

- `../sparse_production_rc8/results.json`:54 baseline inputs,32 canonically reconstructed supported archives and128 verified full existing-format candidate restores.
- `../literal_ppm_rc8/results.json`:378 complete literal research frames, exact parent reconstruction and source restore.
- `../numeric_ppm_rc8/results.json`:441 complete instruction-channel research frames, exact parent reconstruction and source restore.
- `validation_policy.json` and `validation_results.json`: frozen policy and22 disjoint later-window results; policies and worker binaries are SHA-256 fingerprinted.
- `fast-controls.log`, `fast_results.json`:360 bit-equivalence controls and123 stream/order timing paths.
- `integration_base_fingerprints.json`: main source state copied for isolated integration; not a claim of external attestation.
- `frozen_parents.tar.gz` and `frozen_parent_fingerprints.json`:76 exact archived research incumbents, retained because stock rc8 keyword ordering is not deterministic. These contain compressed corpus excerpts; retain the original corpus notices if redistributing.
- `candidate-targeted-tests.log`, `candidate-full-tests.log`: direct capability, exact archive checksum, malformed bounds and ordinary regression suites.
- `direct_results.json`: actual CLI density, timings, restores, unchanged-mode controls and compatibility.
- `same_parent_results.json`: adaptive-only complete-byte bounds and source checks on all 76 exact saved parents.
- `standard-bench.md`: fresh three-input compatibility smoke against Zstd raw/trained, Gzip and Brotli Q6 using the built-in default-profile suite. Its same-input dictionary training is not a disjoint-training result and is not used for the adaptive density claims.

The experiment entrypoints are `tools/probe_sparse_production.py`, `tools/probe_literal_ppm_production.py`, `tools/probe_numeric_ppm_production.py`, `tools/validate_adaptive_native.py`, and `tools/bench_adaptive_direct.py`. The first four are research scaffold experiments; the last compares actual direct CLI builds. Original corpus manifests preserve URLs, excerpts, sizes and hashes. Retain their notices when redistributing datasets.

`paired_model_design.md` describes the additional bit-identical shared-learning prototype. `paired_learning.patch` is an unapplied follow-up against the isolated candidate: its encoder component has the 240-output/123-path audit, while the runtime call-site integration still needs compilation and complete archive-identity checks after application. `NEXT_OPTIMIZATIONS.md` ranks the remaining measured reservoirs and carefully scopes untested hypotheses. `reference_size_comparison.json` retains the historical same-source reference totals and win counts without relabeling old timings as a fresh release panel.