# Exact numeric-run density improvement — 5.0.0-rc5

**Implemented and verified in standalone Native/Hybrid thin compression.** The 30% complete-size threshold is exceeded on several real logs and later nonoverlapping source windows. Across the full thirty-source development panel, the gains are **5.81% Native / 5.07% Hybrid**, with additional time and memory. The folder name identifies the frozen rc4 baseline, not the delivered version.

## Production results

| Matched profile | Previous bytes | New bytes | Reduction | Improved files / 30 |
|:--|--:|--:|--:|--:|
| native_thin | 805,043 | 758,258 | 5.81% | 12 |
| hybrid_thin_L9 | 726,532 | 689,715 | 5.07% | 10 |

Both profiles retain the incumbent when the numeric candidate loses: zero stored-size regressions across all forty inputs / eighty production cases. The 30-source panel comprises sixteen LogHub datasets, two historical NASA windows, four GitHub JSON windows and eight separately labeled parser fixtures. Ten additional inputs are project API and reconstruction controls, not ten independent real production captures.

| Example | Profile | Before → after bytes | Reduction |
|:--|:--|--:|--:|
| 04_loghub_zookeeper.log | hybrid_thin_L9 | 8,049 → 5,457 | 32.20% |
| 09_loghub_windows.log | native_thin | 17,470 → 10,889 | 37.67% |
| 09_loghub_windows.log | hybrid_thin_L9 | 15,015 → 9,642 | 35.78% |
| 13_loghub_healthapp.log | native_thin | 34,560 → 20,526 | 40.61% |
| 13_loghub_healthapp.log | hybrid_thin_L9 | 29,399 → 18,083 | 38.49% |

These are matched ordinary thin comparisons, not a claim of 30% improvement on every input or against every possible option combination. The separate actual-CLI comparison includes Hybrid template factorization and two extra forced-line-sort controls. The minimum of those five previous settings is reported, rather than quietly ignoring a stronger existing PIZD option. On the original HealthApp excerpt, improvement versus that stronger prior minimum is about 20.4%, not 38.5%. Forced line sorting does not improve any previous minimum in these eleven cases.

## Later windows

Before downloading/evaluating new source ranges, freeze mode 1 (integer deltas), maximum width 38, delimiter context 12, both groupings, input cap 16 MiB and column cap 4,096. The encoder source hash is recorded in integration_parameters.json. Acquire fixed complete-line windows after 1 MiB and 2 MiB from the same Spark, Windows and HealthApp sources, beyond both prior 512 KiB test and 128 KiB training prefixes. Retain exact URLs, archive members, byte ranges and hashes in later_inputs.json. The six windows are disjoint bytes but correlated previously seen source families, not independent unseen systems.

| Later source | Profile | Before → after bytes | Reduction |
|:--|:--|--:|--:|
| spark_1024k.log | native_thin | 32,611 → 20,185 | 38.10% |
| spark_1024k.log | hybrid_thin_L9 | 27,322 → 18,254 | 33.19% |
| spark_1024k.log | hybrid_thin_L9_template | 22,865 → 18,254 | 20.17% |
| spark_2048k.log | native_thin | 31,945 → 19,891 | 37.73% |
| spark_2048k.log | hybrid_thin_L9 | 26,149 → 17,926 | 31.45% |
| spark_2048k.log | hybrid_thin_L9_template | 22,292 → 17,926 | 19.59% |
| windows_1024k.log | native_thin | 14,748 → 8,334 | 43.49% |
| windows_1024k.log | hybrid_thin_L9 | 12,578 → 7,350 | 41.56% |
| windows_1024k.log | hybrid_thin_L9_template | 12,578 → 7,350 | 41.56% |
| windows_2048k.log | native_thin | 16,209 → 8,569 | 47.13% |
| windows_2048k.log | hybrid_thin_L9 | 13,936 → 7,562 | 45.74% |
| windows_2048k.log | hybrid_thin_L9_template | 13,936 → 7,562 | 45.74% |
| healthapp_1024k.log | native_thin | 38,263 → 19,000 | 50.34% |
| healthapp_1024k.log | hybrid_thin_L9 | 32,449 → 16,547 | 49.01% |
| healthapp_1024k.log | hybrid_thin_L9_template | 26,346 → 16,547 | 37.19% |
| healthapp_2048k.log | native_thin | 38,278 → 20,027 | 47.68% |
| healthapp_2048k.log | hybrid_thin_L9 | 31,994 → 17,766 | 44.47% |
| healthapp_2048k.log | hybrid_thin_L9_template | 26,643 → 17,766 | 33.32% |

Native improves all six later windows by 37.73–50.34%. Hybrid improves them by 31.45–49.01%. Against prior Hybrid template factorization, the later Windows/HealthApp windows still improve 33.32–45.74%; Spark gains only about 20%. Every outcome is retained in cli_results.json. No thresholds were tuned on those later windows.

## Representation and full accounting

Discover numeric columns from exact ASCII digit width, optionally preceded by up to twelve delimiter bytes. Keep exact skeleton bytes and column references. Store per-column signed numeric differences with unsigned 128-bit Zigzag varints. Original leading zeros, signs, fraction/exponent separators, whitespace, NULs and ordering are reconstructed exactly; digit runs longer than 38 remain literal. No log-family, field-name, timestamp or HTTP assumptions are hardcoded.

Keep Native and Hybrid entropy backends unchanged. Serialize every model byte and encode it as a complete ordinary self-contained PIZD archive. The new outer source wrapper adds 55 bytes, including original size/hash and nested length. Research candidates paid a 48-byte envelope, so production pays **seven additional bytes**. All eighty production results match the predicted minimum of the complete old archive and the two eligible delta candidates plus seven bytes. Independently prepared Python model bytes match production Rust bytes.

The implementation uses explicit container 10 (SHA-256) or 11 (BLAKE3) with zero outer flags/patterns. The nested archive must use versions 1–9 without patterns or remote dependencies. Recursive numeric wrappers, dimension/count/width/reference overflow, unused scalar data, trailing bytes and output beyond the declared limits are rejected. Decoder column cursors avoid allocating arrays of scalar values. This is bounded validation, not an exhaustive security audit. See [the format](../../numeric_columns.md).

The ordinary winner is computed first and retained on ties/losses. Numeric trials require standalone clients without a local store/server, at least sixteen matcher ways, structured/typed density options, nonzero pattern budget and 512 B–16 MiB input. Default/fast and external-dictionary/server behavior stays on its previous path. Larger inputs can use bounded stream frames. Old archives remain readable; pre-extension readers explicitly reject versions 10/11.

## Time and memory costs

Actual old/new CLI subprocess timing includes launch, source/target file IO and normal production source verification. Two warmups precede three measured medians; old/new order alternates. Verified decode medians use the new reader for both archive versions, isolating representation cost. Every emitted output is restored exactly and independently SHA-checked outside timers. Aggregates sum the eleven source medians, not the median of a mixed source pool. Shared-host load limits precision; p-value or universal latency guarantees are not claimed.

| Profile / 11 sources | Encode old → new ms | Change | Verified decode old → new ms | Change |
|:--|--:|--:|--:|--:|
| native_thin | 365.3 → 542.5 | +48.5% | 107.0 → 127.6 | +19.3% |
| hybrid_thin_L9 | 394.8 → 592.6 | +50.1% | 102.1 → 137.9 | +35.1% |
| hybrid_thin_L9_template | 436.3 → 624.4 | +43.1% | 105.1 → 130.6 | +24.2% |

This is a deliberate thin density tradeoff. Rejected models also add work. The matched measured cost increases are not hidden behind the smaller archives. Memory samples use one complete-process macOS time-l run per version/engine on unmodified approximately 3 MiB prefixes: Native peaks about 30 MiB versus 23–25 MiB before; Hybrid about 42–46 MiB versus 29 MiB. These are observed samples, not worst-case bounds or IoT-memory guarantees. Arrays/skeletons/workspaces increase encoder memory. Four real multi-frame stream controls (old/new, both engines, 1 MiB chunks) also restore exactly; they do not claim a 20 GiB test.

## Strong references and losses

C Brotli Q11, C Zstd L19 and deterministic no-name Gzip L9 are independently compressed, decoded and SHA-verified on all eleven CLI sources. Their versions, executable hashes, commands, frame hashes and sizes are retained. Raw reference frames keep their native framing; PIZD pays its complete model and original-source integrity. These stronger settings are controls, not equal-effort promises. All values are exposed in the HTML report, including reference wins.

The refreshed ordinary public 30-file panel additionally includes Zstd with/without a 16 KiB disjoint-window trained dictionary, Gzip and Brotli across fast/default/thin equivalents: [540 verified configurations](../public_logs_30_5.0.0-rc5/index.html). Full dictionary cost is charged to each file in single-file totals, with shared-batch costs shown separately. The standard eighteen-archetype panel was also rerun with rc5; encoding-speed wins against the best reference remain **0/18**.

API and GitHub JSON controls retain their previous production archives: no new JSON density benefit is claimed. An unconditional numeric replacement would enlarge some files; all 800 research candidates, including losses, remain in results.json/comparison.csv. Research dictionary/bitpacking modes are not shipped. Earlier sparse/global JSON, implicit-context-copy and bounded PPM experiments miss the goal and are retained in sibling evidence folders. Their small component gains are not added to production percentages.

## Verification and reproduction

- Full library/bin/integration suite: **283 passing tests across 58 targets**, zero failures. Separate doctests complete successfully (none defined).
- Eighty production source/engine cases restore exact bytes/SHA, match independently prepared Python models and complete stored-cost predictions.
- Actual CLI: **333 exact/SHA restores** across 33 profile/source cases, including backward reading and old-reader rejection of new versions.
- Thirty-three strong-reference restores, eight whole-process resource restores, four stream restores and twenty-two forced-old-sorting restores. Four additional actual-CLI SHA-256 numeric controls emit container 10 and restore exactly; all default BLAKE3 cases emit container 11 when selected. Native nested frames are checked to contain no Zstd/Deflate payloads.
- Research: 800 complete candidate restores, 80 baseline controls, 40 Rust Brotli Q11 controls and 80 additional numeric/binary lexical checks. Do not double count the overlapping initial/final research stages.

Production: build with `cargo build --offline --locked --release --bin pizd`; run standalone `--thin`, optionally `--engine hybrid`. The new reader is required for new numeric containers. The actual release binary is built locally as 5.0.0-rc5; there is no tag, push, deployment or independently attested release from this work.

The frozen pre-change dirty source tree is baseline_source.tar.gz, with every source hash recorded before integration. candidate_source.tar.gz and source_manifest.json record the verified implementation. baseline_cli.json/cli_builds.json identify actual binaries, kept under ignored target/. Raw public source bodies, generated models and compiled binaries remain under target/ and are not added to Git.

Replay the original research against the frozen old core with `python3 docs/benchmarks/numeric_density_rc4/baseline_replay.py`; it writes a new result under target/ and preserves reviewed evidence. Do not rerun the old worker against the new core and relabel that output as an rc4 baseline. For current production comparison, build/run `numeric_integration_review`; it reads the frozen input/results and checks exact output/cost identities. `cli_benchmark.py` reruns the captured old/current binary pair. `resources_and_references.py` reruns memory, stream, stronger references and sorting controls. Cached public inputs are required; acquire_later_windows.py reproduces the fixed later ranges and validates original-prefix identities.

Source hashes, build logs, test exit markers and raw trial data are retained. Re-render with `python3 tools/summarize_numeric_density.py`. Historical measured versions are preserved; current website data points to the completed rc5 public panel. The initial aborted integration-review log records a small-fixture prediction that omitted the production 512-byte gate; the review oracle was corrected without changing the codec. The initial public-panel launch required copying the unchanged frozen corpus manifest; no timing data came from that failed preparation.

## Contribution and remaining limits

This is useful PIZD engineering: generic byte-preserving numeric discovery, two reversible column contexts, bounded decoding and exact complete-cost competition. It combines established techniques rather than inventing a new entropy coder or proving a compression maximum. Specialized CLP/Logzip/LogPrism comparisons and independent source-family validation remain future publication work. The [manuscript recommendations](../../paper/REVISION_RECOMMENDATIONS.md) explicitly separate implemented results from novelty and unmeasured claims.