Generic template-column integration — 6.0.0-rc1

**Implemented in the CLI, opt-in.** Compare the frozen rc7 binary with the current release build using `--thin --standalone --template-factorization`, Native and Hybrid. The baseline already has optional legacy template factorization. The new option retains that complete incumbent whenever a candidate ties or loses.

Containers12/13 identify the new SHA-256/BLAKE3 composed representation. Older readers reject them explicitly; the new reader retains old formats1–11. The major candidate version records this archive capability change. No commit, push, tag, deployment or public binary release has been performed.

Complete stored size and measured cost

Actual CLI subprocess measurements include process launch and file I/O; exact-byte and independent SHA checks are outside timers. Alternate before/after order with two warmups and five measured rounds. Category times sum per-file medians; decoder times include archive checksum verification. These are shared-host observations on inspected sources, not isolated codec-kernel speed or machine-independent confidence intervals. The final capture runs after the source verification jobs finish.

Positive time changes mean more elapsed time. Every no-gain/slower source remains below. A larger global model search is not enabled by default. The standard comparator presets do not silently include the optional template switch.

What is generic

The common representation consumes record templates, value spans and chronological family IDs. No source filename, HTTP code, IP range, city name, log-product name or JSON field name selects a candidate. Logs retain the existing numeric/IP/token scanner. Validated JSON arrays of records and NDJSON provide exact scalar spans to the same representation, including nested values, duplicate keys and arbitrary key order. Quotes, escapes, number spelling, punctuation and whitespace restore exactly. A record above256 scalar spans is retained literally inside the model, rather than dropping it or rejecting all other records. Other unsupported inputs keep their ordinary archive.

Three log candidates: positional typed/numeric columns, family-plus-position delta columns and family-plus-position typed/numeric columns. Families with fewer than16 records pool by position. JSON compares positional columns, pooled-family columns and a shared lexical-value stream; the latter preserves possible reuse across changing schemas. Exact duplicate model bytes are coded once. This is deterministic complete-cost competition, not a claimed optimal model, trained semantic ontology or new entropy coder.

The four inspected GitHub windows and four additional API JSON fixtures currently retain their incumbents; generic support is not evidence of a measured JSON size gain on this panel. Single JSON objects with nested record arrays are not currently promoted as record batches by this adapter. Existing parsers/models remain available. CSV keeps its existing structured routes and the numeric-family scanner where eligible.

Decoder and compatibility

Raw and delta columns borrow encoded slices. Fixed-width packed integer columns use the shared numeric cursor without constructing synthetic NUMSTR frames or an allocation per packed scalar. Typed modes reuse the existing bounded column reader; some modes allocate decoded backing storage. This is bounded in-memory reconstruction, not a zero-allocation or20-GiB streaming claim.

Before final output allocation, validate catalog/ID/mapping references, expected value counts, checked fixed-template expansion and aggregate decoded lengths. The frame budgets are8MiB source,1,000,000 records/parameter occurrences,65,536 templates,4,096 columns and262,144 mapping entries, with source-derived backend limits. The JSON span adapter does not copy a field-path dictionary per scalar. The preceding rc7 literal-run representability guard remains in force.

The current suite passes **291 unit/integration tests**; **1536 actual CLI source restorations** match original bytes and SHA-256. **21 selected new-format archives** are explicitly rejected by the frozen old reader. **24 no-option controls** preserve archive bytes across default/fast/thin and both engines. Targeted tests cover SHA/BLAKE3, optional rANS4, lexical JSON, wide-record exceptions, capability downgrade, source-hash corruption, truncation, expansion and mutations. Source/binary hashes and exact replay snapshot are retained. Earlier interrupted runs are labeled partial and are not counted as passing final verification.

Reference stored-size positions

Reference archive sizes come from the same exact public inputs in the recorded6.0.0-rc1 panel. This table compares size only; it does not reuse reference timing for a speed claim.

Every source, including no-gain and slower cases

Each size comparison is a complete self-contained archive. Times are medians in milliseconds; the JSON raw trials retain all repeats.

Reproduce

Build the recorded source snapshot with `cargo build --release --bin pizd`. Keep the frozen rc7 reader at `target/composition-integration/pizd-rc7`; its source snapshot and binary identity are recorded in the preceding study. Run `python3 tools/verify_template_composition.py`, then `python3 tools/render_template_composition_integration.py`. Cached inputs are identified by exact source hashes; raw public logs and binaries are not added to Git.

`pizd compress input.log --offline --standalone --thin --template-factorization --engine native -o input.pizd`

Use `--engine hybrid` for the Zstd-backed variant. Use the current reader to restore archives produced with the new capability. Older reader rejection is intentional; ordinary retained archives keep their prior formats.

Remaining work

Broaden independent real structured workloads after freezing this policy. The generic JSON route still needs a measured density advantage on the current losing GitHub data. Deeper/shared schemas, generic key/value record adapters and actual stream-model reuse remain separate research; no gain is booked from those ideas.

Ordinary fast-policy boundary

The fresh maintained public30 panel has540 verified rows; the18-source standard panel records17/18 size,1/18 encode-speed and11/18 verified-decode-speed wins in its cross-profile best-reference summary. Those ordinary presets do not contain the optional template switch; these counts are not its isolated effect.

Against the earlier recorded public rc6 source, Hybrid fast HealthApp changes36,922 ->45,430 B, Apache27,102 ->35,284 B and the Traefik fixture941 ->938 B;537 other stored-size rows match. The earlier public source used a22-percent structural-benefit gate. The frozen rc7 baseline already used24 percent and a broader direct fallback. Three additional CLI controls prove the affected fast archives remain byte-identical between frozen rc7 and this integration, with exact restoration. Preserve the larger historical fast results and keep that source boundary separate from composition. Details: fast_policy_boundary.json and ordinary_panel_identity.json.