# Generic composed template columns

Available in the unpublished **6.0.0-rc1** candidate with:

```sh
pizd compress input.log --offline --standalone --thin --template-factorization --engine native -o input.pizd
pizd decompress input.pizd --offline -o restored.log
```

Hybrid uses the same representation with `--engine hybrid`. Extra candidates compete on complete archive size and retain the existing archive on every tie or loss. This option adds encoding effort; it does not alter default/fast or dictionary/server routing policy.

## Model rather than a known schema

The common discovery interface consists of borrowed template anchors, parameter spans, family IDs and row boundaries. It does not name applications, filenames, keys or field semantics. Logs reuse existing token discovery. JSON arrays of records and NDJSON use validated exact scalar spans, preserving key order, duplicate keys, nesting, whitespace, separators, escapes and number spelling. String quotes become anchors; escaped string contents remain original bytes. There is no conversion of `23.0` into `23` or normalization of JSON strings.

The three log choices are positional typed/numeric columns, family/position delta columns and family/position typed/numeric columns. Sparse families with fewer than16 records pool by position. JSON compares positional, family and global scalar columns, allowing lexical reuse across different schemas. Duplicate encoded models are not tried twice. A JSON record with more than256 scalar spans becomes a literal template inside the same ordered model; it is not omitted. Single JSON objects containing nested record arrays are not currently recognized as batches by this adapter. Unsupported cases retain their existing archive.

The typed/numeric choice reuses existing constant, dominant-default, dictionary, prefix, exact decimal and uniform-width integer const/stride/RLE/bit-packed implementations. Numeric renderers preserve original fixed widths, including leading zeros. Adding a parser adapter does not require an application-specific compression backend. The existing CSV routes remain available; this integration does not claim a new general CSV parser.

## Explicit archive capability

Containers **12** (SHA-256) and **13** (BLAKE3) use the existing51-byte header and4-byte compressed payload length. The original-size/hash fields describe the restored source. The payload contains the composed model, compressed by Native flat/segregated Huffman or optional rANS4, or by Hybrid Zstd.

Allowed flags are structural+typed with Native and an optional flat long-copy capability, or structural+typed with secondary Zstd. Pattern count must be zero. Other flags are rejected. Segregated payloads prohibit the flat long-copy flag. The new version identifies the entire grammar, including its typed-column capabilities; it does not silently extend legacy kind17/PZTF02. Older readers reject12/13. The current reader still accepts old formats1–11. The package major candidate records this format capability addition.

The decoded model is:

| Field | Encoding |
|:--|:--|
| Model identifier | `PZTC03\0\0` (8 bytes) |
| Template count, record count | Two little-endian u32 values |
| Composed layout selector | Byte6 |
| Template catalog | u32 byte length; each template has a varint anchor count, then varint-length-prefixed original anchor bytes |
| Chronological family IDs | u32 byte length; u8 IDs for at most256 templates, otherwise little-endian u16 |
| Template-to-column maps | One unsigned varint per parameter position, in catalog order; parameter count is anchor count minus one |
| Column strategy | Byte1 for legacy raw/delta columns, or byte3 for typed/numeric columns |
| Columns section | u32 byte length; varint column count followed by column records |

Each column starts with its value count, which must match the occurrences derived from the family IDs and maps. Strategy1 retains the existing raw-string or width-preserving integer-delta tags. Strategy3 has a varint payload length and one existing typed-column stream, or tag240 followed by lexical width and the shared packed numeric kind/payload. Packed numeric payloads do not duplicate NUMSTR framing or a synthetic skeleton.

The packed kinds retain the existing definitions: constant, fixed delta, delta frame-of-reference bitpacking, RLE of delta codes, or delta varints. Their decoder is shared with the existing numeric-run model. Schema maps and every model/header byte are part of the transmitted archive.

## Reconstruction budgets

This is a bounded in-memory model for source frames up to8MiB. It supports up to1,000,000 records/parameter occurrences,65,536 templates,4,096 columns and262,144 mapping entries. Backend output is bounded by twice the source size plus64KiB; flat token decoding has its own corresponding finite bound.

Before final output allocation, the reader checks section boundaries, family and map references, expected parameter counts, checked anchor expansion and aggregate decoded lengths. Raw/delta values borrow their serialized bytes. Packed values use a cursor and stack digit buffer. Existing typed readers may allocate backing storage for their modes, so this is not a zero-allocation claim. JSON span extraction does not retain a copied path dictionary per scalar. Exact output length and consumed column values are required even when source-hash verification is explicitly disabled by a library caller.

The earlier segregated literal-run guard remains active: runs above65,535 bytes decline the segregated plan and retain flat coding. No external `.pz`, schema service or source file is required to decode the composed archive.

## Evidence and limits

[Integrated CLI evidence](benchmarks/template_composition_integration/index.html) records exact source identities, complete sizes, repeated elapsed times, no-gain/slower inputs and old-reader rejection. [The preceding research](benchmarks/template_composition_rc6/index.html) remains a frozen study with a different experimental reader and exhaustive search; do not substitute its minima or one-shot timings for production results.

Generic model coverage does not imply a size win on every JSON source. The inspected GitHub/API data remains a priority when it retains its incumbent. Cross-chunk schema sharing, a generic key/value record parser, nested record-array extraction and independent larger workload validation remain separate work. No unmeasured density gain or entropy-coder novelty is claimed.