Conformance & CI Technical Specification

Date: 2026-08-08 · Status: Draft 1 · Purpose: how conformance is claimed, proven, and continuously enforced. This is the machinery behind the project’s central claim — spec §12’s “CI MUST fail on divergence” and PRD CL-7/M1/M2.

1. Principles

1. A conformance claim is a build artifact, not a README sentence. Every claim traces to a machine-readable report produced by running the vectors in CI at a named vector-suite version.
2. Cross-implementation round trips run on every merge to the integration branch, not just releases. The Phase 1 exit criterion (PRD §8: “a value written by Python is read by TypeScript, and vice versa, in CI”) is a permanent gate once first met — regressions on it block merge.
3. Third-party implementations can run everything without this repo’s CI. The harness contract (docs/08 §5), report format (§4 below), and the CC0-licensed vectors (settled 2026-08-09 — LICENSES.md) are the entire interface; PRD metric M2 (an implementation not written by us passes) depends on nothing here being private.

2. Repository and workflow topology

Monorepo (current layout). GitHub Actions [assumption: GitHub remains the host — the design below uses only artifacts + matrix jobs and ports to any CI]:

.github/workflows/
  vectors.yml        validates vector files against schema + MANIFEST hashes; runs on any vectors/ change
  core-python.yml    lint, typecheck, unit, property, vector harness  (paths: core/python/**, vectors/**)
  core-typescript.yml  same for TS
  adapter-django.yml   adapter matrix (Django versions × Postgres/SQLite), needs core-python
  adapter-prisma.yml   adapter matrix (Prisma version × Postgres/SQLite), needs core-typescript
  cross.yml          the N×N cross-implementation job (below)
  conformance.yml    assembles per-run conformance reports into the published summary

Delta, updated 2026-09-22: what ships is still one workflow, conformance.yml, now covering typescript-core, python-core, cross-core-result-ids, django-adapter (SQLite and Postgres), prisma-adapter (SQLite and Postgres), vectors-reproducible, suite-integrity, unicode-tables, cross-produce/cross-consume and python-lint. cross-core-result-ids was added 2026-09-04 and is the one job that reads more than one report: it compares the synchronous result-id sets of every core its --cores list names, which docs/08 §9 (line 414) had claimed since the second core shipped with nothing enforcing it. It compared exactly two until 2026-09-22 (WS-L, docs/26 §1 item 4). Since then each core’s set is compared with the union of all of them, and a finding names the core that did not run an id and the cores that did. A report that is missing or unreadable is a finding by name, and the remaining reports are still compared with each other. The per-core jobs cannot catch what it catches — each compares a core against the vectors that core chose to run, so a core that stopped running a family entirely stays green. It runs under if: !cancelled() rather than the implicit success() a bare needs: gives, because a red core is when a silently dropped family is most likely to be missed. Three things keep its own comparison honest, each of which it would otherwise assume: every first-pass result must be twinned where a report declares async_companions (§4 line 126 — presence of #async results alone would pass a partial second pass, since stripping the suffix cannot shrink the synchronous set); skipped results are excluded from the sets and reported, because two cores skipping the same family is not agreement; and every file in MANIFEST.json must be reached by every core, which is the only part that can see a drop hitting every core at once. It compares out_of_band ids too — nothing else in CI asserts those three checks exist. What it still cannot see: a family removed from the manifest as well, which is a suite change and belongs to suite-integrity and vectors-reproducible. Since 2026-09-19 the comparison is tools/conformance/compare_result_ids.py rather than Python inline in the workflow, and the job runs test_compare_result_ids.py first: until then every guard above had been verified by hand only. The test starts from a synthetic set of agreeing reports (two cores, and three for the N-core cases) and breaks one input per case — including a duplicate id standing in for a dropped one, an invariant with no other home, since pytest suffixes a duplicated test id and the TypeScript harness collapses results in a Map. The per-file split this section anticipated “when adapters arrive” did not happen when they arrived, and on the evidence it should not: one workflow keeps the cross jobs’ needs: edges in one file, which is where the ordering that matters lives. There is no cross.yml; the cross jobs run inside conformance.yml on every push and on the nightly schedule (catching toolchain drift — a Node or OpenSSL upgrade changing behaviour is exactly the class of bug this project exists to surface).

3. The cross job (the one that matters)

Two stages, artifact-mediated, full N×N including self-pairs:

stage produce (matrix over implementations):
  each impl runs its cross-producer (docs/08 §4.7) with its REAL production path
  (runtime CSPRNG, no injection), against vectors/keys/test-keys.json
  → uploads cross-<impl>.json artifact

stage consume (matrix over consumer × producer pairs):
  consumer downloads producer's artifact, decrypts every case, compares plaintext byte-exact
  → uploads verdict-<consumer>-<producer>.json

gate:
  all pairs green, or the workflow fails. A pair may be `skipped` ONLY for a suite the
  consumer does not claim (0xFF02), and skips are visible in the summary, never silent.

Self-pairs (python→python) stay in the matrix deliberately: they distinguish “producer broke” from “pair broke,” halving diagnosis time.

Implemented 2026-08-23 in conformance.yml (cross-produce matrix → artifacts → cross-consume matrix, plus the nightly schedule): each producer encrypts the 16-case shared corpus (vectors/cross/corpus.json, generator-emitted so it cannot drift between languages) through its production path; each consumer decrypts every producer’s artifact, including the 2000-byte-context envelope that failed silently on 2026-08-22. Four producers as of 2026-08-31 — the two cores (core/python/tests/cross_produce.py, core/typescript/tests/cross/produce.ts, npm cross:produce) and the two adapters (adapters/django/tests/cross_produce.py, adapters/prisma/tests/cross/produce.ts), which is the claim tested at the layer people deploy rather than only between cores; two consumers, both cores. The producer list lives in one place, the workflow’s CROSS_PRODUCERS: each consumer’s arguments are built from it, and a step before any consumer runs fails if a listed producer uploaded nothing or an unlisted one uploaded something, which is what keeps it from drifting from the cross-produce matrix (WS-L, 2026-09-22). Both halves as of 2026-08-31: each consumer checks the envelope cases and the cross/v2 index cases, and its verdict tags every pair envelope or index — they fail for different reasons and want different reading, since an envelope mismatch is a decrypt problem while an index mismatch is a silent lookup miss in production. The pair count is deliberately not restated here — a hand-copied integer in prose is how the status line in vectors/README.md went two suite versions stale — and each consumer’s verdict artifact carries its own. A verdict artifact is uploaded per consumer.

key_ref resolution uses vectors/keys/test-keys.json, emitted by the generator since suite 0.2.0 (format in docs/08 §4.7; two refs, tenant-a-dek-v1 and tenant-b-dek-v1). Static cross vectors (vectors/cross/static/) are additionally verified by each core’s ordinary vector harness — they catch drift against released implementations, while the dynamic job catches drift at head.

4. Conformance report format

Every harness (core or adapter) emits conformance-report.json:

{
  "schema": "fieldseal-conformance/v1",
  "implementation": { "name": "python-core", "version": "0.1.0", "commit": "…", "language": "python" },
  "vector_suite_version": "0.1.0-provisional",
  "spec_version": "0.1-draft",
  "claimed_levels": { "L0": true, "L1": false },
  "suites_supported": ["0xFF01"],
  "provisional_suites": true,
  "environment": { "runtime": "CPython 3.12.x", "os": "ubuntu-24.04", "crypto_backend": "OpenSSL 3.x",
                   "unicode_platform": "CPython unicodedata 16.0.0 (not used for nfc-casefold-v1)",
                   "unicode_tables": "vendored UCD 17.0.0 (NFC + CaseFolding C+F)" },
  "pinned_decisions": {
    "decrypt-order": "recognition → LENGTH_EXCEEDED → SUITE_NOT_ALLOWED → KEY_UNAVAILABLE → per-candidate commitment-then-open …",
    "aad-mismatch": "never raised on the 0xFF01 path …"
  },
  "harness_notes": [
    "docs/08 §5 item 2 (schema validation) not performed: vectors/schema/ is empty"
  ],
  "results": [
    { "id": "envelope/ff01/basic-roundtrip", "status": "pass" },
    { "id": "envelope/ff02/basic-roundtrip", "status": "skipped", "reason": "suite 0xFF02 not implemented" }
  ],
  "held_out": [
    { "path": "<family>.json", "status": "not-run", "reason": "why the suite does not count it, and what would unblock it" }
  ],
  "out_of_band": [
    { "id": "spec/3.5/length-bound", "status": "pass", "basis": "direct", "method": "unit test asserting 2^31 refused with LENGTH_EXCEEDED" }
  ],
  "async_companions": false,
  "summary": { "pass": 412, "fail": 0, "skipped": 18, "held_out": 5 }
}

out_of_band carries normative requirements that have no vector. Each entry’s status is one of pass, fail or not-run; no other value is defined. (Until suite 0.8.0-provisional the TypeScript harness’s report type admitted a fourth, not-verified, which this section never defined; G26, #176.) A harness that cannot run one on its runtime records "status": "not-run" with the reason (docs/08 §5 item 8) rather than omitting the entry; a not-run entry is not pass and blocks the level claim on the same terms as a failure. Each entry also carries basis, how it was established: direct (the operand was built and the public operation called on it), seam (the refusal was proven on synthetic operands through an internal path every public call takes) or representability (the language cannot hold the operand, and the entry records why the requirement holds by construction). An absent basis reads as direct. The basis is part of the claim: a level claim quoted outside the report MUST name every entry whose basis is not direct. Two requirements qualify, for opposite reasons:

• spec §3.5, the plaintext length bound — spec/3.5/length-bound (encrypt) and spec/3.5/length-bound#decrypt (an envelope whose implied plaintext length, the received length minus the suite’s fixed overhead, is at least 2³¹; the entry is defined by that property, not by an overhead constant). Excluded because a 2-GiB input is not a repository artifact. A runtime that tried to build the operand and could not allocate it records not-run. Any core MAY instead prove the refusal through the docs/09 §4 length seam and record pass with basis: "seam" (G26, #176), and a core whose byte type cannot hold the operand (Java and .NET: int array lengths) has no other route to pass. A seam pass requires: synthetic operands (length 2³¹ for encrypt; an implied plaintext length of at least 2³¹ for decrypt) driven through the seam every public encrypt, decrypt and rotate call constructs; LENGTH_EXCEEDED, zero key-provider calls and zero reads of operand content, all asserted; a test that fails when the guard is moved one statement later; and a method that says it is a seam, names the public surface the seam carries, and, where it applies, says that the guard is unreachable from the public byte API.
• docs/09 §7.1, the lone-surrogate refusal — docs/09/7.1/lone-surrogate-refusal. Excluded because the operand is not portable. blind-index/ keys its input as hex bytes and an unpaired surrogate has no UTF-8 encoding, so the case cannot be written in the family’s shape; widening that field to text would not help, because Go string literals may not hold a surrogate value and Rust’s String is UTF-8 by invariant — two of the five target languages cannot carry the operand at all. What the family cannot carry, it now carries in bytes (G26, #176): the blind-index/ refuse vectors (docs/08 §4.4) pin the generalized-UTF-8 forms of two lone surrogates as invalid input every core must refuse with INVALID_ARGUMENT, which is the only form a surrogate can take where strings are bytes. A core in such a language records this entry with basis: "representability", and pass only once those vectors are pinned and pass in that core, its harness shows the two byte preimages refused distinguishably, and method states the premise (docs/08 §5 item 9); until then it records not-run. not-run is not widened to mean established by argument: it is the status that keeps a core from counting toward PRD metric M1 (docs/25 §10 item 3), and the always-reported basis is what makes the wider pass safe to read. The entry asserts both that two distinct unpaired surrogates are refused and that they are refused distinguishably: identical diagnoses would leave the two values indistinguishable to the caller, which is precisely the property the refusal exists to deny them. Verified in Python and TypeScript, whose string types can represent the operand.

The second entry is the more interesting precedent. The first is excluded by size, which is an accident of the test harness; the second is excluded by representability, which is a property of the requirement itself and will not go away. A requirement that cannot be a vector in every target language is exactly the requirement most likely to be quietly skipped, which is why it is named in the report rather than left to each core’s test suite.

pinned_decisions (reserved here since 2026-08-23; docs/18 D-19) is where a Gate 0a implementation declares the choices spec §9 obliges it to declare — every place the specification leaves an observable behaviour open or silent and the implementation had to pick one. It is an object of free-text values under fixed keys, so that two implementations’ reports can be diffed key by key and a disagreement names the clause. The keys every core MUST carry, and what each states:

A core MAY add keys for pins of its own; it MUST NOT omit one above. When the specification settles a row, the row is removed here and the key retires.

Retired 2026-08-24, when issue #48 (G15) closed:

A pinned decision records where an implementation had to choose with nothing behind it. Once the clause exists there is nothing left to declare, and a report that kept the key would misreport the core as still making a choice. Both cores’ reports dropped all four on 2026-08-24; tests/vectors.test.ts asserts their absence.

harness_notes (reserved on the same date) is a list of free-text statements about what the harness could not do or had to assume — a schema directory that was empty, a vector family whose fields did not match docs/08 and were mapped, an id-suffix convention. The convention both shipped harnesses use: <id>#decrypt is the decrypt direction of an envelope/ vector (docs/08 §4.1 requires both directions, and a separate result makes a one-sided failure visible); <id>#pipeline is a blind-index/ vector run end to end through the public blind_index operation rather than through the primitives; <id>#async is the G9 async pass (docs/08 §5 item 10). The suffix is applied to the synchronous result id, not to the vector id, so it lands last and composes with the others: <id>#decrypt#async, <id>#pipeline#async.

async_companions (reserved on the same date, defined here 2026-09-04) is true iff both halves hold: the implementation exposes the spec §11.1 asynchronous companions, and this report carries the second pass — every result of the first pass twinned by an #async result. It is not a capability advertisement: a core that shipped a companion and did not run the suite through it reports false, because the flag’s consumer is a reader deciding whether the #async results in front of them are the whole suite or a selection. Where an operation has no companion the twin re-runs the synchronous operation, so the pass is the entire suite by construction; harness_notes is where a harness says which operations that applies to. Independent per implementation, as §11.1 makes the choice: the TypeScript core reads true (356 results, 178 of them #async), the Python core false — docs/10 declines companions because its frameworks cannot await in the value path.

environment.unicode_platform is recorded alongside crypto_backend for the same reason: the nfc-casefold-v1 normalizer’s output depends on the Unicode version the core folds with, and two cores on different versions can disagree on a stored index value without any vector noticing (docs/18 D-10(b)). As of 2026-08-23 the Python core folds with the interpreter’s table (CPython 3.14: Unicode 16.0) and the TypeScript core with a vendored one (17.0).

held_out mirrors MANIFEST.json’s held-out list, and is an empty array when the suite holds nothing out — which is the case from 0.6.0-provisional, so the block above is an illustration of the shape rather than a description of the current suite. (blind-index/argon2id.json filled this example until it was pinned on 2026-08-31; docs/07 §7.) A harness MUST iterate the manifest’s files and MUST NOT iterate held_out; the block is reported so that a held-out family is visibly not run rather than silently absent. status is not-run, never pass or skipped — “skipped” already means “this implementation does not claim that suite,” which is a different statement and one that a reader could mistake for a capability gap rather than a suite-integrity decision. An implementation MAY exercise a held-out family in its own development, and MUST NOT report the result here.

provisional_suites is true whenever any identifier in suites_supported falls in spec §4.8’s reserved 0xFF00–0xFFFF range. It exists so that a report cannot be quoted as evidence of conformance to a frozen format when no format has been frozen (PRD §8, Gate 0b).

Rules: fail > 0 ⇒ no level claimable ⇒ CI red. An out_of_band entry that is not pass counts as a failure for level-claim purposes on the same terms. claimed_levels must be consistent with the vector families passed (L0 requires every family the implementation’s suites reach; adapter levels additionally require the adapter’s own integration matrix green — adapters attach a coverage_matrix block mirroring the coverage-matrix table they publish, in the README or the adapter’s REFERENCE.md, so the claim and the docs cannot drift apart). environment.crypto_backend is recorded because FIPS conversations turn on it (PRD CL-9).

An adapter’s report (first built: adapters/prisma, 2026-08-31)

An adapter report is the same fieldseal-conformance/v1 document with a different centre of gravity, and the differences are worth stating because the first one built raised all of them:

• It claims no L0 and carries no results. An adapter contains no cryptography (AD-1, spec §11.3), so it runs no vector families; the families and the six mandatory pinned_decisions keys belong to the core beneath it. The report says so in harness_notes rather than leaving the empty results array to be interpreted.
• pinned_decisions becomes the adapter’s own list, under the “MAY add keys for pins of its own” clause. The Prisma adapter declares five, and the first is the one no core report can carry: codec-renderings, the mapping from a declared logical type to plaintext bytes. It was the adapter’s own decision until spec §3.6 pinned it (G25, #123), and a consumer that decoded one differently would decrypt successfully and read the wrong value — the one failure a round trip cannot catch. It is now pinned by the codec/ family under MANIFEST.adapter_files (docs/08 §4.8), and the entry records that the adapter runs it. The others pin the storage forms, the where sites L2 can verify, the L4 warm policy, and an error-code divergence between the two adapters that no vector can see.
• coverage_matrix is generated from the published table, not written to match it. The generator parses the adapter’s own coverage-matrix table, resolves each row’s cited test names against the run, and takes the row’s status from those tests. A row naming a test that no longer exists, or claiming behaviour and citing no test at all, fails the report — which is what “so the claim and the docs cannot drift apart” has to mean if it is to mean anything. On its first run it found a row (on_unindexable: "bucket" without the §7.2 ceremony) that claimed a refusal and pointed at a fixture instead of a test; the refusal was real and is now asserted.
• claimed_levels is boolean and some levels are not. Spec §10.1 marks Prisma’s L3 ⚠️ partial — tenant binding through a documented side channel — and there is no ⚠️ to record. It is recorded false, with the reason in harness_notes: under-claiming is the safe direction, and the note is what keeps false from being read as “not implemented”.

Row status is pass / fail / not-implemented (an honest ❌ row, e.g. L3-row binding) / unverified (a claim with no cited test — always a report failure). environment gains database and orm, since an adapter’s behaviour is a property of both.

Reports are uploaded as artifacts on every run; conformance.yml assembles the latest per-implementation reports into bench/conformance-summary.md (committed by CI on release tags only, so history is release-granular and the working tree stays quiet).

5. Release and versioning discipline

• Vector suite: semver, tagged vectors-vX.Y.Z. Additive = minor; retirement of a vector = major (consumers must re-check); no in-place edits ever (docs/08 §1).
• Cores/adapters: independent semver per package; each release records the vector-suite version it certifies against. A core release is blocked unless the cross job at that commit is green.
• Spec: already versioned (02-spec-v0.1.md → spec/ as versioned releases per repo layout). A spec change that alters bytes ⇒ new vectors ⇒ vector-suite major bump — the CONTRIBUTING.md chain (issue → citation → breakage statement → vectors) is enforced by review checklist, and mechanically by the fact that changed vectors fail every implementation until they’re updated in the same PR train.
• Toolchain pinning: CI pins language toolchains by version file (.python-version, .nvmrc, lockfiles committed). Nightly cross runs use floating latest-patch toolchains on purpose — divergence between pinned and floating runs is signal, not noise.

6. Non-CI conformance (third parties)

docs/ gains (Phase 1, DO-track) a short “Certifying an implementation” page: run the harness contract against a tagged vector suite, produce the §4 report, open a PR adding it under bench/third-party/. Listing requires: report with fail: 0, named contact, and the implementation being publicly available — mirroring M2’s spirit that the strongest signal is an implementation we didn’t write.

7. Explicitly not in CI

Benchmarks (bench/ is honest measurement, PRD DO-4 — CI machines produce noise, so benchmarks run on dedicated hardware with pinned specs and are published with methodology, not gated); coverage percentage gates (the vector suite and path matrices are the meaningful coverage measure); auto-formatting commits; scheduled dependency-bump auto-merges for cryptography/crypto-adjacent packages (human review required — supply-chain caution for a crypto project).

View source: docs/14-conformance-ci.md