Fieldseal Core Specification

Version: 0.1-draft · Date: 2026-08-08 · Status: Working draft. Not for production use. Not independently reviewed.

The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL are to be interpreted as described in RFC 2119 and RFC 8174.

Reviewer note. Every normative statement below carries its justification inline. Where the underlying literature is contested, or where a design choice was made against a plausible alternative, this is stated rather than hidden. Sections marked [OPEN] are unresolved and are listed in §13.

1. Scope

This specification defines:

1. A self-describing ciphertext envelope for a single field (cell) of a database record.
2. A registry of cipher suites, each frozen as a complete unit.
3. A key hierarchy and derivation scheme.
4. A blind-index construction for equality search, with a declared leakage budget.
5. A key-provider interface.
6. Conformance levels L0 through L4.

It does not define a wire protocol, a key-management service, a query language, or an ORM API. Adapter behavior is described normatively only where interoperability or safety depends on it (§10).

2. Threat model (normative)

An implementation MUST document its threat model in terms of this section. Claims beyond it are out of scope for conformance.

2.1 In scope

2.2 Explicitly out of scope

An implementation MUST state the following, in substance, in its user-facing documentation.

2.3 Logs are in scope as sensitive artifacts (normative)

An implementation MUST document, and a deployment SHOULD enforce, that database query logs, slow-query logs, audit logs, and replication logs are treated as sensitive artifacts subject to the same access controls as the ciphertext.

Justification. Grubbs, Ristenpart and Shmatikov (HotOS ‘17) showed that “logs, caches, and data structures kept by DBMS’s leak information that is not accounted for in the threat models used by the designers of encrypted databases,” with query text surviving in MySQL memory across thousands of subsequent operations. This was confirmed against a shipping product: the ETH Zurich analysis of MongoDB Queryable Encryption (USENIX Security ‘23) recovered 40–100% of field values purely from MongoDB’s queryLog and opLog — the opLog attack required zero client queries, only one inevitable compaction. A field-encryption design that ignores logs is not reviewable.

2.4 Relationship to storage-layer encryption

This specification does not replace full-disk, volume, or transparent database encryption. Deployments SHOULD retain those controls. They defend a different adversary (physical media loss) and cost almost nothing.

3. Envelope format

3.1 Layout

A conformant ciphertext is the concatenation:

+---------+----------+---------+----------+--------+------------+-------+--------------+
| fmt_ver | suite_id | key_id  | msg_seed | nonce  | ciphertext |  tag  | [commitment] |
|   1 B   |   2 B    |  16 B   |   32 B   | 12 B*  |    var     | 16 B* |    32 B*     |
+---------+----------+---------+----------+--------+------------+-------+--------------+
 \________________ header (51 B) ________________/

Fields marked * have sizes determined by suite_id; the values shown are for suite 0x0001.

3.2 Header authentication (normative)

fmt_ver, suite_id, key_id, and msg_seed MUST be included in the AEAD’s additional authenticated data (§6.2–§6.3). They MUST NOT be an unauthenticated prefix. (msg_seed is additionally self-authenticating: tampering with it changes the derived key and fails the tag check — §5.3.)

Justification. These fields drive key selection. Google’s own Tink documentation states that its 5-byte prefix “is not authenticated and cannot be relied on for security purposes” — it is a hint. An implementation that selects a decryption key from an unauthenticated field is inviting key-substitution steering. The AWS Encryption SDK authenticates its header; this specification follows that model.

3.3 Column storage type (normative)

Implementations MUST support storing the envelope in a binary column type (BYTEA, VARBINARY, BLOB). Implementations MAY support base64 text storage, and if they do, MUST document the resulting overhead.

Justification and honest cost. A 9-byte US Social Security Number under suite 0x0001 becomes 51 (header, including the 32-byte derivation seed) + 12 (nonce) + 9 (ciphertext) + 16 (tag) + 32 (commitment) = 120 bytes binary, or 160 bytes base64 — a 13.3× or 17.8× expansion. The fixed overhead is 111 bytes per encrypted field; across a 20-encrypted-column, 100-million-row table that is roughly 220 GB of envelope overhead alone, before index bloat and before WAL/replication amplification of the same. Base64 is a 33% tax paid on every row, forever.

This envelope is deliberately heavier than Tink’s 5-byte prefix. The extra bytes purchase authenticated key identification, key commitment, per-write derived keys, and context binding. A deployment that does not need those properties should not pay for them — and should not use this specification.

3.4 Detection

is_ciphertext(bytes) MUST return true only if the input is at least the minimum envelope length for a registered suite, fmt_ver is a recognized version, and suite_id is a registered suite. Recognition MUST be independent of the decrypt allow-list (§4.3): the allow-list governs authorization to decrypt (SUITE_NOT_ALLOWED), not recognition. If recognition consulted the allow-list, ciphertext under a retired suite would be misclassified as unmigrated plaintext in permissive read mode — returning ciphertext bytes as application data and, worse, re-encrypting them on the next write (double encryption, unrecoverable without forensics). is_ciphertext MUST NOT attempt to decrypt, and implementations MUST NOT infer the suite by trial decryption.

4. Cipher suite registry

4.1 Design principle (normative)

A suite_id names a complete, frozen suite: AEAD, nonce policy, KDF, and blind-index construction, as one indivisible unit. There MUST NOT be per-algorithm header fields. There MUST NOT be a caller-settable algorithm parameter.

Justification. This is deliberately the PASETO model rather than the JOSE model. The JWT alg header produced alg=none stripping, RSA→HMAC confusion, and kid abuse — all reducible to attackers controlling which validation rules the target follows. Adam Langley’s position (“have one option. Maybe two. Fight to keep it that small.”) and NIST CSWP 39 §3.2.3 (“if the integrity of algorithm selection during negotiation is not protected, the protocol will be subject to a downgrade attack”) point the same direction.

Data at rest has no peer and therefore no negotiation. The header is a declaration, not a negotiation. This dissolves the usual agility-versus-simplicity tension: there is nothing to downgrade to unless an implementation chooses to support it.

4.2 Registry

Implementations MUST support 0x0001. Implementations MAY support 0x0002. No more than two suites will be defined in v1.0.

4.3 Suite allow-listing (normative)

Decrypt-side suite policy MUST be a configured allow-list. An implementation MUST refuse to decrypt an envelope whose suite_id is not on the allow-list, even if it has code capable of doing so.

Justification. This is how a suite is retired without a downgrade window.

4.4 Nonce policy (normative)

For every encryption operation, including UPDATEs of an existing value, the nonce MUST be freshly generated from a CSPRNG. The nonce MUST NOT be derived from row identity. It MUST NOT be a counter. It MUST NOT be persisted.

Justification. NIST SP 800-38D §8 requires that the probability of IV+key reuse be no greater than 2⁻³², and states that “if even one IV is ever repeated, then the implementation may be vulnerable to the forgery attacks… In practice, this requirement is almost as important as the secrecy of the key.”

A database breaks every construction SP 800-38D permits:

The volume constraint is not solved by counting. It is solved structurally by per-write key derivation from the envelope’s random 32-byte msg_seed (§3.1, §5.3): no derived key ever encrypts more than one value, so the §8.3 ceiling is unreachable regardless of tenant size or row_id configuration. The random nonce is retained as defense in depth on top of key uniqueness.

4.5 Tag length (normative)

Authentication tags MUST be at least 128 bits. Tag truncation MUST NOT be supported.

Justification. NIST opened a second pre-draft comment period on revising SP 800-38D on 1 June 2026, proposing to “remove support for authentication tags whose lengths are less than 96 bits.” A specification fixing 128 bits is future-proof.

4.6 Key commitment (normative)

Every suite MUST provide key commitment, either by using a committing AEAD or by emitting an explicit 32-byte commitment value derived from the key. AAD binding alone MUST NOT be relied upon for this purpose.

Justification. None of AES-GCM, AES-GCM-SIV, or (X)ChaCha20-Poly1305 is key-committing. In a multi-key system where the header names a key ID and an adversary may influence which key is used, a single ciphertext can be crafted to decrypt validly under two keys (“invisible salamanders”), enabling partitioning-oracle attacks (Len–Grubbs–Ristenpart, USENIX Security ‘21; Bellare–Hoang, EUROCRYPT ‘22).

This is not theoretical for databases. AWS shipped security bulletin AWS-2025-032 on 17 December 2025 — “Key Commitment Issues in S3 Encryption Clients,” CVE-2025-14759 through -14764, across six language SDKs — remediating by “introducing the concept of ‘key commitment’ to S3EC where the EDK is cryptographically bound to the ciphertext.” Their advisory states: “There are no known workarounds.” AAD binds context; it does not bind key identity.

4.7 Prohibited constructions (normative)

An implementation MUST NOT support, and this specification will not register:

• Order-preserving encryption (OPE)
• Order-revealing encryption (ORE)
• Any range-queryable index over ciphertext
• Any LIKE, substring, or regex index over ciphertext
• Format-preserving encryption for the field cipher
• Tag lengths below 128 bits
• Runtime algorithm selection derived from stored data
• Any “auto-detect the algorithm” decoder

Justification for OPE/ORE. Grubbs, Sekniqi, Kolesnikov, Boneh and Ristenpart (S&P 2017), abstract verbatim: “attacks that recover 99% of first names, 97% of last names, and 90% of birthdates.” Against the successor CLWW scheme: 98% of first names and 97% of ZIP codes (BCLO managed only 12% on ZIP codes). Against Kerschbaum’s frequency-hiding scheme: top-10 first names 86%. Their conclusion: “the security benefits of deployed schemes is quite marginal.” Naveed–Kamara–Wright (CCS 2015) recovered, on real US hospital data under OPE: admission month, disease severity and mortality risk at 100% for 100% of the 200 largest hospitals; length of stay ≥99.77% of patients for 100% of hospitals.

Justification for FPE. Format-preserving encryption is excluded because it preserves the plaintext’s format and therefore its domain structure, which is precisely what the frequency-analysis results above exploit. Note that SP 800-38G Update 1 remains a current NIST recommendation — this specification’s exclusion is a design choice, not a claim that NIST has withdrawn FPE. SP 800-38G Rev. 1 is at second public draft (3 Feb 2025) and proposes dropping FF3, retaining only FF1.

5. Key hierarchy

5.1 Structure (normative)

  ROOT KEK                        (KMS / HSM — never leaves the boundary)
      │  wraps
      ├──────────────────────────────────┐
      ▼                                  ▼
  TENANT DEK                       TENANT INDEX KEY
  (wrapped at rest; cached         (sibling key, wrapped at rest;
   in memory, TTL + max-uses)       NOT derived from the DEK — §7.2)
      │  KDF(key_id ‖ msg_seed,          │  KDF(context), one key per
      │      context)                    │  (table, column, index)
      ▼                                  ▼
  RECORD KEY                       BLIND-INDEX KEY
  (per write; never stored)

An implementation MUST implement all three data-path tiers. A single global DEK MUST NOT be the default configuration. The tenant index key exists only where blind indexes are used.

Justification for separation of roles. NIST SP 800-57 Pt.1 Rev.5 §5.2: “In general, a single key shall be used for only one purpose.”

5.2 Tenant DEK granularity (normative)

The tenant DEK MUST be the crypto-shredding and blast-radius boundary. Deployments without a tenancy concept MUST still define a DEK scope (for example, per data-subject or per data-classification domain) and document it.

Justification. Granularity trade-offs:

Per-record KMS-generated keys are an anti-pattern at scale: AWS KMS shared cryptographic-operation quota is 100,000 req/s in the largest regions and 1,800 req/s, non-adjustable, for custom key stores. A per-row KMS call is architecturally impossible. Per-write derived keys give per-value isolation at zero KMS cost.

Index keys are siblings, not derivatives (normative). The tenant index key (§7.2) MUST be a distinct key wrapped by the KEK and MUST NOT be derived from the tenant DEK. Consequences: (a) rotating or re-encrypting data under a new DEK version never invalidates blind indexes — the failure mode that makes rotation and searchability mutually exclusive in Rails Active Record Encryption; (b) rotating an index key is a separate, explicitly costed operation requiring an index-column rebuild (§7.8); (c) crypto-shredding a tenant MUST destroy both keys, because an index value — a keyed hash of the plaintext — survives destruction of the encryption key alone.

5.3 Record key derivation (normative)

record_key = KDF(
    ikm     = tenant_dek,
    salt    = key_id ‖ msg_seed,
    info    = canonical_context(ctx),        // purpose = "encrypt"
    length  = suite.key_length
)

where KDF is the suite’s KDF, canonical_context is the length-prefixed encoding defined in §6.2, and msg_seed is the envelope’s random per-write derivation seed (§3.1). Because msg_seed is fresh on every encryption, every derived key is single-use — this, not row_id, is what makes key uniqueness structural.

The KDF MUST be one-way, and it MUST NOT be possible to determine one derived key from another (SP 800-57 §8.2.4).

5.4 Key update chaining is forbidden (normative)

An implementation MUST NOT derive a new key version from the value of a previous key version.

Justification, verbatim from SP 800-57 Pt.1 Rev.5 §8.2.3.2: “If the ‘value’ of the new key is dependent on the value of the old key, the process is known as key update… Key update could result in a security exposure if an adversary obtains a key in the chain… Federal applications shall not use key update.”

5.5 DEK caching (normative)

An implementation MUST provide an in-memory tenant-DEK cache with both:

• a max-age threshold, and
• a max-uses threshold that MUST NOT exceed 2³².

The cache MUST zeroize evicted key material. Where the platform supports it, the implementation SHOULD prevent the cache from being paged to swap (mlock or equivalent). Documentation MUST describe cache TTL as a security parameter, not a performance tuning knob.

Justification. AWS’s normative caching thresholds require a max-age > 0 and a max-messages value in 1..2³²; their guidance is “use the minimum amount of caching that is required to meet your cost and performance goals,” and their published per-tenant case study frames the trade-off exactly: “shorter TTLs reduce the window of exposure in the event of a memory dump, while longer TTLs reduce KMS call volume.” The 2³² max-uses bound is the same number as the SP 800-38D ceiling — the cache threshold is the nonce-safety control in any implementation that skips per-write derivation (which this specification does not permit — the bound is retained as defense in depth).

Honest limitation, which MUST be documented: an in-memory plaintext DEK cache is exposed to memory dumps, core files, and swap. This is a real, acknowledged residual risk.

5.6 Key versions (normative)

An implementation MUST support multiple simultaneously-decryptable key versions with exactly one version marked active-for-write. Rotation without this is a hard cutover and will cause an outage.

5.7 Cryptoperiods

Deployments SHOULD target the guidance in SP 800-57 Pt.1 Rev.5 Table 1: symmetric data-encryption keys, originator-usage period < 2 years, recipient-usage period < OUP + 3 years; symmetric master / key-derivation keys, about 1 year.

This specification does not claim NIST requires annual rotation. It does not. SP 800-57’s Table 1 preamble is explicitly non-binding, and §5.3.3.1 and §5.3.3.2 directly anticipate the re-encryption cost problem: “Cryptoperiods are generally made longer for stored data because the overhead of generating new keys and re-encrypting all data that was encrypted using the old keys may be burdensome,” and “In some cases, the costs associated with changing keys are painfully high. Examples include the decryption and subsequent re-encryption of very large databases.”

5.8 Rotation strategies

None of these strategies affects blind indexes: index keys are siblings of data keys (§5.2, §7.2), so data-key rotation at any tier leaves every index valid. Rotating an index key is a distinct operation requiring an index-column rebuild (§7.8).

An implementation SHOULD prefer on-write re-encryption plus a background sweep over on-read.

Justification. On-read re-encryption turns SELECTs into writes, breaking read-replica routing, read-only transactions, and query-planner expectations.

5.9 Re-encryption is the crypto-agility mechanism (normative)

The full re-encryption sweep of §5.8 is the mechanism by which a suite is retired. NIST CSWP 39 §5.1: “For the encrypted storage of data at rest, a mechanism must be established to handle encrypted user data when the encryption algorithm is to be replaced by a stronger one.” Agility without a re-encryption sweep is a claim, not a capability.

6. Context binding

6.1 What context is

FieldContext is the tuple supplied to every core operation:

FieldContext {
    suite_id     : uint16
    table_uuid   : bytes(16)      // stable surrogate, NOT the SQL table name
    column_uuid  : bytes(16)      // stable surrogate, NOT the SQL column name
    tenant_id    : bytes | null
    row_id       : bytes | null   // OPTIONAL — see §6.4
    purpose      : "encrypt" | "index:<index-id>"
}

table_uuid and column_uuid MUST be immutable surrogate identifiers, not SQL identifiers.

Justification. Binding to SQL names means a table or column rename renders every ciphertext undecryptable.

6.2 Canonical encoding (normative)

canonical_context(ctx) =
    u64be(len(suite_id))   ‖ suite_id
  ‖ u64be(len(table_uuid)) ‖ table_uuid
  ‖ u64be(len(column_uuid))‖ column_uuid
  ‖ u64be(len(tenant_id))  ‖ tenant_id
  ‖ u64be(len(row_id))     ‖ row_id        // omitted entirely if null
  ‖ u64be(len(purpose))    ‖ purpose

AAD(header, ctx) =
    u64be(len(fmt_ver))    ‖ fmt_ver
  ‖ u64be(len(key_id))     ‖ key_id
  ‖ u64be(len(msg_seed))   ‖ msg_seed
  ‖ canonical_context(ctx)

canonical_context covers only FieldContext fields, and is therefore well-defined both for encryption (§5.3) and for index derivation (§7.2), which has no envelope. The envelope-bound fields — fmt_ver, key_id, msg_seed — enter only the AAD. Naive concatenation MUST NOT be used.

Justification. Unlength-prefixed concatenation is forgeable across field boundaries. RFC 7518 §5.2 and Tink’s AES-CTR-HMAC both use explicit bit-length encoding for exactly this reason.

6.3 Dual-layer binding (normative)

canonical_context(ctx) MUST be used as the KDF info parameter (§5.3), and AAD(header, ctx) MUST be the AEAD’s additional authenticated data.

Justification. Binding at two layers means a context mismatch produces a wrong key (KDF layer) and a failed authentication (AAD layer) — a stronger failure and a clearer diagnostic than either alone.

6.4 row_id is OPTIONAL and defaults to absent (normative)

Implementations MUST support row_id = null. Implementations MAY support row_id binding; if they do, it MUST be off by default and enabled per-column by explicit configuration.

Justification — this is a significant and deliberate concession. Row binding is the correct defense against intra-database ciphertext swapping (copying row A’s encrypted salary into row B, or a value from the ssn column into the notes column where a different code path may expose it). But:

1. Rails, the most mature implementation in existence, omits AAD entirely. activerecord/lib/active_record/encryption/cipher/aes256_gcm.rb literally sets cipher.auth_data = "", because ActiveModel::Type has no access to the record. Mandating row binding would place the reference implementation of the pattern out of conformance.
2. The primary key is not available at INSERT time in most ORMs. With database-generated identity keys, the PK is NULL when the value transform runs in Django (pre_save during INSERT compilation), SQLAlchemy (before_insert), EF Core (SavingChanges, where it is a temporary value), GORM (populated from RETURNING afterwards), and Prisma (the extension runs before the query). Only Hibernate with a SEQUENCE, TABLE, or assigned/UUID generator has it. Client-generated UUIDv7 primary keys are the general solution, and deployments enabling row binding SHOULD adopt them.
3. It converts data-migration bugs into decryption failures. Any legitimate PK change, tenant migration, table rename, or shard resplit breaks every affected ciphertext.

Implementations supporting row binding MUST document a re-binding procedure for legitimate migrations, and MUST distinguish AAD_MISMATCH from TAG_INVALID in errors (§9).

6.5 AAD contains nothing secret (normative)

No component of FieldContext may be a secret or a sensitive value.

Justification. AAD is reconstructible from schema plus row identity and, in KMS-mediated designs, is logged. AWS states plainly of encryption context: “The encryption context is not secret and not encrypted. It appears in plaintext in AWS CloudTrail Logs… Because the encryption context is logged, it must not contain sensitive information.”

6.6 Record-level MAC (RECOMMENDED)

Implementations SHOULD offer an optional MAC computed over all protected fields of a record together.

Justification. Per-field AAD detects substitution of one field’s ciphertext for another’s. It does not detect a deleted field. The AWS Database Encryption SDK signs the canonicalization of the material description, encryption context, and every encrypted-or-signed field, specifically so it can “detect unauthorized changes to the item as a whole, including adding or deleting attributes, or substituting one encrypted value for another.” This is the strongest published model.

7. Blind indexes

7.1 Purpose and hard limits (normative)

A blind index supports equality and membership (IN) lookups only. It MUST NOT be used for ordering, ranges, prefix matching (except via the explicit derived-field mechanism of §7.9), substring matching, or full-text search.

7.2 Construction (normative)

index_key  = KDF(ikm   = tenant_index_key,        // sibling of the tenant DEK — §5.2
                 salt  = "fieldseal-index-v1",
                 info  = canonical_context(ctx with purpose="index:<index-id>",
                                           row_id = null),
                 length= 32)

raw        = IDF(index_key, normalize(plaintext))

blind_index = truncate(raw, b bits)

where:

• normalize is a declared, deterministic transformation (for example Unicode NFC + case folding for email addresses). It MUST be declared per column and MUST NOT change after writes begin.
• IDF is the index derivation function, selected per §7.3.
• b is the truncation length in bits, selected per §7.4.

truncate(raw, b) is defined bit-exactly: keep the first ⌈b/8⌉ bytes of raw, then set the trailing 8·⌈b/8⌉ − b bits of the final byte to zero. Bits are numbered MSB-first within each byte (the most significant bit is bit 0, mask 0x80); equivalently, interpret raw as a bit string in network order, keep the first b bits, and zero-pad to the byte boundary. The output length is exactly ⌈b/8⌉ bytes. Example: truncate(0xABCD…, 12 bits) = 0xABC0. The representation in which that value is written to the database is defined in §7.11.

Justification. Either bit-order convention is cryptographically equivalent: the IDF output is uniform, so which b bits survive does not change the §7.4 leakage analysis. The convention is pinned solely so that independent implementations write byte-identical index values into a shared database — an interoperability decision of exactly the kind the vector suite exists to verify (§12). Leading-bits/MSB-first is chosen because the truncated value is then a byte-prefix of the untruncated output, which simplifies debugging, and because it matches the network-byte-order conventions used elsewhere in this specification (§3.1 big-endian suite_id, §6.2 u64be).

The index key MUST be distinct per (tenant, table, column, index). It MUST NOT be the field encryption key. Two indexes MUST NOT share a key. Distinctness per index is achieved by the index identifier carried in purpose (index:exact for the default equality index; a §7.9 prefix index declares its own identifier).

The tenant index key is a sibling of the tenant DEK under the KEK, and MUST NOT be the tenant DEK or be derived from it (§5.2). Deriving index keys from data keys would make every data-key rotation silently invalidate every blind index — precisely the rotation-versus-searchability trap this design exists to escape.

Justification. SP 800-57 §5.2 (“a single key shall be used for only one purpose”). Both CipherSweet and blind_index implement per-index key separation; CipherSweet’s design note is explicit: “Each blind index on each column uses a distinct key from your encryption key and each other blind index key.” Cross-column shared keys additionally merge frequency-analysis populations, which is the opposite of what §7.6 requires.

7.3 Index derivation function selection (normative)

Justification. If the index key leaks, an HMAC index over an enumerable domain is invertible by brute force in seconds. Paragonie’s analysis of the chosen-plaintext case is direct: an attacker “can iterate every possible value as a user and then correlate with the resultant blind index value.” blind_index defaults to Argon2id for this reason.

Honest cost, which MUST be documented: Argon2id at 4 iterations / 32 MiB costs roughly 10–100 ms per query term. That is a hard ceiling on query rate and it is a product constraint, not a tuning detail.

7.4 Truncation length (normative)

Let P be the projected number of distinct values in the column (not the number of rows). P MUST be ≥ 16.

b MUST be chosen such that:

2 ≤ P × 2^(−b) < √P

b MUST be rounded down. Both b and the projected P MUST be recorded in schema metadata. The bit-level semantics of truncate(raw, b) are defined in §7.2.

Justification and honest caveat. This is the AWS beacon-length band. Worked example from AWS’s documentation for P = 100,000: recommended range 8–15 bits. At 16 bits, 1.5 collisions per value, “66% likely same plaintext,” retrieve ~15 records per 10 wanted. At 14 bits, 6.1 collisions, “33% likely same plaintext,” retrieve ~30 per 10. “For every bit below 15, the performance cost and the security double.”

These are AWS engineering heuristics, not peer-reviewed leakage bounds, and AWS hedges on them: “Beacon length only estimates the average number of false positives produced. The more unevenly distributed your dataset, the less effective beacon length is.” Implementations SHOULD ship a tool that measures actual distribution skew rather than trusting the formula.

7.5 Application-side re-verification (normative)

A blind index MUST be treated as a filter, never as an answer. After retrieving candidate rows by index, the implementation MUST decrypt and compare the actual values before returning results to the caller.

Documentation MUST state that pagination built directly on an indexed encrypted column is incorrect: the correct pattern is over-fetch → decrypt → filter → paginate.

7.6 Default-deny cardinality gate (normative)

An implementation MUST refuse, by default, to create a blind index on a column whose declared domain has fewer than 2¹⁰ distinct values, or which is declared as heavily skewed. Override MUST require an explicit, logged, reviewed declaration in configuration.

Categories that MUST be gated: booleans, enumerations, sex/gender, region, US state, diagnosis and test-result codes, risk tiers.

Justification. Naveed–Kamara–Wright (CCS 2015) on real US hospital data under deterministic encryption recovered mortality risk and patient death for 100% of patients in ≥99% of the 200 largest hospitals, and disease severity for 100% of patients in ≥51%. Microsoft’s own Always Encrypted documentation warns: “unauthorized users might guess information about encrypted values by examining patterns in the encrypted column, especially if there’s a small set of possible encrypted values, such as True/False, or North/South/East/West region.” AWS is blunter: attributes “with very small populations or highly imbalanced binary outcomes — such as medical test results where NEGATIVE values dominate — cannot be protected using truncation alone.”

7.7 Correlated columns (normative)

Two blind indexes MUST NOT be created over correlated fields.

Justification, verbatim from AWS: “We strongly recommend that you avoid constructing distinct beacons from fields with correlated values” — the canonical example being City and ZIPCode, because “an unauthorized user can easily identify which results are false positives,” stripping out exactly the protection truncation was providing.

7.8 Immutability after first write (normative)

Once any row has been written with a given blind index, its IDF, normalize function, truncation length b, key-derivation context, and stored representation (§7.11) MUST NOT change. Changing any of them requires creating a new index column and performing a full backfill.

7.9 Prefix indexes (OPTIONAL, gated)

Prefix matching MAY be supported only via an explicitly declared derived field (for example, “first 3 characters, NFC-normalized, case-folded”), treated as its own independent index subject to every rule in this section.

Documentation MUST state that every prefix length is an additional leakage channel and that prefixes are lower-entropy than full values. Enabling a prefix index MUST require an explicit written risk acceptance recorded in configuration.

7.10 Honest statement of what is not supported (normative)

Documentation MUST reproduce this table.

7.11 Index column storage type (normative)

The stored form of a blind index MUST be the raw truncated bytes produced by §7.2 — length exactly ⌈b/8⌉, no length prefix, no padding, no encoding — written to a binary column type (BYTEA, VARBINARY(⌈b/8⌉), BLOB).

Implementations MAY support a lowercase hexadecimal alternative for text-only storage paths: exactly 2·⌈b/8⌉ characters, digits 0–9 and a–f, no 0x prefix and no separators. Where it is supported, the representation MUST be declared per index column and is immutable after first write (§7.8).

Index comparison MUST be exact byte equality, or for the hexadecimal alternative exact string equality. Implementations MUST create the index column with a binary or otherwise case- and accent-sensitive collation, and MUST NOT rely on the database’s default collation.

Justification. This is the §3.3 rule applied to the index column, and it exists for the same reason: two implementations sharing one database must write byte-identical values or equality matching silently fails. That failure mode is worse than an envelope divergence, because it produces no error — a Python-written index simply never matches a Node-issued WHERE, and the query returns zero rows as though the value were absent. No envelope vector detects it, which is why §12 requires the stored form to be asserted directly. The collation requirement is ordinary SQL behavior rather than a cryptographic claim: under a case-insensitive collation a hexadecimal column equates AB and ab, so the database would match two values the core treats as distinct, widening the §7.4 collision band by an amount the leakage analysis never accounted for. Raw bytes are the MUST rather than hex because hex doubles a column that sits in every index page and every WHERE clause on the table.

8. Key provider interface

interface KeyProvider {
    // Returns the tenant DEK and its key_id for a write.
    encryption_key(ctx: FieldContext) -> (key: bytes, key_id: bytes[16])

    // Returns candidate keys for a read, in preference order.
    decryption_keys(header: EnvelopeHeader) -> [bytes]

    // Optional: async prefetch. MUST NOT be required for correctness.
    warm(contexts: Iterable<FieldContext>) -> void
}

Implementations MUST provide at minimum:

• StaticKeyProvider — a single key. Test and development use only; MUST emit a warning outside test configuration.
• DerivedKeyProvider — keys derived from a root secret via an approved KDF.
• EnvelopeKeyProvider — KMS-wrapped DEKs with the cache of §5.5. This is the production path.

decryption_keys MUST return keys for all currently-valid versions, enabling §5.6.

When ctx.purpose names an index derivation (§7.2), encryption_key MUST return the tenant index key, never the tenant DEK. Blind-index values carry no envelope; the key and parameters that produced them are identified by schema metadata (§7.8) — which is why §7.8’s immutability rule exists.

8.1 KMS availability is a hard dependency (normative)

Documentation MUST state that the key service becomes a hard dependency in the read path of every query touching an encrypted field, and MUST document the deployment’s degradation mode: fail-closed (return an error) or serve-cached (serve only what the cache can decrypt).

Justification. AWS states this more bluntly about external key stores than most vendors state about their own products: “The greater risk to availability and latency will, for most customers, exceed the perceived security benefits of external key stores”; “If you temporarily revoke access to your external key manager… ciphertext encrypted under your KMS keys can’t be decrypted. If you permanently revoke access… all ciphertext encrypted under a KMS key in your external key store becomes unrecoverable.” NIST names the same risk from the other side (SP 800-57 §5.3.2): “short cryptoperiods may be counter-productive, particularly where denial-of-service is the paramount concern.”

8.2 Key destruction is unrecoverable data loss

Implementations MUST NOT provide a key-destruction API without a configurable delay window and an explicit confirmation step. Google Cloud KMS’s 30-day “scheduled for destruction” window exists precisely because operators do this.

9. Errors

An implementation MUST distinguish at least these error types, and MUST NOT collapse them into a single “decryption failed”:

MODE_VIOLATION is raised at the API boundary, before any cryptographic processing or key acquisition begins, and therefore sits outside the decrypt-path error ordering. Its message MUST name both the rejected operation and the active mode: a mode violation is a deployment-configuration error, and an implementation that reports only “operation not permitted” sends the operator looking in the wrong place. One code covers all present and future modes rather than one code per mode, so that adding a mode is not a breaking change to the error taxonomy.

Error messages MUST NOT include plaintext, key material, or derived key values.

10. Conformance levels

Each level is independently claimable. An implementation MUST state which levels it claims, per ORM adapter.

L0 — Envelope conformance (REQUIRED for any conformance claim)

Envelope format, suite registry, key IDs, previous_schemes decryption chain, strict/permissive/readonly read modes, and the full test-vector suite. Achievable in any environment, including a bare encrypt(value) helper with no ORM integration at all.

L1 — Transparent value mapping

Read and write transforms installed at the ORM’s type layer, covering single-row writes, ORM-managed bulk insert, and ORM-managed bulk update.

Achievable in: Django, SQLAlchemy, Prisma, TypeORM, Hibernate, EF Core, GORM — with documented carve-outs (§10.2).

L2 — Indexed equality

Every registered suite is randomized (§4.2), so ciphertext equality never reflects plaintext equality. L2 therefore always requires the blind-index column of §7. What varies per ORM is how a query reaches that column:

• (a) Index-typed property. The index column is declared as its own property whose type-layer transform derives the blind index from a plaintext parameter. WHERE email_bidx = :plaintext then works through the ORM’s ordinary parameter conversion — no query rewriting, but the query surface is explicit (callers name the index property).
• (b) Transparent rewrite. Predicates on the encrypted property are rewritten onto the index column at query-compile time.

(a) available in: Django, SQLAlchemy, Hibernate (HQL/criteria parameter conversion), EF Core, TypeORM (find-options and FindOperator values only — string-condition where() bypasses transformers). (b) achievable with real work in: Django (Lookup compiling a sibling Col), SQLAlchemy (hybrid_property + Comparator), EF Core (IQueryExpressionInterceptor), GORM (clause.Where tree rewrite), Prisma (args-tree path rewriting, with the §10.2 mandatory throws). (b) not available in: TypeORM (no predicate-rewrite extension point) and Hibernate (no supported comparator hook; StatementInspector sees SQL text but not parameter values — CompositeUserType gives a type-safe explicit surface, which is (a), not (b)).

L3 — Context binding

Requires sibling-field access in the value path.

Cleanly available in: GORM (Value(ctx, field, dst reflect.Value, …) receives the whole struct), Hibernate (Interceptor.onPersist receives a mutable state array plus the entity). Available with a documented side channel (contextvar / AsyncLocal / CLS set by an earlier hook of the same operation): SQLAlchemy, Django, EF Core, Prisma. Tenant-only, via ambient request-scoped CLS (no per-operation hook covers all write paths): TypeORM. Not achievable: Rails (ActiveModel::Type has no record access), Sequelize getters.

L3-row (binding to row_id) is a separate sub-level, achievable only in Hibernate with sequence/assigned generators, or anywhere with client-generated primary keys.

L4 — Async key acquisition in the value path

Available in: Prisma ($allOperations is async), EF Core (SavingChangesAsync, *ExecutingAsync — write path only, not materialization), GORM (context-aware blocking). Not available in: Django, SQLAlchemy (attempting to await inside a type processor raises MissingGreenlet), TypeORM, Hibernate, Rails, Sequelize.

This is why the core API is synchronous (§11.1). An async-first core would be unimplementable in the majority of target ORMs.

10.1 Level matrix

10.2 Mandatory adapter carve-outs (normative)

Each adapter MUST publish a coverage matrix stating exactly which write, read, and query paths it intercepts. Where a path is not intercepted and would silently write plaintext or silently return wrong results, the adapter MUST throw, not degrade silently.

Known cases requiring an explicit throw:

• GORM: map-based Updates(map[string]interface{}{...}) and single-column Update("col", v) bypass the serializer entirely and write plaintext. Verified in callbacks/update.go. The adapter MUST intercept or reject these.
• Prisma: where.field.in: [...], contains:, and startsWith: are not rewritten by the path-surgery approach; the value is encrypted instead, silently returning zero rows with no error. The adapter MUST reject these filter shapes over encrypted fields. orderBy on an encrypted field MUST throw rather than being silently dropped.
• TypeORM: the dirty-check (SubjectChangedColumnsComputer) runs the transform, and every registered suite is randomized, so every save() marks every encrypted column dirty and rewrites it with a fresh envelope. The adapter MUST document this spurious-rewrite behavior — it is correct, but it inflates UPDATE volume and WAL. Equality is available only through the explicit index-typed property (L2 (a), find-options); the adapter MUST NOT claim transparent L2 (b). A deterministic AEAD suite that would lift both constraints is deliberately not in the v0.1 registry — see §13.6.
• All ORMs: raw SQL parameters are never encrypted by any ORM surveyed. Django and GORM decrypt raw results; nobody encrypts raw parameters. Adapters MUST document this.
• All ORMs: application-level caches (Django’s django.core.cache, Hibernate’s second-level cache, EF Core second-level cache interceptors, expire_on_commit=False session identity maps) hold plaintext. Adapters MUST document which caches are affected and how to exclude encrypted fields. Hibernate’s UserType.disassemble() is the correct place to keep ciphertext in the L2 cache; AttributeConverter gives no such control, which is a reason to prefer UserType.

10.3 Read modes (normative)

A mode fixes two independent behaviors: what a read does with non-envelope input, and whether operations that produce ciphertext for storage are permitted. Both are specified for every mode.

In both permissive and readonly, implementations MUST warn when the mode is active and SHOULD emit a metric counting plaintext reads.

The ciphertext-producing operations are encrypt() and rotate() (§11.1). decrypt(), is_ciphertext(), and blind_index() are permitted in every mode. blind_index() in particular MUST NOT be refused in readonly: an index value is required to construct a query, not to write one, and a mode that could not compute one would be unable to look anything up — which is the entire purpose of a read-only client.

Justification, with its cost stated. readonly inherits permissive’s pass-through behavior rather than strict’s raise because the mode exists for migration and rollback windows, which are exactly the windows in which unmigrated plaintext is expected to be present; a readonly client that raised on it would fail precisely where it is meant to be deployed.

The alternative considered was making the two axes independently configurable. Three named modes cover three of the four combinations the axes allow, and the omitted one — raise on non-envelope input and refuse writes — is a legitimate configuration, not an absurd one: a fully migrated production read replica would reasonably want both. This specification does not currently offer it. That is a real limitation and is recorded here rather than glossed, on the reasoning that a fourth mode is cheap to add later once a deployment demonstrates the need, whereas an orthogonal-knob configuration surface is not cheap to remove once implementations and vectors depend on it. A deployment that needs the combination should raise a specification issue rather than adding a local option, because read-mode behavior is observable in the shared error vectors and a private fourth mode would put two implementations quietly out of agreement.

11. Core library API

11.1 Synchronous primary API (normative)

encrypt(plaintext: bytes, ctx: FieldContext) -> bytes          [SYNC]
decrypt(ciphertext: bytes, ctx: FieldContext) -> bytes         [SYNC]
blind_index(plaintext: bytes, ctx: FieldContext) -> bytes      [SYNC]
is_ciphertext(value: bytes) -> bool                            [SYNC]
rotate(ciphertext: bytes, ctx: FieldContext) -> bytes          [SYNC]

These MUST be synchronous and MUST NOT perform network I/O.

Justification. See L4 in §10. Django field hooks, SQLAlchemy type processors, TypeORM transformers, Hibernate converters, Rails ActiveModel::Type, and Sequelize getters are all synchronous by signature. A blocking KMS call inside any of them holds a pooled database connection inside an open transaction — a 20 ms round-trip during a flush of 1,000 rows holds the connection for 20 seconds and exhausts the pool under load.

11.2 Asynchronous prefetch

warm(contexts: Iterable<FieldContext>) -> void                 [ASYNC OK]

All KMS interaction happens here or in a background refresh task. Never in the value path.

11.3 Separation of concerns (normative)

The core MUST NOT know about rows, columns, SQL, or ORMs beyond the opaque identifiers in FieldContext. Adapters MUST NOT contain cryptographic code. An adapter’s only permitted calls into the core are the five functions in §11.1 plus warm.

12. Test vectors

An implementation MUST pass the full vector suite to claim any conformance level. Vectors are JSON, in /vectors, and MUST cover:

• Every registered suite: encrypt/decrypt round trip with fixed key, derivation seed, nonce, and context
• Key derivation: fixed tenant DEK + key_id + msg_seed + context → expected record key
• Index-key derivation: fixed tenant index key + context → expected per-index key
• Canonical context and AAD encoding: byte-exact output for representative contexts, including row_id present and absent
• Blind index: fixed key + plaintext + normalization + truncation → expected index, for both Argon2id and HMAC IDFs — including at least three vectors whose truncation length is not a multiple of 8 (the §7.2 final-byte masking is observable only there) and one multiple-of-8 control
• Blind index storage: for every blind-index vector, the exact bytes written to the database per §7.11, and the hexadecimal form where an implementation supports it — asserted as its own field, because a stored-form divergence produces no error, only an empty result set
• Commitment values
• Every error case in §9, including deliberately malformed envelopes — and for the §10.3 modes, encrypt() under readonly raising MODE_VIOLATION alongside the positive controls that bound it: decrypt() of a valid envelope, blind_index(), and non-envelope input, each under readonly
• Cross-implementation: ciphertext produced by implementation A decrypted by implementation B. CI MUST fail on divergence. This is the only test that proves the specification’s central claim.

13. Open questions

13.1 Profile the AWS structured-encryption format, or define fresh? aws/aws-database-encryption-sdk-dynamodb/specification/structured-encryption/ is a normative, RFC-2119 specification written against generic “Structured Data” with Terminal fields, not DynamoDB types. Profiling it buys interoperability with a shipping implementation and reduces novelty risk. Defining fresh buys freedom from DynamoDB item semantics and AWS KMS assumptions in the key-provider model. This is the highest-leverage unresolved decision and should be settled before any code is written.

13.2 Which FIPS-approvable AEAD for suite 0x0001?

The registry currently names AES-256-GCM plus an explicit commitment. AES-CBC-HMAC is committing natively and is what CipherSweet’s FIPSCrypto backend and JOSE’s A256CBC-HS512 use, at 20–36 bytes more per field. Caveat: “FIPS-approved” for the composite rests on composing two approved primitives rather than a NIST-defined AEAD mode — a validation-strategy argument, not a CAVP algorithm listing. Confirm with a testing lab before asserting it to a buyer.

13.3 Should the registry reserve space for a NIST accordion mode? NIST announced (6 June 2025) development of cryptographic accordion modes based on HCTR2 for future SP 800-197x publications. These are wide-block, tweakable, and are the eventual standards-track answer to precisely this problem. The suite registry should be structured to absorb one.

13.4 Non-relational stores. Document stores, key-value stores, and search indexes have the same problem with different mechanics. Probably out of scope for v1.0.

13.5 Vector and embedding encryption. As RAG stores become PII repositories, encrypting embeddings while retaining approximate-nearest-neighbor search becomes a real gap. IronCore’s distance-preserving vector encryption is the only shipping option found and it is paid. Genuinely early; noted as future work.

13.6 Should the registry add a deterministic AEAD suite (AES-SIV, RFC 5297)? TypeORM’s transformer cannot support a blind-index column transparently, and its dirty-check makes randomized encryption rewrite every encrypted column on every save() (§10.2). A deterministic suite would let ciphertext equality work through its existing type layer and would quiet the dirty-check — at the cost of moving the §7 leakage controls (cardinality gate, truncation, per-index keys) into the suite itself, where they are harder to enforce, and of deterministic ciphertext leaking equality by construction. AES-SIV is not FIPS-approved (composition arguments over CMAC+CTR notwithstanding), and a third suite strains the “one option, maybe two” commitment of §4.1. Deferred; revisit only with concrete adapter-demand evidence.

14. Contested claims — reviewer flag list

Stated explicitly so a reviewer does not have to find them.

1. Range-reconstruction attack severity under realistic workloads is contested. KKNO'16, LMP'18 and GLMP'18–‘19 assume uniform or well-behaved query distributions. Kamara et al.’s LEAKER SoK (EuroS&P ‘22) found IKK achieved <15% recovery on real logs while SUBGRAPH was stronger than reported. Bindschaedler et al. (VLDB ‘18) show results are sensitive to auxiliary-data modeling. Not contested: OPE/ORE and deterministic-on-low-entropy attacks, which require no query observation at all — which is why §4.7 and §7.6 are unconditional.
2. Whether key destruction satisfies GDPR Art. 17 is genuinely unsettled. EDPB Guidelines 02/2025 (v2, July 2026) ¶51 states “encrypted personal data is still personal data” and conditions the argument on algorithm strength, key non-leakage and time; ¶104 says it is “not advisable” to register personal data in encrypted form where erasure may be required. CJEU C-413/23 P (EDPS v SRB, Sept 2025) cuts the other way on the relativity of personal data. This specification does not claim key destruction is erasure.
3. NIST IR 8547 is still an initial public draft (Nov 2024) as of August 2026. Cite as draft. Its position that symmetric standards need no PQC transition is the basis for scoping §5 to key wrapping rather than the field cipher; CNSA 2.0 (final) supplies the hard dates.
4. SP 800-38D is under active revision. A second pre-draft comment period closed 31 July 2026. Fixing tags at 128 bits (§4.5) is future-proof.
5. AES-GCM-SIV’s FIPS status could change if the accordion-mode program delivers an approved misuse-resistant mode. See 13.3.
6. Beacon-length formulas (§7.4) are AWS engineering heuristics, not peer-reviewed leakage bounds. AWS hedges on them explicitly. Treat as a starting point requiring dataset-specific evaluation.
7. The HHS breach safe harbor’s technical basis is stale. It points at NIST SP 800-111 (2007, Guide to Storage Encryption Technologies for End User Devices — scoped to end-user devices, not servers) and says “FIPS 140-2 validated,” while all FIPS 140-2 certificates move to the CMVP Historical List on 22 September 2026. This is a weakness in the cited authority, not in the design, and should be stated rather than papered over.

15. Normative references

• RFC 2119 / RFC 8174 — requirement keywords
• NIST SP 800-38D — GCM/GMAC, §8 IV uniqueness, §8.3 invocation limits
• NIST SP 800-57 Part 1 Rev. 5 — key management; §5.2 single-purpose keys, §5.3.6 Table 1 cryptoperiods, §8.2.3.2 key-update prohibition, §8.3.4 key destruction
• NIST SP 800-56C Rev. 2 / SP 800-108 Rev. 1 — key derivation
• NIST SP 800-88 Rev. 2 — media sanitization; §3.1.2, §3.2.2 cryptographic erase preconditions
• NIST CSWP 39 — crypto agility; §3.2.3 downgrade, §5.1 data-at-rest algorithm replacement
• FIPS 140-3 and the CMVP transition schedule
• RFC 7518 §5.2 — AAD length encoding for encrypt-then-MAC
• RFC 8452 — AES-GCM-SIV
• CNSA 2.0 — AES-256 requirement and transition dates

16. Informative references

AWS Database Encryption SDK structured-encryption spec · AWS beacons and beacon length · AWS encryption context · AWS external key stores · AWS-2025-032 · Tink wire format and bind ciphertext to context · Vault Transit · Rails Active Record Encryption · CipherSweet · ankane/blind_index · Always Encrypted · Naveed–Kamara–Wright CCS'15 · Grubbs et al. S&P'17 · Why Your Encrypted Database Is Not Secure, HotOS'17 · MongoDB QE analysis, USENIX'23 · LEAKER SoK, EuroS&P'22 · Partitioning Oracle Attacks, USENIX'21 · Langley on crypto agility · Soatok on versioned protocols · EDPB Guidelines 02/2025 v2

View source: docs/02-spec-v0.1.md