Durable product records

Infrahub Sync provides one storage contract for compact product records and immutable artifacts. It is independent of Prefect: a standalone run has no Prefect links, and a managed run can retain any number of purpose-labelled execution links after Prefect no longer has the corresponding flow-run detail.

The managed Sync HTTP API and its worker use the local profile for durable run, plan, result, artifact, execution-link, mutation-receipt, and audit integration. Standalone CLI and public Python operations can opt into the same product-run and plan-review contract by setting an explicit product-cache location. Their established run-directory and saved-plan cache remains the execution artifact source.

For Python, pass product_cache_location on each version 1 request. For the CLI, pass --product-cache-location to diff, sync, and apply; the same option is available when reviewing a saved plan through diff --from-plan. The path must be absolute after user expansion, an unresolvable ~user is rejected as invalid configuration, and every stage continuing a plan must use the same location.

This is minimum product-projection configuration. It does not register or version a configuration package, select a new provider type, or add a release-management API.

For configured direct sync, the immutable review artifact is published after the saved plan commits and before the first destination write, in both serial and tiered execution. If publication fails, synchronization stops before contacting the destination write surface and retains typed failure evidence on a terminal failed ProductRun. A reserved artifact remains unavailable until an exact publication retry completes it.

Provider profiles

The local profile requires an explicit cache directory and creates a SQLite database plus an artifact directory beneath it:

from pathlib import Path

from infrahub_sync.product_store import local_product_projection

records = local_product_projection(Path("/var/lib/infrahub-sync/product-cache"))

The path must be absolute after ~ expansion. Relative paths are rejected, and the profile never falls back to the process working directory.

The production profile combines a PostgreSQL DB-API connection factory with an S3-compatible client implementing the small S3Client protocol (put, get, copy, and delete):

from infrahub_sync.product_store import production_product_projection

records = production_product_projection(
    connect=postgres_connection_factory,
    s3_client=object_client,
    bucket="sync-product-artifacts",
    prefix="production",
)

Database-driver and object-client packages remain deployment choices rather than base Infrahub Sync dependencies. Both profiles implement the same ProductProjection operations and are covered by the same behavioral contract tests. The production-profile tests use a PostgreSQL-paramstyle local DB-API emulator and a deterministic S3-compatible client; they prove the injected provider contract, not live PostgreSQL, driver, S3 service, authentication, or network integration.

Use local_product_projection or production_product_projection to construct a supported profile. The record-store and artifact-store protocols accepted by ProductProjection are internal implementation seams, not a public custom-provider compatibility contract. S3Client remains the documented deployment adapter required by the production factory.

Constructing the supported production projection runs idempotent CREATE TABLE IF NOT EXISTS statements for the product-record schema. The PostgreSQL connection role must have the DDL privileges required to create those tables in its configured schema, in addition to the privileges required to read and write their rows.

Record and identity contract

ProductRun owns the stable Sync run_id, requested operation, immutable configuration reference, actor and audit links, product phase and outcome, timings, summary and results, artifact references, and Prefect correlations. Relational child tables store artifact references and Prefect execution links separately from the compact run row.

Each PrefectExecutionLink records:

• a distinct flow_run_id;
• an optional deployment_id;
• its purpose or stage;
• its attempt number;
• optional last-observed state and timestamp.

Duplicate Sync run IDs and duplicate flow-run IDs within one record are rejected. A confirmed sync creates its own record. A reviewed-plan apply instead advances the original planning record and attaches its result artifacts to the same run_id; it does not create a second Sync identity. add_prefect_execution appends stage and retry links as they become known without changing the Sync record's identity.

Mutations against a missing Sync run raise RunNotFoundError. Read operations continue to return LookupResult with reason="run-not-found", so absence remains a normal, non-exceptional lookup result.

create_run accepts an unfinished ProductRun, including initial actor, audit, summary, results, phase, and Prefect-link metadata. It rejects records that already have a finish timestamp, outcome, or artifact reference; those completion fields must be added through the publication and finish operations so their integrity checks cannot be bypassed.

Managed mutations reserve a MutationReceipt unique by actor and SHA-256 digest of the client idempotency key. A receipt binds the operation, target, request fingerprint, reason, Sync run, opaque Prefect key, state, and exact accepted response. The raw client key is not stored. Run creation commits its receipt and unfinished product run in one relational transaction. AuditEvent records secret-safe actor, reason, operation, and outcome evidence for accepted mutations and refusals.

record_results updates retained result evidence without changing product phase, outcome, or finish time. Managed verification uses this operation because verification is read-only for both the destination and product lifecycle.

Artifact publication and lookup

Artifact keys contain their SHA-256 digest and never change. Publication first reserves a non-readable relational reference for the run-owned artifact identity. It then writes artifact data and its manifest, and finally marks that exact reference published in a second relational transaction. The filesystem profile commits data and manifest using an atomic directory rename. The S3-compatible profile copies staged data to its immutable key and uses a create-only manifest put as the object-store commit point. A crash before the final relational mark leaves durable pending evidence rather than exposing the artifact. The run cannot be finished successfully while any pending publication exists. It can be finished with outcome="failed" so publication failure evidence reaches a safe terminal state while the pending artifact remains unavailable.

An S3Client implementation must make put(..., if_absent=True) an atomic, create-only write and translate an existing-key conflict to DuplicateArtifactError. This prevents a racing publisher from replacing the immutable manifest.

To recover from an interrupted publication, retry publish_artifact with the same run and artifact IDs, content, kind, and media type. Matching is performed after secret redaction. If data and manifest are already complete, the retry verifies them and performs only the missing relational mark. If the manifest is absent, the retry resumes object publication and then marks the reference published when needed. On S3-compatible storage, that resume re-uploads the already-redacted data to its same content-addressed object key before the create-only manifest commit. This exact-match repair also covers a relational row already marked published whose manifest is missing. Different content or metadata is rejected without changing the pending reservation or writing to the artifact provider; correct the caller input and retry the original publication. Once published, the artifact remains immutable and any further publication with that identity is rejected as already published.

A lookup returns LookupResult rather than treating absence as an empty record. Reasons include run-not-found, artifact-reference-not-found, manifest-unavailable, artifact-publication-incomplete, data-unavailable, artifact-expired, and integrity failures. Normal run lookup includes only published references. An unavailable or pending artifact does not prevent the remaining product record from being read.

artifact-expired is a forward-compatible read behavior for references that already contain an expiry timestamp. The public publication operation does not set expiry, and the MVP does not expire or delete product records or artifacts automatically. Prefect retention is independent.

Secret boundary

Pass the credential values collected by collect_secret_values to mutation, audit, run, execution-link, result, artifact, and finish operations. Values are redacted from nested record data and raw artifact bytes before either provider is called. Do not put secrets into identifiers; credentials should continue to come from environment variables or a secret manager.

Reproducible sizing evidence

The sizing test repeats VAL-8's fixed-density 88k retained-artifact shape and the smaller 10k workload from the same source. Decimal bytes reproduce the published Parquet sizes: two payload stores, two hash indexes, and one plan. The separate 12.61 MB raw JSONL current-side wire measurement is not a retained artifact in this fixture.

Run:

uv sync --extra dev
uv run pytest -q -s tests/product_store/test_sizing.py

Observed on macOS with Python 3.13.3 on 2026-08-09:

Fixture	Records (baseline/current)	Artifact payload	Manifests	Relational store	Total files	Total logical bytes
VAL-8 88k	88,117 / 87,868	23,015,700 B	2,732 B	40,960 B	11	23,059,392 B
Representative 10k	10,051 / 10,023	2,722,700 B	2,877 B	40,960 B	11	2,766,537 B

The five artifact payloads account for ten object files (data plus manifest) and the SQLite run/reference/link schema accounts for one database file. Filesystem allocation, database page size, and production object-store metadata can change physical billing; the payload and manifest byte counts are the portable sizing inputs.