Secure AI Containers with SBOMs and Provenance

An AI inference service is not just a model endpoint. It is a release artifact assembled from application code, a base image, operating-system packages, language dependencies, model weights, tokenizer files, GPU libraries, and a build pipeline. If that bundle is promoted with only a mutable tag such as latest, a green test run does not tell you which bytes reached production or who produced them.

The practical fix is an evidence chain: build an immutable image, inventory what it contains, record how it was produced, sign the result, and make the cluster verify those facts before scheduling a workload. This guide shows a small production pattern for teams shipping Docker-based AI services to Kubernetes. It is deliberately conservative: an SBOM is an inventory, not a clean bill of health; a signature authenticates an identity, not the safety of the code; and provenance helps explain a build without proving that every source decision was good.

Original architecture diagram by Essa Mamdani. It is a conceptual design based on Docker Build attestations, SLSA v1.2, Sigstore Cosign, Kubernetes image references, and Kyverno image verification.

The diagram proves the intended control flow, not that any particular deployment is secure. The important boundary is between “an artifact was built” and “this cluster is willing to run it.” Keep those decisions separate so a compromised registry tag, stale scan, or unreviewed model file cannot silently become a production workload.

Courtesy: Docker. Source: https://docs.docker.com/build/metadata/attestations/. Accessed: 2026-07-24.

This screenshot proves what Docker’s current documentation means by an attestation: metadata created at build time and attached to the final image. Docker describes SBOM data as a list of software artifacts and provenance as information about how an image was built. That distinction drives the rest of the design.

Start with the artifact, not the scanner

The release unit should be an image digest, not a tag. A tag is a name that can be moved; a digest identifies a content-addressed image manifest. Kubernetes documents both forms, while Kyverno’s verifyDigest control exists specifically to require digest-based references for verified images. The operational consequence is simple: the digest scanned in CI must be the digest admitted to the cluster.

For an AI service, record the model separately even when the weights are copied into the image. A model file can be huge, externally downloaded, or mounted at runtime. Put its URI, version, checksum, license, and approval status in the release manifest. If the model is fetched on startup, the container signature alone cannot prove which weights the process loaded.

The minimum release record is:

This is a trust argument, not a compliance checkbox. The NIST Secure Software Development Framework gives the broader practice vocabulary; the artifact controls below turn that vocabulary into a release gate.

Make the Docker build inspectable

Use a multi-stage build, a pinned base image, a non-root runtime user, and no secret-dependent RUN step. The digest below is illustrative; replace it with a digest selected through your own base-image review. Do not copy this example and pretend the placeholder is a verified image.

1FROM python:3.12-slim@sha256:<reviewed-base-digest> AS runtime
2
3ENV PYTHONDONTWRITEBYTECODE=1 \
4    PYTHONUNBUFFERED=1
5
6WORKDIR /app
7COPY requirements.lock ./
8RUN pip install --no-cache-dir --require-hashes -r requirements.lock
9
10COPY src ./src
11COPY model-manifest.json ./model-manifest.json
12
13RUN addgroup --system app && adduser --system --ingroup app app \
14    && chown -R app:app /app
15USER app
16
17EXPOSE 8080
18CMD ["python", "-m", "src.server"]

--require-hashes is a package-manager control, not a substitute for an SBOM. The lockfile says which package artifacts the resolver should accept; the SBOM records what the image scanner observed. Both can be wrong, so inspect their generation and retain them with the build evidence.

Docker BuildKit can produce SBOM and provenance attestations during the build. A compact command for an OCI registry looks like this:

1docker buildx build \
2  --platform linux/amd64 \
3  --provenance=mode=max \
4  --sbom=true \
5  --tag registry.example.com/ai/inference:${GIT_SHA} \
6  --push .

Treat GIT_SHA as a human-friendly label only. After the push, resolve the resulting digest and pass that immutable reference to every later step. Do not run a second build for the scan, because a second build can produce different bytes even when the source commit is unchanged.

Inventory and scan the same digest

Generate an SBOM from the pushed digest, then scan that SBOM. Syft can emit CycloneDX or SPDX; Grype can match an SBOM against vulnerability data. The formats serve different consumers: CycloneDX is convenient for application-security relationships, while SPDX is an ISO/IEC 5962:2021 standard used widely for licensing and exchange. Keep the choice explicit in your pipeline rather than claiming that one format is universally superior.

1IMAGE="registry.example.com/ai/inference@${IMAGE_DIGEST}"
2
3syft "${IMAGE}" -o cyclonedx-json=sbom.cdx.json
4grype sbom:sbom.cdx.json -o json > vulnerabilities.json
5grype sbom:sbom.cdx.json --fail-on critical

The --fail-on critical threshold is a policy choice, not a universal security threshold. A critical match can be unreachable, fixed only in an unreleased package, or incorrectly detected. Conversely, a medium issue can be exploitable in an internet-facing parser. Require a documented exception with an owner, expiry, affected component, exploitability reasoning, and replacement plan. Use VEX or an equivalent justification process when your tooling supports it, and rescan retained SBOMs when new advisories arrive.

For GPU images, pay special attention to what the scanner can and cannot see. A host driver, device plugin, sidecar, init container, or mounted model volume may not appear in the application image’s SBOM. Write the deployment boundary down: image packages, node software, runtime libraries, model artifacts, and external services need separate inventory owners.

Sign the digest and attach provenance

Signing is useful only when the verifier checks the signer’s identity and the exact artifact. Sigstore documents keyless Cosign signing through an OIDC identity and transparency-log workflow. In CI, grant id-token: write only to the job that publishes the release. Keep source checkout, tests, signing, and deployment permissions in separate jobs where practical.

1cosign sign "${IMAGE}"
2
3cosign attest \
4  --type cyclonedx \
5  --predicate sbom.cdx.json \
6  "${IMAGE}"

The commands above create a signature and an SBOM attestation associated with the digest. Provenance is a different statement: it describes the builder, source revision, invocation, and materials. SLSA is a specification for describing and improving supply-chain security; it is not a magic rating emitted by the word “provenance.” Confirm which predicate format your builder emits and which fields your admission policy actually verifies.

Verification should be exact and boring:

1cosign verify \
2  --certificate-identity="https://github.com/acme/ai-inference/.github/workflows/release.yml@refs/heads/main" \
3  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
4  "${IMAGE}"
5
6cosign verify-attestation \
7  --type cyclonedx \
8  --certificate-identity="https://github.com/acme/ai-inference/.github/workflows/release.yml@refs/heads/main" \
9  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
10  "${IMAGE}"

Do not replace the exact workflow identity with a broad organization wildcard until you have modeled who can trigger that workflow. A trusted issuer plus an over-broad subject is still an over-broad release authority.

Enforce the decision at admission

CI can produce evidence, but it cannot stop a developer or another automation path from deploying an old tag. Put the final check at the cluster boundary. Kyverno’s verifyImages rule supports signatures, attestations, digest mutation, and digest verification. The following is a starting shape; test it against your controller kinds, registry, certificate identity, and failure behavior.

1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4  name: verify-ai-inference-images
5spec:
6  validationFailureAction: Enforce
7  rules:
8    - name: require-release-evidence
9      match:
10        any:
11          - resources:
12              kinds: [Pod]
13              namespaces: [ai-production]
14      verifyImages:
15        - imageReferences:
16            - "registry.example.com/ai/*"
17          verifyDigest: true
18          mutateDigest: true
19          attestors:
20            - entries:
21                - keyless:
22                    issuer: "https://token.actions.githubusercontent.com"
23                    subject: "https://github.com/acme/ai-inference/.github/workflows/release.yml@refs/heads/main"

A signature policy alone does not require an SBOM or a vulnerability result. Add attestation conditions only after you know the predicate path and schema produced by your build. Kyverno’s documentation notes that verified image metadata can be checked as signed attestations; use that capability to enforce the fields that matter to your release, such as a source repository, branch, builder, or model-manifest digest.

The policy should fail closed for production, but the rollout should not start there. Apply it in audit mode or a staging namespace, test a valid digest, a mutable tag, a wrong signer, a missing attestation, an expired certificate, and a registry outage, then move to enforcement with an emergency break-glass path that is logged and time-limited.

AI-specific threat model

Container supply-chain controls reduce one class of risk; they do not make an AI service trustworthy by themselves.

This is where the site’s MCP threat-modeling guide is a useful companion: tool permissions and egress policy are runtime boundaries, not properties granted by a signed image. The Go incident-debugging guide applies the same evidence discipline to post-deploy diagnosis, while the OpenTelemetry GenAI guide covers how to observe requests without exporting prompts and customer data by default.

A release checklist that survives the first incident

Before production, require a release packet containing the source commit, Dockerfile, lockfile, image digest, SBOM, scan database/version, provenance predicate, signature verification output, model manifest, exception records, and deployment manifest. Hash the packet and retain it according to the organization’s incident and audit policy; do not assume a hash makes sensitive data safe to share.

Then test the negative path. Delete the attestation and confirm admission rejects the image. Change one byte in the model manifest and confirm the checksum check fails. Push the same tag to a different digest and confirm the production manifest cannot use it. Revoke or remove the CI identity from the policy and confirm the next release is blocked. Disconnect the registry and document whether the cluster fails closed or uses a deliberate cache policy.

Keep the release gate separate from the application’s model-quality gate. A signed container can still hallucinate, leak a prompt, call an unsafe tool, or violate a product policy. Pair the artifact gate with evaluation and abuse tests, and keep the human owner for exceptions, rollback, and model promotion.

Methodology and limitations

This guide makes no benchmark, prevalence, cost, or latency claims. Confirmed statements about Docker attestations, Sigstore commands, SLSA terminology, Kubernetes image references, Kyverno verification fields, and NIST’s framework are taken from the official sources linked in the text and checked on 2026-07-24 UTC. The code is illustrative: action versions, certificate identities, registry behavior, predicate schemas, and cluster policy syntax must be tested against the versions you actually run. Recommendations and threat-model inferences are labeled as design guidance, not protocol guarantees.

Frequently asked questions

Is an SBOM enough to secure an AI container?

No. It is an inventory that supports vulnerability triage, license review, and later incident response. It does not prove that the image is free of unknown vulnerabilities, that mounted model weights are approved, or that runtime permissions are safe. Pair it with a digest, provenance, signature, and admission policy.

Should model weights live inside the image?

Either choice can work. Baking weights into the image gives you one immutable release object but may create large rebuilds. A separate artifact can improve promotion and caching, but then its checksum, signature, access policy, and loading behavior must be part of the release evidence. Never let “download latest model at startup” be an unreviewed production default.

Do I need both CycloneDX and SPDX?

Not always. Pick the format required by your consumers and preserve the original generator output. CycloneDX is often convenient for security-oriented relationships; SPDX is a standardized exchange format with strong licensing use cases. Producing both is reasonable when different scanners, customers, or auditors require them, but duplication is not a substitute for accuracy.

Are keyless signatures safer than long-lived signing keys?

They can reduce local key-management burden by binding a signature to an OIDC identity and recording it in a transparency log. They do not make a CI workflow trustworthy automatically. Restrict who can trigger the workflow, verify the exact subject and issuer, protect the release branch, and monitor identity changes.

How often should an image be rescanned?

Scan at build time and rescan retained SBOMs when vulnerability data changes. There is no universal interval that fits every exposure and patch SLA. An internet-facing inference gateway, a regulated workload, and an offline research cluster should have different response targets documented by their owners.

Can admission policy replace human review?

No. Admission can enforce mechanical facts such as signer identity, digest use, and required attestations. Humans still need to review source changes, model approval, exceptions, exploitability, rollback impact, and whether the evidence describes the artifact that is actually being deployed.

Author context and next step

I write about AI systems and full-stack engineering from the boundary between architecture and production operations. If you are shipping an inference service, start with one repository and one staging namespace: pin the base image, emit an SBOM and provenance, sign the digest, and prove that a deliberately invalid artifact is rejected. Once the negative tests are reliable, extend the same evidence contract to model artifacts, GPU runtime dependencies, and every promotion environment.