Sidecar injection

Sidecar injection is the deployment-side half of enforcement: a mutating admission webhook adds an enforcement sidecar to each agent pod, wires every container in that pod to it, mints the sidecar’s credentials, and keeps them out of the pod spec. Nothing about the agent’s image, its build, or its credentials changes.

Use this model when the enforcement point should be a property of the deployment rather than of the application — a container you do not build, or a fleet of workloads you want to govern uniformly by namespace. If you would rather intercept in-process, use the SDK integration instead. Both paths produce the same allow / block / escalate / audit decisions from the same policy model.

The PEP is cooperative — the sidecar is asked, it does not intercept

The injected sidecar is a decision point the agent calls. It listens on a pod-private Unix socket (/var/run/norviq/norviq-proxy.sock, exported to every container as NRVQ_SOCKET_PATH) and on an HTTP fallback at 127.0.0.1:8282; the agent sends a tool call, gets back forward or drop, and the agent executes the tool itself. A process in the pod that never asks reaches its tools directly.

That is a real bypass, and Norviq documents it rather than papering over it. Bound it at the network layer with agentEgressPolicy.enabled=true, which renders a default-deny egress NetworkPolicy (or CiliumNetworkPolicy) for the agent namespaces so a pod may only reach the Norviq API, DNS, and an operator-approved allowlist. It requires a NetworkPolicy-enforcing CNI — Calico or Cilium; kindnet ignores NetworkPolicy entirely.

How it works

flowchart TB
  subgraph admission[Pod admission -- once per pod]
    create[Pod created with<br/>norviq.io/agent-class] --> apiserver[kube-apiserver]
    apiserver --> mwc[MutatingWebhookConfiguration<br/>norviq-sidecar-injector]
    mwc --> whk[norviq-webhook /mutate]
    whk --> sec[(Secret<br/>norviq-sidecar-&lt;workload&gt;)]
    whk --> injected[Pod admitted with the<br/>sidecar + socket volume]
    sec -.->|secretKeyRef| injected
  end
  subgraph runtime[Runtime -- every tool call]
    tool[Agent asks for a<br/>forward/drop decision] -->|unix socket or<br/>127.0.0.1:8282| sidecar[norviq-sidecar]
    sidecar -->|POST /api/v1/evaluate<br/>mTLS + service JWT| api[norviq-api]
    api -->|allow / block /<br/>escalate / audit| sidecar
    sidecar -->|forward / drop| tool
  end
  injected -.-> tool

Enable injection

Injection is off by default (webhook.injection.enabled: false). Turn it on with a Helm upgrade:

helm upgrade norviq oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 -n norviq --reset-then-reuse-values --set webhook.injection.enabled=true

Enabling injection renders three things:

• The MutatingWebhookConfiguration named norviq-sidecar-injector — CREATE on pods only, timeoutSeconds: 5, reinvocationPolicy: Never.
• A hook Job (pre/post-install and pre/post-upgrade) that self-signs the webhook’s serving TLS cert, writes it to the norviq-webhook-tls Secret the webhook mounts, and patches the webhook’s caBundle — the self-signed cert acts as its own CA. No cert-manager is required.
• A namespaced Role + RoleBinding named norviq-sidecar-credentials in each namespace listed in webhook.injection.credentialSecret.namespaces (which defaults to policyQuotaNamespaces), granting the webhook get/create/update on Secrets in those namespaces only. See credential delivery below.

The webhook’s failurePolicy defaults to Fail — fail-closed. If the injector is unavailable, creation of a pod the webhook routes is rejected, so a routed agent pod can never start un-guarded. The control-plane namespace, kube-system, kube-public, kube-node-lease and the AKS system-namespace labels are excluded from the selector, so this posture never deadlocks the cluster. Set webhook.injection.failurePolicy: Ignore (fail-open) only for a dev or eval cluster where an un-injected pod is acceptable.

Fail-closed is the correct default — but check the HA primitives

Leave failurePolicy: Fail in production. The webhook runs 2 replicas by default with topology spread on (webhook.spread.enabled: true), so a single replica restarting does not block admission.

webhook.pdb.enabled ships false, because a PodDisruptionBudget blocks node drains on a single-node cluster. On a multi-node production cluster set webhook.pdb.enabled=true, otherwise one drain can evict both replicas and reject pod creation in every governed namespace until they reschedule.

Opt in: two labels, not one

Enabling the webhook injects nothing on its own. A pod is governed when both are true:

# 1. the namespace opts in — this is the webhook's namespaceSelector
kubectl label namespace chatbot-prod norviq-injection=enabled

# 2. the POD declares itself an agent — this is the webhook's objectSelector.
#    Put it in the Deployment's pod template, not on the Deployment.
spec:
  template:
    metadata:
      labels:
        norviq.io/agent-class: customer-support

The label value is also the policy scope: customer-support selects the customer-support agent-class policy, so it must match the agent_class in your NrvqPolicy. The injector validates the value as a Kubernetes label; an invalid value is logged as NRVQ-WHK-4015 and the pod is injected with an empty class, which means only namespace-scoped and cluster-baseline policy will match.

The admission rule is CREATE only. Running pods are never touched, and pods that already exist are not injected retroactively — restart or roll the workload after labeling.

The pod label is required, and its absence is silent

webhook.injection.gateOnlyAgentPods defaults to true, so the namespace label alone is not enough. A pod without norviq.io/agent-class is not routed to the webhook at all — nothing errors, it simply runs ungoverned.

This default bounds the blast radius of a Norviq outage. Gating on the namespace alone routes every pod created there through the injector — databases, ingress controllers, Jobs — so with failurePolicy: Fail an injector outage blocks pod creation for workloads Norviq does not govern.

Upgrading from a release that injected namespace-wide? Add norviq.io/agent-class to every agent Deployment before upgrading. Existing pods keep their sidecar until they restart, so the gap appears at the next rollout rather than immediately. Nothing in the product computes injected-vs-expected — /system-health reports infrastructure verdicts read back from the audit log and does not raise an “ungoverned agent pods” warning — so verify by hand:

kubectl get pods -n chatbot-prod -l norviq.io/agent-class -o name   # routed, will be injected
kubectl get pods -n chatbot-prod -L norviq.io/agent-class           # a blank column = NOT routed

Set gateOnlyAgentPods=false to restore namespace-wide behaviour.

Name the workload tier

The injector resolves the pod’s workload from its owner reference — a pod owned by checkout-7d9f8b5c4 (a ReplicaSet) resolves to checkout — and binds it into the sidecar’s token as a workload claim. That claim is what makes workload-tier policy (norviq policy apply --target-type workload) match real traffic; without it the tier is inert.

A bare pod has no owner and gets no workload, and the tier then does not apply. For a bare pod, a CRD-managed workload or an Argo Rollout, state it explicitly — an explicit label always wins:

metadata:
  labels:
    norviq.io/workload: checkout

Verify it worked

kubectl -n chatbot-prod get pod -l app=your-agent
# 2/2 containers on a single-container app = injected.  1/1 = NOT injected —
# check the namespace label first, then the pod's norviq.io/agent-class label.

The injector also stamps norviq.io/injected: "true" on every patched pod. Treat it as an operator-visible marker only: it is not a trust input, because a tenant can self-stamp it on an unadmitted CREATE. The webhook recognizes an injected pod by its structural wiring instead.

Opt a pod out

Within a routed namespace, an individual pod opts out with either:

• the label norviq-injection=disabled, or
• the annotation norviq.io/skip-injection: "true".

A pod that already carries a correctly wired Norviq sidecar is skipped as well (NRVQ-WHK-4008), so re-applying a manifest is safe.

The opt-out is an in-namespace bypass — disable it or govern it

By default the injector honors both keys, so anyone who can create or edit a pod spec in a routed namespace can exempt their own workload from enforcement. Set webhook.injection.allowPodOptOut: false and the injector ignores them and injects anyway, logging NRVQ-WHK-4009 for each attempt.

That closes the opt-out, not every exit: while gateOnlyAgentPods is true, simply omitting norviq.io/agent-class is an earlier exit that allowPodOptOut=false cannot reach. Pair it with RBAC (or a ValidatingAdmissionPolicy / Gatekeeper constraint) on who may write the agent-class label as well as the opt-out label and annotation.

A pod that carries Norviq enforcement artifacts but is not fully and correctly injected — a decoy container taking the norviq-sidecar name, a pre-occupied norviq-socket volume, a preset NRVQ_SOCKET_PATH — is denied, not skipped, with NRVQ-WHK-4034. The injector cannot safely wire over plumbing it does not own, and skipping would run the pod unpoliced.

What gets injected

For each routed pod the patch adds:

• The norviq-sidecar container, from the same image as the engine (images.engine), pinned by digest on a published chart. The injector refuses any image outside its allowlist and fails admission with the reason in the kubectl error (NRVQ-WHK-4033).
• Its security context: runAsNonRoot, uid 65534, readOnlyRootFilesystem, allowPrivilegeEscalation: false, all capabilities dropped, seccompProfile: RuntimeDefault.
• A startupProbe on /healthz (up to 90s), plus liveness on /healthz and readiness on /readyz, both on port 8282. Readiness gates the pod on the sidecar actually serving enforcement, so a mis-wired sidecar surfaces as NotReady rather than silently forwarding.
• An emptyDir volume norviq-socket (10Mi) mounted at /var/run/norviq into the sidecar and into every app and init container, plus NRVQ_SOCKET_PATH in each. Init containers are wired on purpose: an agent workload placed in an init container would otherwise run before the sidecar with no socket at all. The socket does not exist until the sidecar starts, so an init-phase call fails closed.
• A tmpfs norviq-tmp (16Mi, medium: Memory) mounted into the sidecar only. The mTLS client key is materialized there at 0600, unlinked immediately, and never touches real disk or the app container.

Proxy mode and embedded mode

webhook.injection.sidecarMode defaults to proxy.

proxy holds no policy engine. It POSTs each tool call to the central API’s /api/v1/evaluate and enforces what comes back. Redis, OPA and Postgres all stay centralized, nothing is evaluated per-pod, and the central /evaluate writes the audit record. This is the right choice for a normal in-cluster deployment.

embedded runs a whole engine inside the pod — Redis client, policy loader, audit emitter, and an OPA subprocess fork per call. It exists for air-gapped and edge deployments; the chart then wires NRVQ_REDIS_URL, NRVQ_PG_URL and NRVQ_DB_SSL_MODE through to every injected sidecar, which means each agent pod needs direct datastore reachability. If you also use agentEgressPolicy, set agentEgressPolicy.embeddedDatastores=true or the sidecar cannot reach them.

The chart selects the sidecar’s resource budget from the mode, and these numbers are measured, not guessed:

	proxy	embedded
requests	50m / 64Mi	200m / 256Mi
limits	200m / 128Mi	2000m / 384Mi

The embedded CPU limit of 2000m is load-bearing. Measured on AKS through a real injected sidecar, back-to-back in one session:

Configuration	p50	p95	CFS throttling
proxy	59.0 ms	92.7 ms	—
embedded @ 500m	72.0 ms	93.1 ms	58.9% of periods
embedded @ 2000m	30.7 ms	58.0 ms	0%

At 500m, embedded was slower than the proxy default it exists to beat. Only the limit is generous — requests stays at 200m, so scheduling density is unchanged and this cannot make a pod unschedulable.

What the sidecar does at runtime

In proxy mode the sidecar authenticates to the API with a namespace-scoped role=service JWT the injector mints at admission. The token is bound to the identity the pod was admitted with: agent_class from the pod label, workload from the owner chain, and (in the default mock SPIFFE mode) a deterministic spiffe_id. Without those claims a namespace-scoped token could assert a sibling class and run its looser policy, or another SPIFFE id and shed an operator’s trust freeze.

When internal TLS is on (config.internalTls.enabled: true, the default), the injector also mints a per-namespace client certificate signed by the internal CA — CN=norviq-sidecar, OU=<namespace>, ExtKeyUsage=ClientAuth, 30-day validity — and the sidecar does mTLS to https://norviq-api.<control-plane-ns>.svc:8443, presenting the client cert and the JWT as defense in depth. The chart injects the namespace-qualified FQDN, not a bare norviq-api, so it resolves from the tenant namespaces sidecars actually run in. This is turnkey: you run no openssl, issue no CSR, and install no cert-manager. See the Security model for the internal CA.

Upstream calls use a 1s connect / 2s total timeout and retry twice with exponential backoff from 100ms. Only transport errors and 5xx are retried.

Outage posture: fallbackMode is fail-open

webhook.injection.fallbackMode ships allow, and it is stamped into every injected sidecar as NRVQ_SDK_FALLBACK_MODE — so it wins over any Python-side default on every real install.

Setting	Behaviour on a sustained engine outage
allow (default)	Fail-open. The tool call is forwarded and audited with rule_id=thin_proxy_fail_open, never as a policy allow.
block	Fail-closed. No ungoverned tool call, ever — at the cost of taking every agent in the cluster down for as long as the engine is down.

Two constraints on this, and both matter:

• It applies only to a genuine outage — 5xx, timeout, connection error. A 4xx is the engine answering with a refusal (an expired token, a malformed request) and always blocks, regardless of this setting, with rule_id=thin_proxy_fail_closed. allow cannot turn a revoked credential into a silent bypass.
• Both data-plane paths retry with backoff first, so this only decides what happens after a sustained outage, not during a rolling restart or a brief blip.

This defaulted to block until it was measured against the product’s actual posture. Norviq sits in the request path of production agents, and a security control whose failure mode is a customer outage gets removed from that path — which protects nobody. Set it to block deliberately, for a workload where an ungoverned call is worse than no call.

Whichever you choose, alert on thin_proxy_fail_open in the audit log. It is the countable record of every call that went unjudged.

Credential delivery goes through a Secret

webhook.injection.credentialSecret.enabled ships true. NRVQ_API_TOKEN, NRVQ_CLIENT_CERT_PEM and NRVQ_CLIENT_KEY_PEM are written to a Secret and referenced from the pod spec by valueFrom.secretKeyRef instead of appearing as literal value: entries.

Why it matters: Kubernetes deliberately excludes Secrets from the built-in view ClusterRole — view grants get pods and does not grant get secrets, and that exclusion is the whole reason view is considered safe to hand an auditor, an SRE, a dashboard or a CI service account. A credential in the pod spec sits on the other side of that line: kubectl get pod -o yaml returns a working 30-day workload JWT. Pod specs also live in etcd (unencrypted unless the cluster enables encryption at rest), in kubectl describe, and in any GitOps diff.

NRVQ_API_CA_PEM is deliberately left as a literal value — a CA certificate is public by construction.

Practical details:

• The Secret is named norviq-sidecar-<workload> (hash-suffixed when the name must be sanitized or truncated) and is keyed per (namespace, workload, agent class), not per pod — a pod’s name is not knowable at admission for anything a ReplicaSet creates, and every pod of one Deployment is entitled to the same claims.
• It is rewritten on every admission, not created-if-absent, so a replacement pod gets freshly minted material. Overwriting is safe for pods already running: secretKeyRef env is resolved by the kubelet at container start and the running container holds its own copy. The norviq.io/minted-at annotation is your rotation evidence — a stale timestamp means injection stopped happening.
• A dry-run admission gets the same patch shape but writes nothing.

No namespace list means no Role, and the credential falls back into the pod spec

The Role is namespaced on purpose — the webhook already holds admission-time mutation power over every pod in routed namespaces, and cluster-wide Secret write would make a compromise of it far more valuable. It is rendered only into the namespaces in webhook.injection.credentialSecret.namespaces, which defaults to policyQuotaNamespaces.

If a routed namespace appears in neither list, nothing grants the write. The injector then falls back to literal pod env and logs NRVQ-WHK-4049 at ERROR for every injected pod — naming the consequence and the fix — rather than refusing to schedule, because this webhook runs failurePolicy: Fail and a cluster-wide “no pods may be created” outage is worse than the exposure.

To refuse instead, set webhook.injection.credentialSecret.required: true. Admission then fails with NRVQ-WHK-4052 and a message naming the namespace whose RBAC to check.

# Confirm credentials are NOT in the pod spec: this should show secretKeyRef, not a JWT.
kubectl -n chatbot-prod get pod -l norviq.io/agent-class -o yaml | grep -A5 NRVQ_API_TOKEN

# Catch the fallback across the whole fleet.
kubectl -n norviq logs deploy/norviq-webhook -c webhook | grep NRVQ-WHK-4049

The 30-day credential cliff

Both credentials the webhook injects last 30 days, and nothing renews either in place:

• the service JWT — NRVQ_SIDECAR_TOKEN_TTL_HOURS, 720 hours, not exposed as a chart value;
• the mTLS client certificate — NotAfter = now + 30 days.

There is no rotation loop in the webhook and nothing in the sidecar re-reads a credential. A pod that keeps running for thirty days ends up holding two expired ones. At that point the API answers 401, the sidecar sees a 4xx, and a 4xx overrides fallbackMode to fail closed — so the workload’s tool calls stop.

That behaviour is deliberate and must not be “fixed”: a refused credential silently becoming a governance bypass would be far worse than an outage. The defect was that it used to arrive with no warning.

The forewarning band

The API observes credential expiry on the authentication path it already runs and surfaces it on GET /api/v1/system-health — the console’s status banner — as a warning band, severity: warning, seven days ahead:

Injected sidecar credentials expire soon — N injected workload(s) hold a credential expiring within 7 days. Nothing renews these in place… Roll the affected Deployments before the date above.

Three things about it are worth knowing before you rely on it:

• It is best-effort by construction. It writes nothing until a role=service token is inside the 7-day window, swallows every error, and degrades to “no warning” if Redis is unreadable. An authentication path that failed because a reporting write failed would be a much worse bug.
• It is driven by traffic. A workload whose sidecar never authenticates in its final week is never observed and never warned about.
• Injected sidecars and service keys get separate bands, because their remediations have nothing in common. The workload claim — which only the injector mints — is what distinguishes them. A band titled Service keys expire soon means an operator-minted key (an MCP proxy, the fleet relay, a CI caller): rolling a Deployment does not rotate those.

/system-health also diagnoses the cliff after the fact, as engine_rejected_request — “typically an expired or wrong sidecar token”, remediation “restart affected pods”. Diagnosing an outage in progress is not the same as preventing one.

Rotation is pod replacement

There is no rotate command, and that is the design, not a gap. Admission is the minting event, so a replacement pod is issued a new token (later iat, unchanged identity) and a new client certificate, and the Secret is rewritten in the same admission.

# Rotate every injected credential for one workload.
kubectl -n chatbot-prod rollout restart deployment/your-agent

# Confirm the material was re-minted.
kubectl -n chatbot-prod get secret norviq-sidecar-your-agent -o jsonpath='{.metadata.annotations.norviq\.io/minted-at}'

Any normal rollout does this for free, which is why a team on a weekly or fortnightly release cadence never sees the band. It is the long-lived, never-redeployed agent that walks off the cliff — so if you have workloads that are not routinely rolled, put a monthly restart on a schedule rather than waiting for the warning.

Clean uninstall

NrvqPolicy custom resources carry the finalizer norviq.io/policy-protection, and only the CRD controller — which runs inside the webhook — clears it, after syncing the deletion to the API. Helm removes that controller during the same helm uninstall, so without help the CRs strand in Terminating: the uninstall blocks until it times out, the release is left half-removed, and every CR has to be hand-patched before a reinstall.

The chart handles this. crdFinalizerCleanup.enabled ships true and renders a pre-delete hook Job that releases the finalizers first, while the API is still up. It is best-effort by design and never fails the uninstall — a stuck finalizer is recoverable by hand, an un-uninstallable release is worse.

Set crdFinalizerCleanup.enabled: false only if your cluster forbids hook Jobs; then delete the CRs yourself while the controller is still running:

# 1. Delete NrvqPolicy CRs first, while the webhook/controller is still up to clear finalizers.
kubectl delete nrvqpolicies --all --all-namespaces

# 2. Confirm they're gone (none should linger in Terminating).
kubectl get nrvqpolicies --all-namespaces

# 3. Now uninstall the chart.
helm uninstall norviq -n norviq

Recovering a stuck Terminating resource

If a NrvqPolicy (or a namespace containing one) is already wedged in Terminating, clear the finalizer manually so the object can finalize:

kubectl patch nrvqpolicy <name> -n <ns> \
  --type merge -p '{"metadata":{"finalizers":null}}'

Patching finalizers to null removes the block; use it only for recovery, since it skips the controller’s normal delete-sync to the API. If the webhook/controller is still running you do not need this patch — a delete that cannot sync to the API self-heals when the controller force-removes the finalizer after 15 minutes (NRVQ-WHK-4041). Reach for the manual patch only once the controller has already been uninstalled.

Next steps

• Get started — install the control plane and enable injection end-to-end.
• Writing policies — author the NrvqPolicy and NrvqClass objects the sidecar enforces.
• Configuration — the full webhook.injection and config.internalTls value reference.
• Security model — the internal CA, mTLS, and the cooperative-PEP boundary.
• SDK integration — the in-process alternative to the sidecar.