Deployment

Norviq ships as one Helm chart. The base install is secure by default — zero-touch internal mTLS on the control plane, auto-generated strong secrets, hardened containers — while everything prod-specific (HA, autoscaling, multi-cluster fleet, SPIFFE workload identity, network-layer egress lockdown) is values-gated and off by default.

This page covers deploying on any conformant Kubernetes cluster — AKS, EKS, GKE, or a vanilla 1.30+ distribution (kubeVersion: ">=1.30.0-0"; helm install refuses on older). The full key reference is in Configuration; the trust model behind the mTLS is in the Security model.

1. Install the control plane

Install from the published OCI chart. The CRDs ship inside it, so Helm installs NrvqPolicy, NrvqClass and NrvqConfig for you on first install — there is no separate kubectl apply step, and no crds/ directory at the repo root to apply.

A single-node cluster is enough to stand the control plane up and evaluate everything except multi-node HA (§6), a real fleet (§11) and real SPIFFE identity (§12).

kubectl create namespace norviq

# Your TENANT namespace — the one that will run agent workloads. It must exist before you install,
# because the chart renders a baseline policy into it.
kubectl create namespace chatbot-prod

helm install norviq oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 \
  -n norviq \
  --set-json 'policyQuotaNamespaces=["chatbot-prod"]'

The chart does not template a Namespace — the install namespace comes from helm -n/--namespace (the standard Helm contract). Use --create-namespace instead of kubectl create namespace if you prefer; there is no namespace value.

policyQuotaNamespaces is required — an empty list fails the install

baselineClusterPolicy.enabled defaults to true and renders one NrvqPolicy per entry in policyQuotaNamespaces (named baseline-cluster-guard-<ns>, clusterPriority: 900, preset: strict). With the list empty it would render zero baselines while the install notes claim a baseline is present, so templates/baseline-cluster-policy.yaml calls fail and aborts the render:

baselineClusterPolicy.enabled=true but policyQuotaNamespaces is EMPTY.
No baseline guard would be rendered, leaving every agent class without the fail-closed cluster baseline.

List your tenant namespaces, or set baselineClusterPolicy.enabled=false to run deliberately without a cluster baseline.

The baseline arrives observing, not blocking

baselineClusterPolicy.enforcementMode defaults to audit in 0.2.5. This is the most consequential posture change in the release, and it is deliberate.

The rendered baseline still evaluates every control on every tool call and records a non-compliance event — but the call proceeds. Previously this shipped as block, which put the full strict control set in front of every tool call in every tenant namespace on day one, before the operator had written a policy or seen a decision. That is not a theoretical problem: an unreviewed strict preset in front of live traffic on day one means every false positive the preset carries lands as a dropped call before anyone has looked at a decision. (One such false positive — deny_shell_execution misreading a base64-decoded order ID as a shell pipe — was found this way and has since been fixed in the shipped strict.rego; it is not a reason to distrust today’s preset, but it is the shape of risk that shipping block on day one carries.)

Practically: a fresh 0.2.5 install observes and records; it does not drop. Use the Policy Compliance view to see exactly what would have been blocked and by which control, then promote controls once the blast radius is known. To restore the old cluster-wide posture:

--set baselineClusterPolicy.enforcementMode=block

Note this is separate from config.enforcementMode (default block), which governs your own policies, and from config.noPolicyDecision (default allow — a namespace nobody has written a policy for is not governed, so the call proceeds). See Configuration.

Verify the chart before you install it

The published chart is cosign-signed and the release workflow stamps every Norviq image reference with an immutable sha256 digest, so --version 0.2.5 deploys exactly the bytes that release published rather than whatever -latest resolves to today.

cosign verify ghcr.io/norviq-dev/charts/norviq:0.2.5 \
  --certificate-identity-regexp '^https://github.com/norviq-dev/norviq/.github/workflows/release.yml@.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

The in-tree values.yaml shows floating engine-latest / api-latest / ui-latest / webhook-latest tags with empty digest fields. That is the source; the digests are stamped at package time. Digest pinning is a property of the published chart, not of the working tree.

Wait for the rollout

The webhook is part of the control plane — it hosts the CRD controller — so wait for it too, even with sidecar injection off.

kubectl -n norviq rollout status deploy/norviq-api
kubectl -n norviq rollout status deploy/norviq-webhook
kubectl -n norviq rollout status deploy/norviq-ui

The chart deploys the API, engine, console UI, mutating webhook (+ CRD controller), and bundled PostgreSQL (postgres:16-alpine), Redis (redis:7-alpine) and an OPA (openpolicyagent/opa:1.19.1-static) sidecar in every API and engine pod.

Then prove it actually serves traffic. The chart ships a helm test hook — a short-lived curlimages/curl:8.10.1 pod that curls /healthz and /readyz through the in-cluster Service:

helm test norviq -n norviq

First login

On a stock install the chart auto-generates both the JWT signing secret and the admin password — there is nothing weak to pin, so nothing to accidentally go live with.

kubectl get secret norviq-secrets -n norviq \
  -o jsonpath='{.data.NRVQ_AUTH_ADMIN_PASSWORD}' | base64 -d

Sign in as admin; you are forced to change the password on first login.

That Secret is the FIRST password only

The Secret is written at install and never updated again. Once you complete the forced change it is permanently stale — the live password lives hashed (bcrypt) in Postgres. Coming back weeks later and pasting the Secret value gets you invalid username or password, which reads like a broken login rather than a stale lookup. Store the password you set. There is no command that reads it back. If it is lost, reset it with norviq admin reset-password (see the CLI) or reinstall.

config.requireStrongSecret: true (the default) is the fail-closed backstop, not the mechanism: it fires only if an operator explicitly pins the placeholder JWT secret (change-me-in-production) or the sentinel admin password (norviq), in which case the API refuses to start rather than run insecure — logging NRVQ-API-7099 for the weak-JWT-secret case and NRVQ-AUTH-14014 for the default-admin-password case. Auto-generation is what keeps a stock install secure; the backstop catches a deliberate downgrade.

Installing from a clone

Only for contributors or a modified chart. From a clone the CRDs are not installed for you:

git clone https://github.com/norviq-dev/norviq.git
cd norviq
kubectl apply -f helm/norviq/crds/
helm install norviq ./helm/norviq -n norviq --create-namespace \
  --set-json 'policyQuotaNamespaces=["chatbot-prod"]'

This tracks whatever is on your checkout — which is what you want when changing the chart, and what you do not want when evaluating a release.

Overlays for local/eval installs

Two overlays ship inside the chart and trade posture for footprint on a throwaway cluster. Neither is for production:

• values-dev.yaml — fixed, non-secret dev credentials (api.secretKey, DB/Redis passwords), config.logLevel: DEBUG, and config.enforcementMode: audit, so a fresh install logs decisions instead of enforcing them.
• values-light.yaml — the smallest viable single-node footprint: api.replicas: 1, api.workers: 1, webhook.replicas: 1, PDBs off, tighter limits. Enforcement is byte-identical to the default chart; only replica counts, PDBs and resource envelopes change.

Get them on disk with helm pull oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 --untar.

Sidecar injection is off by default; enabling it and opting namespaces in is covered in Sidecar injection.

2. Access the console — ingress

For any shared or production cluster, front the console with a TLS-terminated ingress. A single host serves both the console SPA and the API: the console’s nginx already reverse-proxies /api/* to norviq-api, so one ingress backend routes both — no path rewrite (a rewrite-target here breaks SPA asset paths and /api routing).

ingress.values.yaml

ingress:
  enabled: true
  className: nginx
  host: norviq.example.com
  tls: true
  tlsSecretName: norviq-ingress-tls
  annotations:
    # Issue/rotate the edge cert automatically (or pre-create the secret named above).
    cert-manager.io/cluster-issuer: letsencrypt-prod

helm upgrade --install norviq oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 \
  -n norviq --reset-then-reuse-values -f ingress.values.yaml
kubectl -n norviq get ingress norviq-ingress

The chart installs no ingress controller and deliberately does not generate a certificate (same convention as Grafana and Istio — a self-signed cert is useless for real HTTPS and hides a false “TLS works”). Either pre-create the tlsSecretName Secret or let a cert-manager issuer populate it via the annotation. With tls: true and no such cert the ingress silently falls back to the controller’s default self-signed certificate. tls: false serves plaintext and is dev-only.

Use --reset-then-reuse-values, not --reuse-values

Helm’s --set/-f on an upgrade replace the release’s values — anything you don’t re-supply reverts to chart defaults.

But not --reuse-values. That flag replays only the values you supplied last time and does not merge in values a newer chart added, so upgrading across a release that introduces one aborts with nil pointer evaluating interface {}.<key>. The chart carries explicit defences against this in the places it bit hardest — crdFinalizerCleanup guards its own lookup with default dict plus hasKey/ternary so a missing key defaults rather than crashes, and other values (like the evaluator’s in-process cache keys) use Sprig’s dig the same way — but it cannot cover every key. Use --reset-then-reuse-values (Helm ≥ 3.14), or re-pass your own -f values.yaml on every upgrade.

Port-forward is a dev/eval fallback

The console’s nginx also proxies /api/*, so one port-forward covers the whole app:

kubectl -n norviq port-forward svc/norviq-ui 8080:80   # http://localhost:8080

Note the Service listens on :80 and targets container port 8080. Use the ingress above for anything a second person needs to reach.

3. Internal mTLS (secure by default)

Ingress TLS (§2) protects traffic between users and the edge. Internal TLS is a separate layer that protects control-plane traffic between pods — controller→API and sidecar→API — and it is on by default with no operator action:

config:
  internalTls:
    enabled: true                    # DEFAULT — zero-touch
    proxyImage: "nginx:1.27-alpine"

• A Helm pre-install/pre-upgrade hook auto-mints an internal CA (Secret norviq-internal-ca, ca.crt/ca.key) and a CA-signed API serving cert (Secret norviq-api-tls). It is idempotent — existing secrets are reused so the CA identity stays stable across upgrades — and self-deletes on success. No openssl on your side, no CSR, no cert-manager. The bootstrap image carries openssl and curl but deliberately not kubectl, so the hooks work air-gapped and pass a restricted SCC.
• The API pod runs an nginx TLS terminator sidecar on :8443. The app and its probes stay on loopback HTTP unchanged; only cross-pod traffic is TLS. Plaintext to :8443 is rejected.
• The CRD controller (in the webhook) is wired to https://norviq-api.<ns>.svc:8443 and verifies the API’s serving cert against the internal CA — fail-closed if the CA is missing or invalid (an empty trust pool means every handshake fails rather than falling back to plaintext).
• The injector mints a per-namespace client cert signed by the same CA, so injected sidecars do mTLS to the API. The sidecar needs a writable path to materialise that key, which is why every injected pod gets a medium: Memory emptyDir at /tmp (16Mi, sidecar only) — it keeps key material off disk, and without it an injected pod with readOnlyRootFilesystem crash-loops.

Client mTLS is verified-if-presented, not required

The terminator runs ssl_verify_client optional. A client cert, when presented, is verified against the internal CA and its identity forwarded to the app as X-Nrvq-Client-Verify / X-Nrvq-Client-Subject headers — but a client that presents no cert still reaches the API over TLS and authenticates with its bearer token. This is what lets the console and CLI work at all.

Two consequences worth knowing:

• The transport guarantee is TLS everywhere, mTLS where a cert exists. Authorization still rests on the JWT/API key, not on the client certificate.
• If the injector cannot read the CA or mint the leaf, it logs NRVQ-WHK-4047 / NRVQ-WHK-4048 and falls back to plaintext + JWT for that sidecar rather than refusing to inject. Grep the webhook logs for those codes after enabling injection if you are relying on mTLS being in place.

Set config.internalTls.enabled: false only for a throwaway plaintext dev cluster.

Note

Auto-mTLS is the zero-infra default for transport security. SPIFFE/SPIRE (§12) remains the opt-in high-assurance mode for cryptographic workload identity — the two are complementary, and SPIFFE is not required for the default secure posture.

4. External Postgres and Redis

The chart runs bundled single-replica Postgres and Redis StatefulSets by default. For production the usual shape is a managed database with its own backups, HA and patching — set enabled: false and point the chart at it.

postgresql:
  enabled: false
  existingSecret: my-pg          # a Secret YOU manage, holding the full URL
  existingSecretKey: url         # default
redis:
  enabled: false
  existingSecret: my-redis
  existingSecretKey: url

kubectl -n norviq create secret generic my-pg \
  --from-literal=url='postgresql://norviq:PW@mydb.postgres.database.azure.com:5432/norviq'
kubectl -n norviq create secret generic my-redis \
  --from-literal=url='rediss://:PW@mycache.redis.cache.windows.net:6380/0'

existingSecret is the production credential path: the API, engine and webhook read NRVQ_PG_URL / NRVQ_REDIS_URL straight from your Secret via secretKeyRef, so no credential ever passes through a values file, through --set (which lands in helm history and your shell history), or through the chart’s own Secret. The URL carries user, password, host, port and database, so postgresql.username / password / port / database are then unused.

The simpler alternative is --set postgresql.host=<host> and let the chart assemble the URL from username/password/port/database. That is fine for staging; it puts the password in helm history.

enabled: false with no address fails the render, by design

postgresql.enabled=false (external database) but neither postgresql.host nor
postgresql.existingSecret is set — Norviq has no address to connect to.

Without a host the chart would emit the bundled Service name and every pod would dial a Service that does not exist. It fails loudly instead. The same guard exists for Redis.

TLS to the database is derived, not guessed. config.dbSslMode defaults to "" and resolves as:

Datastore	Derived sslmode	Why
Bundled Postgres StatefulSet	disable	The bundled Postgres has no TLS listener — nothing in the chart ever gives it a cert, so require could only crash on startup
postgresql.ha.enabled (CloudNativePG)	require	CNPG issues server certs and serves TLS
External host (postgresql.host or existingSecret)	require	Assume a managed / TLS-terminating Postgres

So a managed database gets require with no action from you, and a local install starts without an override. An explicit config.dbSslMode always wins verbatim — set verify-full to go stricter.

Note

If you are carrying --set config.dbSslMode=disable forward from a 0.1.x runbook, drop it. It was a workaround for a hard require default that made a pure-defaults install crash-loop on PostgreSQL server ... rejected SSL upgrade. Leaving it pinned now forces plaintext against a managed database that would otherwise have got require.

Credentials for the bundled stores are generated, not shipped. Leave postgresql.password / redis.password empty and the chart mints a strong random value on first install, stores it in norviq-secrets, and reuses it on every upgrade — it never rotates it out from under a live database (Postgres only honours POSTGRES_PASSWORD at initdb, so rotating under an existing PVC would strand the data). Read the generated values with:

kubectl -n norviq get secret norviq-secrets -o jsonpath='{.data.NRVQ_PG_PASSWORD}' | base64 -d; echo
kubectl -n norviq get secret norviq-secrets -o jsonpath='{.data.NRVQ_REDIS_PASSWORD}' | base64 -d; echo

5. Private registry and air-gap

Norviq’s own images and the third-party images are relocated by two different keys, and getting that wrong is the most common air-gap failure.

Key	Default	Covers
images.registry	ghcr.io/norviq-dev/	The four Norviq components — all share the norviq-engine repository, distinguished by tag prefix (api-, engine-, ui-, webhook-), plus the first-party bootstrap image
global.imageRegistry	""	The third-party images: openpolicyagent/opa:1.19.1-static, redis:7-alpine, postgres:16-alpine, nginx:1.27-alpine (the internal-TLS proxy), curlimages/curl:8.10.1 (the helm test pod), busybox:1.36 (dependency-wait initContainers)

helm upgrade --install norviq oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 -n norviq \
  --set images.registry="myregistry.example.com/norviq/" \
  --set global.imageRegistry="myregistry.example.com" \
  --set imagePullSecrets[0].name=my-registry-pull-secret

Mirror the upstream images into your registry preserving their path — myregistry.example.com/openpolicyagent/opa:1.19.1-static, myregistry.example.com/redis:7-alpine, and so on. global.imageRegistry is prepended as a host, not substituted for the whole reference.

images.registry is a prefix with a trailing slash, and all four components share one repository. Setting it to a bare ghcr.io/ resolves to a non-existent ghcr.io/norviq-engine. If you push to Docker Hub, set images.registry: "" and override each repository to <user>/norviq-engine.

If you run the CloudNativePG HA path (§6), mirror postgresql.ha.image (ghcr.io/cloudnative-pg/postgresql:16) too. It is not images.postgresql and is not interchangeable with postgres:16-alpine: CNPG hardcodes postgres UID/GID 26 and runs initdb as that UID, while postgres:16-alpine uses UID 70 — swap them and the cluster never bootstraps (could not look up effective user ID 26).

Do not point crdFinalizerCleanup.image at bitnami/kubectl. It defaults to empty, meaning “use the first-party bootstrap image”, and that default exists because the old bitnami/kubectl:1.31 default was deleted from Docker Hub — the pull 404s, the pre-delete hook never runs, and helm uninstall dies on a deadline with every CR stranded in Terminating (see §13).

For scale, prefer a registry without an anonymous pull-rate limit — Google Artifact Registry (images.registry: us-docker.pkg.dev/<PROJECT_ID>/<REPO>/) or your own ACR/GHCR. Leave imagePullSecrets: [] for a public registry.

6. Production single-cluster — HA

values-prod.yaml (inside the chart) turns on the multi-node HA posture that is gated off in the single-node defaults.

Prerequisites: ≥3 nodes (so anti-affinity/topology spread actually spreads replicas), metrics-server (the HPAs read CPU and memory), the CloudNativePG operator (Postgres HA), and a Redis HA operator — the chart renders a Spotahome RedisFailover CR plus the master Service itself (redis.ha.serviceName, selecting the operator’s redisfailovers-role: master pod label), because the operator provisions only a sentinel Service and Norviq’s client speaks a plain redis:// URL. A managed Redis (§4) is a perfectly good alternative and needs no operator.

The base chart is already HA-shaped — 2 API replicas with a PDB, surge rollouts (maxSurge: 1/maxUnavailable: 0), strong secrets, derived require DB TLS. What the overlay adds is multi-node posture:

Area	Base chart default	values-prod.yaml
api.replicas / PDB	2 / on, minAvailable: 1	2 / on, minAvailable: 2
api.autoscaling (HPA)	off	on — CPU 70% + memory 75%, 2→10 replicas
engine.replicas / PDB	1 / off	2 / on, minAvailable: 1 + HPA 2→8
webhook.replicas / injection	2 / injection off	2 + PDB + HPA 2→4 / injection on
*.spread (anti-affinity + topologySpread)	off, except webhook.spread: true	on for api, engine, webhook
postgresql.ha	off (single StatefulSet)	on — CloudNativePG Cluster (3 instances), 20Gi
redis.ha	off (single StatefulSet)	on — RedisFailover (Sentinel, 3)
images.*.pullPolicy	Always	IfNotPresent
config.dbSslMode	"" (derived)	require (pinned)
gracefulShutdown.preStopSleepSeconds	15	5
postgresql.password / redis.password	generated	blank — you must supply them

webhook.spread is the one component that spreads by default. The injector is the only workload whose unavailability rejects pod creation in every injection-enabled namespace (failurePolicy: Fail), so both replicas landing on one node turns a routine node drain into a pod-creation outage. The constraint is soft (whenUnsatisfiable: ScheduleAnyway), so a single-node cluster still schedules both.

HA datastores require explicit passwords — the render fails without them

postgresql.ha and redis.ha both refuse to render without an explicit password. This is not the requireStrongSecret gate: the operators manage their own credential Secrets (the Sentinel cluster takes its password through customConfig) and cannot consume the chart-generated value.

redis.ha.enabled=true requires an explicit redis.password (--set redis.password=... or a sealed
secret) — the sentinel cluster takes its password through customConfig and cannot use the
chart-generated one.

values-prod.yaml deliberately blanks both, so the overlay does not render on its own. Supply your own, or better, point redis.existingSecret / postgresql.existingSecret at Secrets you manage — a --set password lands in helm history.

Dry-run the render before you install. It costs nothing, needs no cluster, and catches every guard at once — the chart also validates values against its values.schema.json, so a bad enum or type is rejected with a path and a message before anything applies:

helm pull oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 --untar

helm template norviq ./norviq -f norviq/values-prod.yaml \
  --set-json 'policyQuotaNamespaces=["prod-agents","analytics"]' \
  --set postgresql.password="$PG_PASSWORD" \
  --set redis.password="$REDIS_PASSWORD" >/dev/null

Then install:

# install CloudNativePG, a redis-operator and metrics-server first, then:
helm upgrade --install norviq ./norviq -n norviq --create-namespace \
  -f norviq/values-prod.yaml \
  --set-json 'policyQuotaNamespaces=["prod-agents","analytics"]' \
  --set postgresql.password="$PG_PASSWORD" \
  --set redis.password="$REDIS_PASSWORD"

You do not need to pass api.secretKey. Left at its sentinel default the chart auto-generates a strong random JWT secret on first install and reuses the live one across upgrades, so upgrades never invalidate sessions. Pass it explicitly only to pin your own — a controlled rotation, or a multi-cluster fleet trust root:

# Rotate on the EXISTING release. -n norviq targets the real release (not a stray one in `default`);
# --reset-then-reuse-values keeps every other value; --set-string keeps the base64 +/=/ characters intact.
helm upgrade norviq oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 -n norviq \
  --reset-then-reuse-values \
  --set-string api.secretKey="$(openssl rand -base64 48)"

The single-replica StatefulSets are auto-disabled once *.ha.enabled is set — the operators own the datastores, and both the connection URLs and the initContainer readiness gates retarget the HA services (*-rw / the failover Service) automatically.

Node scaling. The HPAs scale pods. For the cluster to grow nodes to hold them, enable your cloud’s cluster-autoscaler on the node pool and size it to hold at least the HA floor (2× api + engine + webhook, plus the data tier).

Runtime guarantees (live on the base chart, not prod-only):

• initContainers gate api/engine on Postgres and Redis, and the webhook on the API being reachable — so Helm’s apply order never matters. The waits run runAsNonRoot / runAsUser: 65534 / readOnlyRootFilesystem / all caps dropped, with bounded requests and limits, so the container that runs before the hardened app container is not the least-hardened thing in the pod.
• /readyz returns 503 when Postgres, Redis or OPA is unreachable — the pod drains traffic without CrashLooping, and self-heals on reconnect. No manual restart.
• preStop sleep + terminationGracePeriodSeconds: 30 drain in-flight requests during a rollout.

OPA sidecars are loopback-only and carry no kubelet probes — by design. OPA’s admin API is unauthenticated and read-write, so the sidecar binds 127.0.0.1:8181 only. A kubelet probe dials the pod IP, so any probe on that container would be refused forever and pin the pod NotReady; an exec probe is impossible because the -static OPA image is distroless. Instead the app’s own /readyz calls OPA over localhost and ANDs it into readiness — a dead OPA still removes the replica from the Service, and it proves the real consumer can reach OPA. The accepted trade-off: a wedged-but-listening OPA is drained, not auto-restarted. If you are alerting on “is OPA up”, alert on the app’s /readyz; do not add a probe to the OPA container.

7. Resources

Several of these limits are measured, not guessed, and lowering them has a specific cost.

Container	Requests	Limits
norviq-api	100m / 128Mi	2000m / 1Gi
norviq-engine	100m / 128Mi	500m / 256Mi
norviq-ui	50m / 64Mi	200m / 128Mi
norviq-webhook	50m / 64Mi	200m / 128Mi
OPA sidecar (api + engine)	50m / 64Mi	1500m / 512Mi
internal-TLS proxy sidecar	10m / 24Mi	— / 64Mi
Injected sidecar, proxy mode	50m / 64Mi	200m / 128Mi
Injected sidecar, embedded mode	200m / 256Mi	2000m / 384Mi

Control-plane floor, one replica of each component, no injected sidecars. The OPA sidecar runs once inside the norviq-api pod and once inside the norviq-engine pod; the internal-TLS proxy sidecar runs only inside the norviq-api pod (§3) — so the norviq-api pod alone carries three containers (api + OPA + tls-proxy), the norviq-engine pod carries two (engine + OPA), and norviq-ui/norviq-webhook are single-container pods. Summed:

• Requests (this is the number the scheduler bin-packs against): 410m CPU / 536Mi memory across the four pods.
• Limits: memory sums to 2624Mi (~2.6Gi); CPU does not sum to a hard ceiling because the internal-TLS proxy sidecar has no CPU limit (— in the table above), so the true CPU ceiling is “at least ~5.9 CPU” rather than a fixed number.

This is a single-replica floor, not a production sizing number — api and webhook default to 2 replicas each (§6 HA), which roughly doubles their share of the above, and postgres/redis/OPA’s own resources (already broken out in Configuration) are separate from this table.

Why the two large numbers:

• api limits 2000m/1Gi. At 500m the API pod was throttled on 92.5% of CFS periods under a sustained single-client load — 23.8s of throttle across 400 evaluations, ~60ms added to every call. Raising the limit alone took caller-observed p50 from 115ms to 62ms and p95 from 172ms to 101ms, with zero throttled periods. Memory moved for a different reason: the 214Mi-against-a-256Mi original limit was measured at idle on AKS with a single uvicorn worker, before --workers existed. api.workers: 4 now starts four separate processes, each with its own heap (~150Mi resident) — call it ~600Mi+ resident with four workers running, which the old 256Mi limit could never hold. 1Gi is sized for four workers with real headroom, not a re-measurement of the four-worker pod at 214Mi. Only the limits moved — requests stay at 100m/128Mi so the chart still schedules on a small cluster, and a limit that is never reached costs nothing.
• OPA limits 1500m/512Mi. Rego compilation is a CPU and memory burst that happens whenever a policy is created or edited, and OPA stops answering queries while it recompiles. Throttled at 250m that burst ran past the engine’s 2s evaluation budget, so the first tool call after every policy save was wrongly blocked as evaluator_timeout — measured 0/5 correct at 250m, 5/5 at 1500m. Steady-state evaluation uses ~7m CPU and ~117Mi, so these limits exist for the compile burst, not the query path.
• The embedded injected sidecar’s 2000m CPU limit is load-bearing. At 500m, embedded mode was slower than proxy mode with 58.9% CFS throttling.

api.workers and api memory must move together

Each uvicorn worker is a full process holding its own interpreter, SQLAlchemy pool and OPA client — roughly 150Mi resident. The base profile pairs workers: 4 with a 1Gi limit. values-light.yaml cuts memory to 384Mi and sets workers: 1 for exactly this reason; inheriting 4 workers under a 384Mi cap OOMKills the pod in a loop, and the symptom is an install timing out on “Pending termination”, which reads as a chart or cluster problem rather than as two numbers that stopped agreeing.

values-prod.yaml pins api.resources.limits to 1 CPU / 512Mi while leaving api.workers: 4. If you use that overlay, either raise the memory limit or lower workers to match — the base chart’s own arithmetic says 4 workers want closer to 1Gi.

config.inprocCacheTtlS is the other production lever, and it is opt-in (0 = disabled; recommended production value 5). It is a per-pod in-process L1 cache on the enforcement hot path: measured warm read p50 21.9ms → 3.2ms, floor 14.5ms → 1.4ms. What it costs you is convergence — a posture or threshold change from the console is not observed by an already-warm pod until the entry expires. What it never caches is the incident-response path: the admin freeze (agent_frozen) and the trust cap are read fresh on every call, so a freeze still takes effect on the very next call regardless of this setting.

8. PodSecurity and container hardening

securityContext.enabled: true (the default) applies a restricted profile inline on every container the chart owns, rather than assuming a PodSecurity Admission or Kyverno restricted-profile mutation exists on the target cluster:

securityContext:
  enabled: true
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

Applied to api, engine, webhook, the OPA sidecars, the internal-TLS proxy sidecar and the dependency-wait initContainers. Set securityContext.enabled: false to defer entirely to a cluster-level policy.

Two honest caveats:

• readOnlyRootFilesystem is not forced on the api/engine Python app containers. They write to tmp at runtime. The CIS-critical controls (non-root, no privilege escalation, drop ALL caps, RuntimeDefault seccomp) are applied. The webhook, the sidecars and the hook Jobs do run with a read-only root filesystem.
• The bundled Postgres and Redis are not restricted-compatible. The official images start as root to chown their data directory. Under enforced PodSecurity restricted (or an OpenShift restricted SCC) they sit in an admission rejection loop. Norviq’s own workloads install cleanly under enforced restricted — only these bundled dev-convenience datastores do not. Use external or operator-managed datastores (§4, §6) on any namespace with restricted enforced.

OpenShift

--set openshift.enabled=true \
--set postgresql.enabled=false --set postgresql.host=<host> \
--set redis.enabled=false      --set redis.host=<host>

OpenShift assigns each namespace a UID range and the default restricted-v2 SCC rejects a pod that pins runAsUser to almost any value, because the pinned value is rarely inside the namespace’s allocated range. With openshift.enabled: true the chart omits runAsUser/runAsGroup/fsGroup entirely and lets the platform assign, while keeping runAsNonRoot: true — the guarantee is unchanged, the chart just stops dictating which non-root user.

The chart pre-flights this: openshift.enabled=true with a bundled datastore still on fails the render with an explicit message, rather than leaving you to discover two StatefulSets stuck in a PodSecurity rejection loop with nothing pointing at the cause.

Network-layer egress lockdown (optional)

Norviq’s tool-call PEP is cooperative — the agent’s SDK asks the sidecar for a forward/drop decision and the agent executes the tool. A pod that ignores the SDK can therefore reach tools directly. agentEgressPolicy bounds that at the network layer with a default-deny egress policy for agent namespaces:

agentEgressPolicy:
  enabled: false                 # default
  engine: networkpolicy          # or `cilium` for FQDN allowlisting
  namespaces: []                 # empty => reuse policyQuotaNamespaces
  allowDNS: true
  allowedCIDRs: []               # operator-approved tool endpoints; EVERYTHING else denied
  allowedFQDNs: []               # engine=cilium only
  allowedFQDNPatterns: []        # engine=cilium only
  allowedPorts: []               # empty => all ports
  embeddedDatastores: false      # true only for `embedded` sidecar mode

It does not replace the PEP — per-call parameter policy still needs the SDK — and it requires a NetworkPolicy-enforcing CNI (Calico or Cilium; kindnet silently ignores NetworkPolicy). The chart refuses to render it against the Norviq control-plane namespace, or with no namespaces to lock down. See the Security model for where this sits in the threat model.

9. Cloud (AKS / EKS / GKE)

The chart is cloud-agnostic — there is nothing provider-specific in the templates, so the HA posture in §6 deploys the same on AKS, EKS, GKE or a vanilla cluster. Only the LoadBalancer/ingress annotations differ.

• No cloud-specific overlay ships in the chart. Only values-dev.yaml, values-light.yaml and values-prod.yaml exist. For a tight single-node cloud dev/staging cluster, start from values-light.yaml and add the knobs that matter there: replace-in-place rollouts (maxSurge: 0 / maxUnavailable: 1 — a surge pod can never schedule when the node has no CPU headroom), engine.replicas: 0 (the API evaluates in-process against its own OPA sidecar), and a trimmed opa.resources request. Drop it once the node pool has headroom; the base defaults give zero-downtime surge rollouts and a 2-replica API on their own.
• CRDs need no separate step for an OCI install — they ship in the chart. From a clone, kubectl apply -f helm/norviq/crds/ first.
• Secrets. For dev/staging, passing secrets via --set from CI secrets is reasonable. For production, source api.secretKey, the datastore credentials and (if enabled) the fleet signing key from your cloud’s secret store — e.g. Azure Key Vault via the Secrets Store CSI driver mounted into a pre-created Kubernetes Secret that postgresql.existingSecret / redis.existingSecret / fleet.hub.signingKeySecretName point at (AWS Secrets Manager and GCP Secret Manager have equivalents). Never inline a production secret in a values file that is committed to a repo.
• Recovery. If a rollout wedges (all app deployments crash-looping on a bad or partial roll), scale everything to 0 and bring dependencies up in order — Postgres, then Redis, then api, then engine, then webhook/ui. The initContainers + /readyz combination above is what normally prevents needing this at all.
• Verify the deploy actually applied. Compare the running pods’ image digest against the release you intended. An old pod serving stale traffic behind a “successful” rollout is the most common false positive after a cloud deploy.

10. Upgrades within 0.2.x

helm upgrade norviq oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 \
  -n norviq --reset-then-reuse-values

What survives an upgrade, and why you can run this without re-supplying credentials:

• api.secretKey, the admin password seed, and the bundled datastore passwords are read back out of the live norviq-secrets Secret and reused, so sessions/JWTs stay valid and Postgres is never locked out of its own PVC. The one exception is a JWT secret that is itself still the weak literal default — that one is regenerated, which does invalidate live sessions for that single upgrade. Correct: anyone could already forge tokens against a well-known default.
• The internal CA and API serving cert (norviq-internal-ca, norviq-api-tls) are reused by the pre-upgrade hook, so the CA identity stays stable and the controller’s pinned trust keeps working.
• The webhook TLS cert and caBundle are re-bootstrapped by the pre/post-upgrade hook Job.

What Helm does not upgrade: CRDs

This is a Helm limitation for the crds/ directory, not a Norviq choice. CRDs are installed once, on first install, and are never upgraded or deleted by helm upgrade / helm uninstall.

If a release changes a CRD schema, you must apply it yourself:

# pull the chart for the version you are upgrading TO, then apply its CRDs
helm pull oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 --untar
kubectl apply -f norviq/crds/
# nrvqpolicies.norviq.io, nrvqclasses.norviq.io, nrvqconfigs.norviq.io

kubectl get crd | grep norviq.io

Do this before helm upgrade when the new chart’s templates or controller depend on a new field. A missing CRD field does not fail the Helm upgrade — it fails later, when the controller writes a resource the API server strips or rejects.

Two upgrade knobs worth checking

• auth.requireBoundAgentIdentity defaults to true: every non-admin caller must present a credential that carries its agent_class binding, or gets a 403. agent_class selects which Rego program is enforced, so it has to be attested the way namespace already is — left false, a namespace-scoped API key with an empty agent_class lets the request body pick the class, including one with no policy, which then falls through to config.noPolicyDecision: allow.

  Set it to false only for an in-flight upgrade from 0.1.x whose sidecars and API keys predate agent-class binding. Then re-admit the sidecars (the webhook stamps agent_class from the pod’s norviq.io/agent-class label), re-issue workload API keys with an agent_class, and set it back to true.

• --reuse-values is the upgrade trap. It does not merge in values a newer chart added, so an upgrade across a release that introduces one aborts with nil pointer evaluating interface {}.<key>. Use --reset-then-reuse-values.

Resource naming is fixed

The chart is a cluster-singleton control plane — one PDP and one mutating webhook per cluster, like cert-manager or ingress-nginx. Resource names (norviq-api, norviq-webhook, …) are fixed; nameOverride/fullnameOverride affect only the app.kubernetes.io/name label. Two releases in one namespace is not a supported topology.

11. Multi-cluster fleet

Multi-cluster (fleet) mode is opt-in and off by default (fleet.enabled: false) — a single-cluster install renders zero fleet resources and behaves exactly like §1/§6.

The model: one hub centrally monitors and manages any number of spoke clusters, each an otherwise-normal single-cluster Norviq install. Every hub↔spoke interaction is spoke-initiated and outbound: the spoke calls the hub once to enroll, the spoke’s relay POSTs heartbeats and rollups on an interval (fleet.relayIntervalSeconds, default 60), and the spoke’s puller GETs the hub’s signed policy bundle, verifies it locally, and applies it (fleet.pullIntervalSeconds, default 60). The hub never dials into a spoke — joining a cluster to a fleet needs only spoke→hub outbound traffic; no inbound access to the spoke is required.

flowchart TB
    subgraph hub["Hub cluster"]
        fapi["fleet-api"]
        fpg[("fleet Postgres")]
        sign["Signed bundle<br/>(private signing key)"]
        fapi --- fpg
        fapi --- sign
    end
    subgraph spokeA["Spoke cluster A"]
        relayA["relay → heartbeats / rollups"]
        pullA["puller → verify + apply bundle"]
    end
    subgraph spokeB["Spoke cluster B"]
        relayB["relay → heartbeats / rollups"]
        pullB["puller → verify + apply bundle"]
    end
    relayA -->|outbound POST| fapi
    relayB -->|outbound POST| fapi
    fapi -->|signed bundle GET| pullA
    fapi -->|signed bundle GET| pullB

All arrows are spoke-initiated and outbound; the hub is never a client of a spoke.

Bring up a hub:

helm upgrade --install norviq oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 \
  -n norviq --reset-then-reuse-values \
  --set fleet.hub.enabled=true \
  --set-file fleet.hub.signingKey=./fleet-signing-priv.pem \      # RS256 private key — HUB ONLY
  --set-file fleet.bundlePubkey=./fleet-signing-pub.pem           # this cluster's own trust root

This renders the norviq-fleet-api deployment plus its own dedicated Postgres (fleet.hub.pgUrl / fleet.hub.postgresql.*). In production prefer fleet.hub.signingKeySecretName (a pre-created Secret) over inlining the private key via --set-file. With config.requireStrongSecret: true the render fails on an empty or shipped-default (norviq_dev) fleet DB password, including one embedded in fleet.hub.pgUrl.

Enroll a spoke — no per-spoke --set apiUrl/bundlePubkey needed. Install the spoke plain (single-cluster default), then join it with a token minted at the hub:

1. Hub console: Fleet → Add cluster → enter the spoke’s cluster id and the hub URL the spoke can reach → Mint join token (POST /api/v1/fleet/clusters/join-token — admin-only, short-lived, single-use, cluster-scoped).
2. On the spoke: norviq fleet join <token>. This claims the token at the hub (replay → 409), persists the enrollment so it survives restarts, and starts the relay and puller.
3. norviq fleet status shows whether the cluster is single-cluster or enrolled, and to which hub. norviq fleet leave de-enrolls and sheds any pushed policy, reverting to single-cluster.

The trust root is the bundle public key (fleet.bundlePubkey). The join token carries it to the spoke alongside the signed token itself, and every signed bundle the spoke pulls is verified against it. It defaults to empty, and an empty pubkey means the spoke applies no bundle at all — fail-closed, not fail-open. The private signing key never appears in a token and never leaves the hub.

On the hub console, set ui.fleetApiUrl: "/fleet-api" so the console shows the Fleet nav and cluster selector (same-origin, nginx-proxied to norviq-fleet-api). Leave it empty on spokes and single-cluster installs — the Fleet page redirects away when fleet is not enabled.

Data classes, so you know what actually centralizes at the hub: bounded rollups and summaries (cluster status, agent list + trust, coverage %, graph summaries) are relayed and shown at the hub labelled with freshness; policy authoring and apply is push-signed-bundle, never a direct write to the wrong cluster; raw audit records never leave the spoke when fleet.residency is set — the hub only deep-links to the spoke’s own console for that.

12. Identity (SPIFFE/SPIRE)

config.spiffeMode controls how an agent’s identity is resolved:

• mock (default) — identity comes from environment variables set on the pod. Works with no additional infrastructure; used for local dev, tests and the attack suite.
• workload-api — the sidecar fetches a real X.509 SVID from the SPIFFE Workload API socket (config.spiffeSocket, default /spiffe-workload-api/spire-agent.sock). Fail-closed: a socket or SVID error blocks the call rather than silently falling back to an env-var identity. This mode requires SPIRE on the cluster and the SPIFFE CSI driver.

config:
  spiffeMode: workload-api
  spiffeSocket: /spiffe-workload-api/spire-agent.sock
  spiffeCsi:
    enabled: true            # mount the CSI volume onto api/engine pods
webhook:
  spiffe:
    inject: true             # injector mounts the socket into every injected agent workload

Leave config.spiffeCsi.enabled: false (the default) on any cluster without SPIRE and the CSI driver installed — a csi.spiffe.io volume with no matching driver or registration wedges pod creation.

See SPIFFE identity for the full setup.

13. Clean uninstall

NrvqPolicy custom resources carry the finalizer norviq.io/policy-protection — the CRD controller (in the webhook) adds it so a policy cannot vanish without the controller first syncing the deletion out of the API. Helm removes that controller in the same uninstall, which used to strand every CR in Terminating forever.

As of the shipped chart this is handled for you. crdFinalizerCleanup.enabled defaults to true and renders a pre-delete hook Job that releases the finalizers on every nrvqpolicy, nrvqclass and nrvqconfig while the API is still up, before Helm deletes anything. It uses the first-party bootstrap image and plain authenticated REST calls against the API server — no kubectl.

helm uninstall norviq -n norviq
kubectl delete namespace norviq

# CRDs live OUTSIDE the release — Helm never deletes them. Remove them explicitly if you want to:
helm pull oci://ghcr.io/norviq-dev/charts/norviq --version 0.2.5 --untar
kubectl delete -f norviq/crds/

The cleanup Job is best-effort by design: it swallows patch errors and always exits 0, because a cleanup hook must never be the reason an uninstall fails. A stuck finalizer is recoverable by hand; an un-uninstallable release is worse.

Disable it (crdFinalizerCleanup.enabled: false) only if your cluster forbids hook Jobs — and then strip the finalizers yourself before uninstalling:

kubectl delete nrvqpolicy --all --all-namespaces   # while the webhook/controller is still up
kubectl get nrvqpolicy -A                          # expect: No resources found
helm uninstall norviq -n norviq

Recovering CRs stuck in Terminating

If the hook is disabled or could not run (most commonly: crdFinalizerCleanup.image pointed at an image that no longer exists, so the pod sat in ImagePullBackOff and the uninstall died on a deadline), the controller is gone and cannot strip its finalizer. The CRs — and any namespace still holding them — hang in Terminating. Patch the finalizer off directly:

kubectl patch nrvqpolicy <name> -n <ns> --type=merge -p '{"metadata":{"finalizers":null}}'

Repeat per stuck CR (or script it across kubectl get nrvqpolicy -A), then the namespace finishes terminating on its own.

Clear these before reinstalling. A reinstall adopts the stranded tombstones, the controller loops on a delete the API answers 422, no policy ever loads, and the cluster blocks every tool call on no_policy_loaded while looking perfectly healthy.