Skip to content

Troubleshooting

Norviq failing is not self-announcing. When the engine is unreachable and the posture is fail-closed, every governed tool call stops and the first signal anyone gets is a user saying the bot “got dumber”. When the posture is fail-open — the shipped default — the calls are forwarded ungoverned and nothing visibly changes at all.

This page is the diagnostic path for both. It covers what the product can prove about its own health, what it deliberately refuses to claim, and where the evidence lives.

flowchart TD
    A["Agents' tool calls<br/>stopped, or are running<br/>unjudged"] --> B{"norviq status<br/>API / Redis / DB"}
    B -->|"any Offline<br/>or Disconnected"| C["Control plane is down.<br/>kubectl -n norviq get pods,<br/>then GET /readyz for the<br/>full dependency set"]
    B -->|"all up"| D{"GET /api/v1/system-health<br/>status"}
    D -->|degraded| E["Read issues[].id —<br/>each carries detail +<br/>remediation"]
    D -->|unknown| F["No real governed call in<br/>15 min. Data plane is idle<br/>OR severed. Not an all-clear."]
    D -->|ok| G["The control plane saw real<br/>traffic and no infrastructure<br/>verdict. Look at policy, not infra."]
    F --> H["Check the sidecar/SDK side:<br/>pod logs for NRVQ-SDC-3031/3036<br/>or NRVQ-SDK-1013/1014"]
    E --> I["Engine faults → engine + OPA logs.<br/>Credential bands → rotate.<br/>See the tables below."]

norviq status is the fastest check. It calls /healthz and /readyz and prints three lines:

Terminal window
norviq --api-url https://norviq.example.com status
API: Online
Redis: Connected
DB: Connected

/healthz is liveness only — it returns {"status": "ok"} with no dependency checks, deliberately, so a transient Postgres or Redis blip does not CrashLoop the API pod.

/readyz is the load-bearing one. It actively probes the hard dependencies and returns 503 when any is unreachable, which flips the pod NotReady and drains traffic off it. It reports more than the CLI prints, so read it directly when norviq status looks clean but something is still wrong:

Key Meaning
db SELECT 1 succeeded (this probe also recycles a dead pooled connection)
redis The cache client answered PING
policies_warm The policy loader finished its warm load. A cold replica is held out of the Service rather than evaluating the first tool calls against an empty policy set
opa Present only when config.opaMode: server (the default). The OPA health check answered
Terminal window
kubectl -n norviq port-forward svc/norviq-ui 8080:80 &
curl -sS http://localhost:8080/readyz | jq
{"status": "ready", "redis": true, "db": true, "policies_warm": true, "opa": true}

A not-ready pod logs nrvq.api.not_ready with code NRVQ-API-7002 and the same field set. Both /healthz and /readyz are unauthenticated and are excluded from HTTP rate limiting.

The injected sidecar has its own /readyz on its HTTP fallback listener, and it is equally load-bearing: in embedded mode it gates on the pod’s own OPA being reachable, returning 503 with {"status": "degraded", "opa": false} and logging NRVQ-SDC-3013. OPA binds loopback, so the kubelet cannot probe it directly — this endpoint is where that gating lives.

2. /api/v1/system-health — degraded, ok, and unknown

Section titled “2. /api/v1/system-health — degraded, ok, and unknown”

This route answers one question: is something wrong right now? It reports only what it can prove from decisions the data plane actually recorded in the last 15 minutes, and it is scoped like every other read — a non-admin sees only their own namespace.

It has three states, and the third one matters as much as the first.

At least one issue. decisions_in_window is null because an incident is already substantiated.

{
"status": "degraded",
"issues": [
{
"id": "evaluator_timeout",
"severity": "critical",
"title": "Tool calls are timing out in the engine",
"detail": "The policy engine is reachable but not answering within the evaluation budget, so calls are being refused fail-closed. These are timeouts, not policy decisions — no rule denied them.",
"remediation": "Check norviq-engine and OPA latency and CPU, and the sdk_timeout_ms budget for this deployment.",
"affected_calls": 38,
"namespaces": ["agents"],
"last_seen": "2026-08-19T09:41:07.220481+00:00",
"window_minutes": 15
}
],
"window_minutes": 15,
"decisions_in_window": null
}

The API logs nrvq.api.system_health.degraded (NRVQ-API-7090) with the issue ids on every such response.

No infrastructure verdict and the data plane demonstrably reached the API.

{
"status": "ok",
"issues": [],
"window_minutes": 15,
"decisions_in_window": 412,
"evidence": "412 real governed tool calls reached this API in the last 15 min and none carried an infrastructure verdict."
}

No infrastructure verdict, and no real governed call was recorded either. The route will not call that healthy, because a data plane that has been severed from the API writes nothing and looks identical to a healthy quiet one.

{
"status": "unknown",
"issues": [],
"window_minutes": 15,
"decisions_in_window": 0,
"evidence": "No real governed tool call was recorded in the last 15 min (Policy-Tester and red-team rows do not count — the console writes those itself). The data plane is either idle or unable to reach this API, and the two are indistinguishable from here, so this is not an all-clear."
}

decisions_in_window: null with a status of unknown is a different failure: the liveness read itself failed (nrvq.api.system_health.liveness_unavailable, NRVQ-API-7091, with NRVQ-API-7092 from the query). “We could not look” is never reported as “we looked and it is clean”.

The banner polls every 30 s and is dismissible per issue, but a dismissal is keyed to the issue and its last_seen — a recurrence re-raises it rather than staying hidden.

The liveness count deliberately excludes two populations that reach audit_log with no data plane involved, both written by the API’s own in-process emitter:

  • the Policy Tester, which POSTs /evaluate under an ephemeral policy-tester-<rand> agent class
  • a red-team run, tagged framework="redteam"

Counting either would let an operator whose sidecars are severed get a green all-clear the moment they opened the Policy Tester to find out why nothing works. The same classifier also drops probe and e2e identities (e2e-, probe-, smoke-, canary-, evtrace-, scorer, and siblings).

The incident query is not filtered the same way, and that asymmetry is intentional: an engine fault recorded during a red-team run is still a genuine engine fault. Evidence of a fault is counted generously; evidence of health is counted strictly.

These are the rule_id values that mean Norviq itself decided this, as opposed to a policy deciding it. /system-health keys on exactly this set.

rule_id Minted by Behaviour Log code
evaluator_error Engine — persistent OPA evaluation failure block, fail-closed NRVQ-ENG-2057
evaluator_timeout Engine — the 2.0 s evaluation budget was exceeded block, fail-closed NRVQ-ENG-2020 / NRVQ-ENG-2021
evaluator_fallback Engine — evaluation raised something unclassified block, fail-closed NRVQ-ENG-2003
policy_load_pending Engine — policy subsystem not warm when the call arrived block, fail-closed NRVQ-ENG-2056
thin_proxy_fail_closed Proxy sidecar — engine unreachable or 4xx block NRVQ-SDC-3031
thin_proxy_fail_open Proxy sidecar — engine unreachable, posture allow allow, ungoverned NRVQ-SDC-3036
engine_rejected_request SDK client — engine answered 4xx block, always NRVQ-SDK-1014

A transient engine error does not produce evaluator_error: the evaluator retries first, so a clean, well-formed input never yields one. Only a persistent engine fault stays fail-closed.

rate_limit_exceeded is deliberately not in this set. The rate limiter working is not an outage, and treating it as one would raise a critical “Norviq is down” banner every time a busy agent hit its ceiling. It is equally not a policy decision — the limiter fires only when the resolved decision was already allow — so it is excluded from policy-compliance and red-team efficacy surfaces too.

A namespace running enforcement_mode: audit softens an operational block to a logged audit decision, and the stored rule_id picks up a prefix:

  • monitor_would_block: — namespace monitor mode
  • policy_audit_would_block: — a per-policy audit-mode policy

So an engine fault in a monitored namespace is stored as monitor_would_block:evaluator_error, not evaluator_error. /system-health and /audit/stats both fold every stored spelling back to the bare id, so the banner and the engine-error count survive the softening. If you are querying audit_log yourself, match the prefixes or you will read zero during a real fault:

Terminal window
curl -sS -H "Authorization: Bearer $NRVQ_API_TOKEN" \
"http://localhost:8080/api/v1/audit/records?range=24h&limit=500" \
| jq '[.[] | select(.rule_id | test("^(monitor_would_block:|policy_audit_would_block:)?evaluator_"))] | length'

Two rules stay hard even under monitor mode: trust_frozen (an admin explicitly froze that agent — incident response outranks posture) and rate_limit_exceeded (configurable via monitor_exempt_rate_limit, default true, meaning it stays hard).

4. engine_rejected_request vs engine_unavailable_fallback

Section titled “4. engine_rejected_request vs engine_unavailable_fallback”

These two are both minted by the SDK client (norviq/sdk/client/engine.py), and telling them apart is the single most useful distinction on this page: one sends you to debug credentials, the other sends you to debug engine pods.

Failure rule_id Decision Honours fallbackMode?
Engine answered 4xx (401/403 expired or wrong token, 422 malformed request) engine_rejected_request block, always No
Engine 5xx, timeout, connect error, or open circuit engine_unavailable_fallback NRVQ_SDK_FALLBACK_MODEallow by default Yes
NRVQ_SDK_FALLBACK_MODE set to an unrecognised value engine_unavailable_fallback coerced to block n/a — logs NRVQ-SDK-1015

A 4xx is not an outage. The engine answered, and it refused. Treating it as one is both a bad diagnostic — operators go and restart healthy engine pods when the real cause is an expired token — and a bypass: with fallbackMode: allow, every 401/403 would become an ALLOW, so a revoked credential silently turns into a total governance bypass. A 4xx an attacker can provoke is worse still: influence a tool parameter into a 422 and the same fallback allows the call.

engine_unavailable_fallback is emitted in both modes on purpose. The block case is equally worth attributing — an operator staring at blocked agents needs to know the cause was an engine outage rather than their own policy.

The proxy sidecar (webhook.injection.sidecarMode: proxy, the default) uses different names for the same two conditions, and it never mints engine_rejected_request. The 4xx case and the fail-closed outage case share one rule_id and are separated by the reason string:

Condition rule_id reason
Engine unreachable, fallbackMode: allow thin_proxy_fail_open Thin-proxy sidecar could not reach the central policy engine; forwarding UNGOVERNED because the configured fallback posture is allow
Engine unreachable, fallbackMode: block thin_proxy_fail_closed Thin-proxy sidecar could not reach the central policy engine (fail-closed)
Engine answered 4xx thin_proxy_fail_closed Central policy engine rejected the sidecar's request (credential or request error, not an outage)

Both paths retry before any of the above fires, so none of this triggers during a rolling restart:

  • sdk_retry_max_attempts: 2 — three attempts in total.
  • sdk_retry_backoff_base_ms: 100, doubling per attempt.
  • Only transport errors and 5xx are retried. A 4xx breaks out immediately — retrying a refusal just delays the same block.

The circuit breaker is the SDK client’s only (the sidecar’s remote evaluator has retries but no breaker): sdk_circuit_fail_threshold: 3 consecutive failures opens it for sdk_circuit_reset_after_ms: 2000, during which calls short-circuit straight to the fallback.

A 4xx never counts toward the breaker, and that is load-bearing. The breaker is checked at the top of evaluate(), before the 4xx rule — so if 401s counted, the third consecutive one would open the circuit and every later call would short-circuit into the fallback and never reach the “4xx always blocks” rule. With the shipped defaults that would flip an expired credential from fail-closed to fail-open. Worse, the breaker only resets on a success, so a permanently bad credential would never recover and would settle into a mostly-ungoverned state. The 30-day credential cliff in §5 produces precisely that 401 storm.

Both credentials the webhook injects are 30-day and are baked into an immutable pod spec at admission: the service JWT, and the mTLS client certificate (NotAfter: now.Add(30*24*time.Hour)). Nothing renews either in place — there is no rotation loop in the webhook and nothing in the sidecar re-reads a credential. A pod that keeps running for thirty days ends up holding two expired ones.

What happens then is correct and is not a bug: the API answers 401, the sidecar sees a 4xx, and a 4xx overrides fallbackMode to fail closed. Every injected pod stops being able to make a tool call, on a timer.

The forewarning band is the mitigation. It is the one /system-health entry not derived from the audit log — the thing it reports has not happened yet and therefore wrote no rows.

Property Value
Warning window 7 days before expiry
Severity warning, never critical — nothing is broken yet, and raising it as critical would train operators to ignore the band that does mean an outage
Appears After the live incidents, so a “will break on Tuesday” never outranks an “is broken now”
Applies to role=service principals only. A human session is short-lived by design

Two bands, because the two kinds share nothing an operator acts on. The kind is decided by the workload claim, which only the injector mints:

issues[].id What it is Remediation
sidecar_credential_expiring An injected sidecar (workload claim present) Roll the affected Deployments. Replacement is how these rotate — a new pod is admitted with freshly minted credentials
service_key_expiring Any other role=service principal: an operator-minted key, an MCP proxy, the fleet relay, a CI caller Mint a replacement key with the same role and namespace, update whatever presents it, then revoke the old one. Rolling a Deployment does not rotate these

Each band carries an expiring[] array of up to 20 rows with kind, namespace, subject, expires_at and days_left.

Short-lived credentials are excluded. A credential whose entire lifetime is shorter than the 7-day window was born inside it, so “expires within 7 days” carries no information about it. The webhook controller signs itself a one-hour service JWT and re-mints it at 60 s to expiry; without this rule that warning lit up on a fresh install, marked the whole system degraded, and could never clear. Lifetime (exp - iat), not remaining time, is the discriminator. A token with no iat warns anyway — missing a real expiry is worse than one extra band.

The band lives in Redis, not Postgres. Records are written on the authentication path under cred_exp:<kind>:<namespace>:<subject>, only once the credential is already inside the window, with nx=true (one write per subject, not one per request) and a TTL equal to the credential’s own remaining lifetime. So a healthy fleet does zero writes on the hot path and no cleanup job is needed.

Redis being unreadable silently removes the warning. expiring_soon returns [] on any failure and logs NRVQ-API-7121, rather than 500-ing the one page an operator opens during an incident. Absence of a band is not evidence that nothing is expiring. Likewise the observe-side write is best-effort (NRVQ-API-7120) — a reporting write must never fail authentication.

nx means a rotated credential does not immediately overwrite the older record. The stale value is always the earlier expiry, so the warning appears early and clears when the old key times out. Warning too early is a nuisance; warning too late is the bug.

The mutating webhook fails closed: failurePolicy: Fail (default), timeoutSeconds: 5. If the webhook is down, pod CREATEs it routes are refused.

Which pods it routes is bounded twice:

  • namespaceSelector — the namespace must be labelled norviq-injection=enabled, and kube-system, kube-public, kube-node-lease, the release namespace, and AKS control-plane namespaces are excluded in-chart.
  • objectSelector — with webhook.injection.gateOnlyAgentPods: true (the default), only pods carrying a norviq.io/agent-class label are routed at all. This keeps the fail-closed guarantee where it matters (an agent pod cannot start un-injected) while a Norviq outage does not stop the namespace’s database or ingress controller from being created.

Refusals arrive as the kubectl error on the pod CREATE. The message is written to be actionable:

Code Denial What to do
NRVQ-WHK-4034 Enforcement-integrity denial. The pod carries injector-owned plumbing but is not fully injected — the norviq-socket volume, a mount at the socket path, a pre-set NRVQ_SOCKET_PATH env, a container named norviq-sidecar, or a sidecar-image container that overrides command/args (a “neutered decoy”) Remove the norviq sidecar / norviq-socket volume+mount / NRVQ_SOCKET_PATH env from your pod spec, or run the pod in a namespace without norviq injection. The injector cannot safely wire over pre-placed plumbing, and skipping would run the pod unpoliced
NRVQ-WHK-4041 MCP injection denial. The pod asked for MCP governance via norviq.io/mcp-servers and it could not be applied — the annotation names a container the pod does not have, or names one that sets no explicit command Fix the annotation or the container spec. Admission fails closed rather than running an MCP server unpoliced
NRVQ-WHK-4052 Sidecar credential Secret unavailable and webhook.injection.credentialSecret.required: true forbids the pod-env fallback Check the webhook’s RBAC for secrets in that namespace. The message names the underlying write error
(generic) sidecar patch creation failed: … — most commonly an unauthorized sidecar image Check the webhook’s NRVQ_SIDECAR_IMAGE against the injector’s image allowlist. The message repeats the underlying error

Two more are logs, not denials, and both matter because the cluster keeps working while something is quietly different:

  • NRVQ-WHK-4049 — the credential Secret write failed and the injector fell back to a literal pod-env credential. The token and client key are then readable by anyone with get pod in that namespace; the built-in view role grants that and does not grant get secrets. Set webhook.injection.credentialSecret.required: true to refuse admission instead.
  • NRVQ-WHK-4062 — the configured sidecar image is not on the injector’s allowlist, so the webhook ignored it and used the built-in default at startup. It does not exit, because refusing to start leaves the same cluster-wide denial in place under failurePolicy: Fail — it would convert a loud misconfiguration into an identical outage.

Other things worth grepping the webhook log for: NRVQ-WHK-4007 (pod opted out of injection), NRVQ-WHK-4009 (pod-level opt-out is disabled cluster-wide, injecting anyway), NRVQ-WHK-4008 (pod already fully injected, skipped), NRVQ-WHK-4037 (no API secret available to mint a sidecar token — the thin-proxy sidecar will fail closed).

Terminal window
kubectl -n norviq logs deploy/norviq-webhook -c webhook --since=30m | grep NRVQ-WHK-

audit_log, in Postgres, append-only and range-partitioned by month on timestamp_utc. Indexed on timestamp_utc, (namespace, agent_id) and decision.

Retention is a background pruner in the API. One loop — hourly by default (audit_retention_prune_interval_s, 3600; env-only, there is no chart key for it) — deletes rows older than config.retention.auditRetentionDays (default 30). A value <= 0 keeps rows forever. One pruner runs per API replica; every statement is idempotent, so concurrent sweeps are harmless. Nothing the pruner touches is read by the evaluator, so pruning can never change a decision.

This is where “the audit log is empty” usually resolves.

Deployment Writes the row framework
Proxy sidecar (webhook.injection.sidecarMode: proxy, default) The central /evaluate — the sidecar has no local emitter, on purpose sidecar
Sidecar HTTP fallback, proxy mode The central /evaluate sidecar-http
Embedded sidecar (webhook.injection.sidecarMode: embedded) The sidecar’s own emitter, writing directly to Postgres sidecar
SDK / MCP proxy calling the API The central /evaluate sdk, and so on
Policy Tester The API’s own emitter, no data plane involved (ephemeral policy-tester-<rand> agent class)
Red-team run The API’s own emitter redteam

The consequence: if the central API is unreachable, the decisions taken during that outage are not recorded anywhere. The proxy sidecar’s thin_proxy_fail_open / thin_proxy_fail_closed verdicts and the SDK’s engine_rejected_request / engine_unavailable_fallback verdicts exist only in the PEP’s own process log for the duration. Ship those pod logs somewhere if you need an outage reconstruction; the audit log will show a gap, not the verdicts.

Audit emission on the API path is fire-and-forget (a background DB write plus an OTel span), bounded by a semaphore so a flood of tool calls cannot exhaust the DB pool and starve every other endpoint.

Terminal window
# Recent blocks, table output.
norviq audit list -d block --range 24h --limit 50
# Aggregates for a namespace, including the engine-error count.
norviq -o json audit stats --range 24h -n agents | jq

/audit/stats returns an engine_errors field: the count of rows whose rule_id resolves to an infrastructure rule, prefix-aware, so it stays correct in a monitored namespace and does not absorb a working rate limiter. An engine_errors spike reads as engine health, not as a wall of policy blocks.

The CLI has no rule_id filter. Use the API for that — rule_id is an exact match, so include the softening prefix if the namespace runs monitor mode:

Terminal window
curl -sS -H "Authorization: Bearer $NRVQ_API_TOKEN" \
"http://localhost:8080/api/v1/audit/records?rule_id=evaluator_timeout&range=24h&limit=100" | jq

Other filters on GET /api/v1/audit/records: namespace, decision, tool_name and agent (case-insensitive substring, applied server-side across the whole range — not a client-side page filter), framework, exclude_synthetic=true (real traffic only, reconciling exactly with the Overview KPIs), range (1h|6h|24h|7d|30d), limit (max 500) and offset. GET /api/v1/audit/records/{id} adds the full payload.

GET /api/v1/audit/export is the always-available authenticated export — format=ndjson|csv, streamed in pages so a large table is never loaded into memory. signed=true (NDJSON only) emits a tamper-evident hash-chained stream: each record carries a _chain link and the stream ends with a _manifest line whose chain tip is HMAC-SHA256-signed when an export signing key is configured.

WS /ws/audit is the live feed behind the console’s Audit Log. The JWT rides in the handshake as Sec-WebSocket-Protocol: nrvq-audit-jwt, <token> — browsers cannot set Authorization on a WS handshake, and a ?token= query string would leak the credential into access logs and browser history. The query-param and Authorization paths remain as a deprecated fallback for non-browser clients. ?namespace= scopes the stream. A close code of 1008 means the token was invalid, revoked, or is still flagged must_change (NRVQ-AUTH-14018) — the same fail-closed gate every REST route applies.

Code Event
NRVQ-API-7002 API replica went not-ready — the payload names which dependency
NRVQ-API-7090 /system-health returned degraded; the payload lists the issue ids
NRVQ-API-7091 / NRVQ-API-7092 The health route’s own liveness probe could not run
NRVQ-API-7121 The credential-expiry read failed — expiry warnings are silently off
NRVQ-ENG-2056 Policy subsystem not ready; call blocked fail-closed
NRVQ-ENG-2057 Persistent OPA evaluation failure, or a missing module
NRVQ-ENG-2020 / 2021 Evaluation exceeded the 2.0 s budget
NRVQ-ENG-2003 Unhandled evaluator fault, fail-closed
NRVQ-ENG-2059 / 2060 A would-block was softened by monitor mode / per-policy audit mode
NRVQ-SDC-3031 Sidecar failed closed — engine unreachable or refusing (nrvq.sidecar.remote_evaluator.fail_closed)
NRVQ-SDC-3036 Sidecar failed open — calls forwarded ungoverned (nrvq.sidecar.remote_evaluator.fail_open, rule_id=thin_proxy_fail_open). This is the signal that agents are running unjudged right now. It moved off NRVQ-SDC-3032 in 0.2.5: a filter written for that code’s two startup INFO lines was silencing it
NRVQ-SDC-3032 Proxy-mode startup only — nrvq.sidecar.mode.proxy and nrvq.sidecar.remote_evaluator.mtls_enabled, both INFO. Safe to filter as noise; it no longer carries the fail-open
NRVQ-SDC-3033 Unanticipated failure in the remote evaluator (nrvq.sidecar.remote_evaluator.unexpected_error). Also used by the embedded-mode startup line
NRVQ-SDC-3035 Sidecar is blocking fail-closed while it waits for the API to create the policy schema
NRVQ-SDK-1013 SDK fallback taken (also logged when the circuit breaker is open)
NRVQ-SDK-1014 SDK call rejected by the engine with a 4xx
NRVQ-SDK-1015 NRVQ_SDK_FALLBACK_MODE is not allow or block; coerced to block
NRVQ-WHK-4034 / 4041 / 4052 Admission refused — see §6
NRVQ-WHK-4049 Sidecar credentials fell back into the pod spec
NRVQ-WHK-4062 Configured sidecar image rejected by the allowlist; built-in default used

NRVQ-WHK-4034 labels two different events — an enforcement-integrity denial in the admission handler and a cross-namespace policy rejection in the CRD controller. The collision is pre-existing; disambiguate by which component logged it.