/production-audit

Production Audit

Using this skill: announce "Using production-audit", make a todo per step below, and do not skip the gates. This skill's worth is its process, not a hand-reproduced outcome. If you were told to "run production-audit", run it, do not improvise the result. (Suite standard: https://github.com/horizon-foundry/foundry/blob/main/reference/skill-authoring.md)

Overview

A whole-application audit that ends in a verdict: safe to ship; ready to ship, risks noted; or do not ship. Core principle: evidence or it doesn't ship. Every finding cites code, and every Critical, High, and verdict-driving finding survives an adversarial refutation attempt or gets downgraded. The audit never modifies code.

Every finding carries a kind: risk or improvement. A risk has a path to harm in production (exposure, data or payment corruption, a crash or unavailability, abuse, or a hidden active failure). An improvement is safe today but makes the system more robust, observable, consistent, accessible, or faster. The distinction is load-bearing: only risks drive the verdict; an improvement never caps it. A report with no risks is clear within its assessed scope even if it lists a dozen improvements, because a punch list of betterments is not a reason to hold a release. This is also what keeps the report honest and readable: risks to weigh first, improvements to schedule second, never one undifferentiated wall of problems. A quality tool must be able to say "this is good, here is how to make it better" without that reading as "this is broken." Think of it as bugs versus tech debt. Severity and kind are related but distinct: a finding minor enough to rate low or informational is an improvement (tech debt or polish), not a ship risk (a low rate-limit gap already mitigated upstream, a rare recoverable race, a consistency divergence, an accessibility refinement). Risks are the ship-relevant harm paths, which in practice sit at medium and above. And a medium-or-higher item with no harm path today, missing tests, missing CI, observability thinness, is an improvement too. What earns "risk" is a live exposure, a corruption path, a crash, or a silent availability failure the shipper must consciously accept.

The verdict states what evidence it rests on. It carries an assessedScope, and a static-only run names that base beneath the recommendation ("static review, runtime not exercised") and lists the checks it skipped under notAssessed, so it reads as a human recommendation without overclaiming a runtime-verified result. A verdict may not claim more than the run could see.

The unit of work is the flow, not the file. Each dimension is one way of examining those flows. Eleven dimensions, and applicability is resolved per project for all of them. Six are usually applicable to any product: security, concurrency, reliability, accessibility, ui, infra. Five need a more deliberate applicability call: operability, testing-confidence, data-migration-safety, release-safety, performance-capacity. Any dimension with no surface here (accessibility or ui on a headless service, migration safety with no datastore) is excluded with a one-line reason recorded in meta.scope.excludedDimensions, never forced onto a product it does not fit and never silently skipped.

When NOT to use

• Reviewing a diff or PR: use a diff-review tool (such as /code-review, if installed)
• Security-only question on specific code: use a targeted security review (such as /security-review, if installed)
• Frontend polish/a11y pass with fixes applied: use a frontend-audit skill that modifies code
• The user wants fixes applied: this skill only reports

Scope arguments

Phases

Phase 0: Context load

Read the project's CLAUDE.md, README, and any ARCHITECTURE/DESIGN/PRODUCT docs. Then build three artifacts that every later phase consumes:

1. Flow inventory. Every privileged or mutating flow, end to end: signup/login, invites, uploads, payments, cron jobs, webhooks, LLM calls, deletes. For each: entry point, auth boundary crossed, data written, side effects.
2. Trust-boundary map. Where untrusted input enters (forms, URLs, file uploads, webhooks, LLM output) and where privilege changes (middleware, layouts, DB policies, service-role clients).
3. Derived checklists. Detect the actual stack and derive checks from it; prune checks that cannot apply. Examples of derivation: Supabase means a per-table RLS coverage matrix and a search for service-role key usage; Next.js App Router means per-server-action authz (middleware does not protect actions) and layout-vs-middleware gating; any LLM feature means prompt injection via user content AND unsafe handling of model output (rendered, executed, or persisted). Derive equivalents for whatever the stack actually is. Grep to confirm; never assume.
4. Applicability resolution for all eleven dimensions. Resolve every dimension against the actual project, not just the five conditional ones. Security, concurrency, reliability, and infra apply to essentially any system. Accessibility and ui apply wherever there is a user-facing surface, and are excluded on a headless service, CLI, or backend job (with the reason). The five conditional ones (operability, testing-confidence, data-migration-safety, release-safety, performance-capacity) each need a deliberate call: a product with no database has no migration-safety surface; a static site has no meaningful capacity surface; a project with no deploy pipeline resolves release-safety down to "how does a bad version get pulled back". Applicable ones join the fan-out. Inapplicable ones are recorded in meta.scope.excludedDimensions with status not-applicable and the reason; ones that apply but cannot be assessed by this run (e.g. runtime-only evidence on a static run) are recorded there as not-assessed. Never silently skip.

Phase 1: Mechanical sweeps

Run before any subagent spends tokens: dependency audit (npm audit or ecosystem equivalent), a secret scan (prefer a real scanner such as gitleaks or trufflehog when available, and scan tracked history where it matters; a secret-pattern grep is the fallback, not sufficient on its own), lint/typecheck if configured. Record outcomes in the report's mechanicalSweeps. Feed results to dimension agents as known issues so they do not rediscover them.

Phase 2: Dimension fan-out

One read-only subagent per dimension, in parallel, each given the Phase 0 artifacts. Every finding must cite file and line. If the harness cannot spawn parallel sub-agents (or requires a permission the run does not have), do not skip dimensions: run them sequentially as focused passes in the main context, or stop and ask the user to enable parallel work. Coverage is what matters; the fan-out is only a way to reach it faster. Required artifacts per dimension:

Every dimension returns two things: its findings, AND the verified-safe controls it confirmed are correct (the strong parts). A report that only lists problems makes a healthy app look broken. Naming what is solid is part of an honest audit, and it belongs in the report's strengths.

• security: an authz matrix: every mutation endpoint/action × who can invoke it × where the check lives (middleware, handler, DB policy) × how verified. Gaps in the matrix ARE findings. Plus injection surfaces from the trust-boundary map, secrets handling, tenant isolation per table.
• concurrency: idempotency judgment for every mutation (what happens on double-submit, retry, refresh, two tabs), TOCTOU in check-then-write paths, cleanup of effects/subscriptions/timers, races between background jobs and user actions.
• reliability: a failure-path table per external dependency (LLM, storage, email, DB, analytics): timeout? retry? user-visible failure state? partial success? idempotent on retry? Empty cells are findings. Plus swallowed errors and stuck loading states.
• accessibility: primitives first. Audit the shared UI primitives (modal, menu, form field, button) deeply before sampling feature screens; a primitive defect multiplies across the app. WCAG 2.2 AA is the bar for what static review can assess.
• ui: token conformance against the project's design source of truth, a sweep for divergence across shared components, copy/terminology drift, state coverage (hover, focus, disabled, loading, error, empty).
• infra: deploy config, security headers/CSP, env handling, dependency audit triage, exposed dev/debug endpoints.

The conditional dimensions (only the ones Phase 0 marked applicable):

• operability: can the team tell the system is healthy and act when it is not? Logs/metrics/traces on the critical flows, a dashboard someone actually looks at, alerts with a named receiver, SLOs or at least an explicit "what is too slow/too broken", runbooks or failure notes for the known failure modes, and clear ownership. A system nobody can observe fails silently.
• testing-confidence: not coverage percentage, but whether the tests would catch the failures this report cares about. Are the critical flows exercised end to end? Do contract/integration tests cover the external seams? Crucially: do tests exercise the specific controls other findings rely on (the authz check, the idempotency guard)? A control this report calls verified-safe that no test pins down is one refactor away from silently vanishing.
• data-migration-safety: migration ordering and reversibility (does the migration run before or after the code that needs it; what happens on rollback), backup existence AND restore path (an untested restore is a hope, not a control), deletion/retention behavior, and recovery from partial or corrupted writes.
• release-safety: how a bad version reaches users and how it gets pulled back. Rollback path and its speed, feature flags or staged rollout for risky changes, forward/backward compatibility between the deployed version and its data/clients, and whether deploy and release are separable.
• performance-capacity: latency budgets on the hot paths, query fan-out (N+1 and unbounded queries on real data volumes), resource cliffs (memory, connections, file handles, quota), and what happens at 10x current load. Static review reads the code shape; real latency numbers are runtime evidence.

Pre-ship gates: mobile and instrumentation. These are release gates, not ordinary dimensions: the audit always checks them and reports where each stands. How a gate failure weighs the verdict follows the project's declared release policy (see foundry): a required gate's failure caps the verdict, an optional gate's failure is reported but does not by itself cap it, and a waived gate is recorded with its reason (for instrumentation, a documented no-telemetry or trust-boundary posture is a legitimate waiver). Absent a declared policy, default both to required for any real audience. A gate is never silently omitted, whatever its disposition.

• Mobile (apply the mobile skill's discipline). The product must hold up on a real device: zero horizontal overflow at a real device width, touch targets, the keyboard not occluding controls, touch vs pointer handled. A core flow that breaks on a phone fails the mobile ship gate, and a failed required gate blocks a broad launch. Keep the gate failure and the finding's severity separate: rate the finding on the rubric (a broken core flow is often High, but earn it on impact, reach, and recoverability), do not auto-assign High just because a gate failed.
• Instrumentation (apply the instrumentation skill's discipline). The product must be measurable. Verify the activation funnel is instrumented on one identity across client and server. An uninstrumented activation path is a ship-gate finding: where instrumentation is a required gate it caps the verdict at "ready to ship, risks noted" at best; where the policy makes it optional or waives it (a documented no-telemetry posture), that is recorded rather than capping.

Report each gate's status. Every report emits a shipGates entry per gate (met, at-risk, not-met, not-assessed, or not-applicable), with a note and any informing finding IDs. This is not optional: if the method calls mobile and instrumentation release gates, the artifact has to say where each one stands. A not-met required gate caps the verdict below safe-to-ship. On a static-only run, the device-level mobile checks and the live activation funnel are runtime work. Report what the code shows, mark the runtime portion not-assessed with the check to run, and never assert a gate you could not verify. A gate that does not apply (for example, mobile on a headless service) is not-applicable with a one-line reason.

Phase 3: Adversarial verification

Every Critical, High, and verdict-driving finding goes to a fresh subagent whose only job is to refute it against the code: Is it mitigated at another layer? Fixed by a later migration? Covered by an existing test? Is the exploit path actually reachable? A verdict-driving finding is any finding cited in verdict.blockingFindingIds, whatever its severity: if it moves the verdict, it gets verified, because the verdict is only as trustworthy as the findings it rests on. Refuted findings are dropped or downgraded, with the refutation recorded in verification.refutationNotes. Other findings are verified when their uncertainty and impact warrant it, and are otherwise marked not-verified. Hard rule: the final verdict may not ship while any finding in blockingFindingIds still has verification.status: not-verified (the repo's make validate enforces this mechanically).

Skeptic independence scales with impact. A fresh subagent is the default skeptic, but it shares the finder's model and its blind spots. For findings that would block the release or claim a Critical, make the check more independent when you can, and record how in verification.method: a different model as the skeptic, a deterministic reproduction script whose exit code settles the question, a runtime test, or a human. The more a finding moves the verdict, the less its verification should share assumptions with whatever produced it.

Phase 4: Report

Lead with posture, not problems. The report opens with the verdict, a one-line posture statement (how healthy is this, honestly), and a strengths list of the verified-safe controls the audit confirmed. Then the findings, in two groups: the risks to weigh (what the verdict rests on) first, then the improvements (safe today, robustness over time) second. A well-built app should read as well-built, with a short risk list and a longer betterment list, not as one undifferentiated pile of failures. blockingFindingIds may cite only risks: an improvement never drives the verdict, and make validate enforces it.

The verdict names its evidence base. Set verdict.assessedScope (static without --runtime, static-plus-runtime with it) and open the justification by naming that scope. static-plus-runtime means static analysis plus a browser pass, not that the runtime was verified end to end: real-device behavior, infrastructure state, capacity, backup and restore, and migration behavior stay not-assessed unless separately exercised. A static-only run that found nothing blocking gives the plain recommendation with its evidence base named beneath it ("static review, runtime not exercised"): the runtime checks it could not perform are listed in notAssessed, so the claim stops where the evidence stops without a jargon headline. Only the full invocation issues a whole-product release verdict at all; a single-dimension run reports scoped posture for that dimension, and --quick reports triage.

Keep the justification short. The reader is a time-poor senior engineer. Name the scope, the reason the verdict landed where it did, and the acceptance call, in a few short sentences. Do not spell out each blocking finding by ID in the prose: verdict.blockingFindingIds carries them, and the report surfaces each one by name next to the verdict, so a justification that re-lists the IDs is a wall of text repeating what is already shown.

Two anti-doom rules:

• Do not pad findings with non-issues. An observation of "we checked X and it is fine / intentional / by design" is a strength, not a finding. It goes in strengths (or is simply not reported), never as an informational finding. Findings are things to change.
• Keep the finding count honest via root-cause dedupe. One root cause is one finding with an instance list, not N rows.

Emit JSON to <project>/tmp/audit-YYYY-MM-DD.json conforming to the report contract, audit-report.schema.json, which is bundled next to this SKILL.md in the installed skill directory. If the bundled copy is somehow missing, fetch the canonical schema from the suite repo: https://raw.githubusercontent.com/horizon-foundry/foundry/main/schema/audit-report.schema.json.

Three schema fields worth stating here so the report does not fail validation late:

• kind is risk or improvement on every finding (see the Overview). make validate rejects a report whose blockingFindingIds cite anything but a risk.
• Finding IDs are dimension prefix + zero-padded ordinal, matching the schema's findingId pattern: SEC (security), CON (concurrency), REL (reliability), A11Y (accessibility), UI (ui), INF (infra), OPS (operability), TEST (testing-confidence), DATA (data-migration-safety), SHIP (release-safety), PERF (performance-capacity), e.g. SEC-01.
• category is one of the schema's enum: security, race-condition, reliability, accessibility, performance, visual-consistency, infra, operability, testing, data-safety, release-process.

Validate before finishing (run from the audited project; <skill-dir> is this skill's installed directory):

npx ajv-cli validate --spec=draft2020 -s <skill-dir>/audit-report.schema.json -d <report>.json

Chat output is the verdict plus top findings, not the whole report. The report in tmp/ is the deliverable; where it goes next (a team dashboard, a ticket, an artifact store) is the operator's call. Suite maintainers only: a report renders on the Foundry site by copying the JSON into the site repo's reports/<project-slug>-<YYYY-MM-DD>.json and deploying; it appears at the gated /reports route, visible to the report's owner.

Report ownership and local history

Reports are the artifact, so the skill that emits them also owns their attribution and history. The auth model, stated plainly: the skills never require an account. Sign-in exists for exactly one thing, viewing your published reports on the Foundry site, where each report is visible only to the email it belongs to.

• Connect an owner (production-audit init, optional, once). Read ~/.claude/foundry.json; if it already holds an owner email, report "connected as <email>" and stop (idempotent; switch only when asked). Otherwise ask for the email (one question), normalize to lowercase, write {"owner": "<email>"} to ~/.claude/foundry.json. From then on every report stamps meta.owner with it. Then point at the site's /unlock for the one-time magic link, but never block on it: sign-in is not part of init succeeding, and no owner file just means no stamp.
• List local history (production-audit list, no sign-in). Gather tmp/audit-YYYY-MM-DD.json files in the current project (plus, for suite maintainers, the site repo's published reports/). Print newest first: project, date, verdict, critical/high counts, file path. These files are readable right here without any account; the web rendering is the only surface behind the magic link.

Severity rubric

Severity is earned, never vibes. Judge each finding on impact x reach x exploitability, tempered by detectability and recoverability (a failure the team would see immediately and can roll back cleanly is less severe than the same failure that corrupts silently and permanently). The table gives the bars, with examples beyond auth so the catastrophic classes are named:

A High must name the business control bypassed or the concrete harm; a Critical must name the unacceptable outcome and why it is reachable today.

Evidence classes (confidence)

Fits no class: not reported. "Traced" and "reproduced" are deliberately separate classes: a static run's strongest honest claim is code-traced.

False-positive kill list

Do not report:

• Secrets stored in env vars (that is where secrets belong; report exposure, not storage)
• "Missing rate limiting" without checking every layer it could live at (middleware, handler, DB, platform)
• "Client validation insufficient" when the server validates
• A gap fixed by a later migration or covered by an existing test (check both before reporting)
• Generic CSRF on mutation paths already protected by same-site cookies or framework tokens

Rule: before reporting a missing control, search for it at every layer it could live.

Static/runtime boundary

Contrast on rendered output, focus visibility, screen-reader behavior, multi-tab races, network-failure UX, and real latency numbers cannot be confirmed from code. Without --runtime, these are reported as needs-verification with the exact runtime check, or listed in notAssessed. Never asserted. The same honesty applies to the verdict: its assessedScope says which evidence base it rests on, and a static run's strongest finding class is code-traced, never runtime-reproduced.

Hard rules

• The audit never modifies code. Read-only throughout, including subagents.
• Root-cause dedupe: one root cause = one finding with an instances list.
• The verdict cites blocking finding IDs. Quick wins state blast radius.
• Nothing speculative above informational. No em dashes in report text.

Common mistakes