GitHub Actions

Run the auditor on every pull request. The minimum setup needs no API key.

The auditor ships as a GitHub Action backed by a container on GHCR. It audits the repository on each pull request, writes findings onto the diff as inline annotations, and optionally pushes verified drift to Slack or Telegram.

You do not have to write any of this by hand. init reads the repository and writes the whole file, including the branches to audit and every optional input already wired to a secret: see the Quickstart. This page is the reference for what it wrote, and for changing it afterwards.

Minimum setup

No API key, no secret. The deterministic layer alone scores F1 0.889 against the project’s own evaluation, so this is worth running before you decide anything about model spend.

.github/workflows/contract-audit.yml

name: Contract audit
on: [pull_request]

permissions:
  contents: read
  security-events: write   # required for the SARIF upload

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7

      - id: audit
        uses: samso9th/contract-auditor@v1
        continue-on-error: true          # report first, block later
        with:
          spec: openapi.json
          source-dir: internal
          strip-prefix: /api/v1
          fail-on: none

      - uses: github/codeql-action/upload-sarif@v4
        if: always()
        with:
          sarif_file: ${{ steps.audit.outputs.sarif }}

Findings now appear as annotations on the exact lines of the pull request diff.

A Markdown summary is written to the run’s job summary automatically, so there is something readable even before the SARIF upload is configured.

Getting the three paths right

Most first runs fail on one of these.

Input	What it means	Finding yours
spec	The OpenAPI document your integrators build against	Commonly openapi.json, docs/openapi.json, api/openapi.yaml, or wherever your generator writes it
source-dir	Where route registrations and handlers live	Per language, below
strip-prefix	The prefix present in code but absent from spec paths	If code registers /api/v1/payouts and the spec documents /payouts, this is /api/v1

The last two are what differ between projects:

Your project	source-dir	strip-prefix	Detected by
Go (net/http, gin, chi, echo)	internal, cmd, or the module root	/v1	go.mod, or any .go file
TypeScript (Express)	src	/api/v1	package.json, or any .ts file
Python (FastAPI, Flask)	the package your app lives in, commonly app	/api/v1	requirements.txt, pyproject.toml, setup.py, Pipfile
PHP (Laravel)	the project root, so both routes/api.php and app/Http/Controllers are readable	/api	artisan, composer.json, routes/api.php

language is detected from the markers in the last column and only needs setting in a polyglot repository, where the first marker found wins and may not be the one you meant.

If a first run reports that every route is missing from the spec, strip-prefix is wrong. Your spec’s servers[].url usually names it.

Telling the contract apart from the dashboard

Most codebases register two APIs in one place: the one integrators hold an API key for, and the one a dashboard or admin console talks to with a session token. Only the first was ever promised to anyone. Their paths do not separate them, which is why a list of path globs goes stale the week after it is written. What separates them is which guard they sit behind.

      - uses: samso9th/contract-auditor@v1
        with:
          spec: docs/openapi.json
          source-dir: src
          strip-prefix: /api/v1
          contract-middleware: authenticate

Only routes registered with that middleware are audited, on both sides. An operation that the spec documents and a non-contract guard protects will not then read as missing from the code, and a route added next week lands on the right side without anyone editing a list.

Both registration styles are read:

router.use(authenticate);                          // guards the whole router
router.get("/orders", listOrders);                 // in the contract
router.get("/stats", adminAuthenticate, getStats); // out of it

Route middleware is extracted for every supported language. Where a project supplies none, the run fails and says so instead of quietly excluding every route. Use exclude-paths there instead.

What the guard tells you beyond the filter

Restricting the audit to one guard also answers a question nobody asks in code review: what can this credential actually reach? The findings that come back are exactly the endpoints behind the guard that reads the credential your spec promises integrators. An endpoint you believed was dashboard-only appearing in that list is a finding about your architecture, not your documentation.

It happens for an ordinary reason. A guard written to accept an API key or a session token protects every route behind it with both, so a key issued to move money also reaches account management. Nothing contradicts it, because nothing wrote it down.

Two rules report the clear-cut cases directly, wherever route middleware is recorded:

Kind	Severity	Fires when
auth_guard_missing	critical	the spec declares security for the operation and the route carries no guard that reads any credential the spec names
auth_guard_undocumented	high	the route is guarded and the spec documents the operation as public, so an integrator following it is answered 401

Both are reported per file and only for files that guard something, so a project applying authentication once at application level gets silence rather than a page of false alarms.

router.use(mw) guards only the routes registered after it. A route above that line is open however protected the rest of the file looks, and that is the shape auth_guard_missing exists to catch.

Routes that are not part of the contract

Most codebases register endpoints no integrator was ever promised: a dashboard’s own session routes, internal health checks, an admin surface. Reporting them as drift is not wrong, just irrelevant, and a report that is mostly irrelevant stops being read.

      - uses: samso9th/contract-auditor@v1
        with:
          spec: docs/openapi.json
          source-dir: src
          strip-prefix: /api/v1
          exclude-paths: |
            /auth/*
            /internal/*

Patterns are matched against the path as your spec writes it, so after strip-prefix has been removed, and * crosses slashes: /auth/* covers /auth/me/password. A trailing /* covers the collection itself as well, so /auth/* also excludes /auth while leaving /authorize alone. Commas work as well as newlines, for a short list on one line.

An excluded path leaves the audit in both directions. It counts as neither missing from the spec nor missing from the code, because the spec is not wrong to stay quiet about a route you have declared internal.

Excluding every endpoint fails the run rather than reporting a clean audit. The usual way to do that by accident is writing the prefix back in: /api/v1/* matches nothing, because the prefix is already gone by the time patterns are applied.

Adding the judgment pass

The deterministic rules settle everything mechanical. Three kinds of drift need reading comprehension instead: a handler that requires a field the spec calls optional, a default that quietly changed, and a validation bound loosened below what is documented.

      - id: audit
        uses: samso9th/contract-auditor@v1
        with:
          spec: docs/openapi.json
          source-dir: src
          language: typescript
          strip-prefix: /api/v1
          api-key: ${{ secrets.OPENROUTER_API_KEY }}
          workers: "16"
          fail-on: critical

Add OPENROUTER_API_KEY under Settings → Secrets and variables → Actions. Any OpenAI-compatible endpoint works; set base-url to point elsewhere. API keys and secrets covers where each credential comes from, and why a pull request from a fork never receives one.

Every claim, from the rules and the model alike, is executed against your handler before it reaches the report. In the project’s own evaluation the model’s raw precision was 0.23; the gate refuted every false claim and the report came out at precision 1.0. What you end up reading has been checked against your own code. See the verification gate.

Cost across the full 16-case evaluation was $0.044 for 159 model calls. One repository audit is a fraction of that.

Handing findings to a coding agent

Every run produces a fix brief: one document covering every finding, what was observed, and the test that proved it. There are three ways to use it, and none of them need storage set up. The files are uploaded as a GitHub Actions artifact, which needs no bucket, no account and no credentials.

      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: contract-audit-brief
          path: ${{ steps.audit.outputs.brief-dir }}

Copy it

The pull request comment contains the whole brief in a collapsible block. Expand it, copy it, paste it into Cursor, Codex or Claude Code. Nothing to download.

Best when you want a fix started immediately and do not care about running the tests yourself.

Download the brief

<name>_brief.md from the run’s Artifacts section. The same document as a file, for attaching to a ticket or keeping alongside a branch.

Download the zip

<name>_brief.zip contains the brief and a tests/ directory holding every generated test as a runnable file.

This is the most useful option. You do not have to tell the agent what “fixed” means, because each test already asserts what the specification promises and currently fails. When it passes, that finding is resolved.

The brief opens by telling the agent the one judgement the tool deliberately leaves alone: whether the code drifted or the document went stale. Getting that backwards turns a documentation edit into a breaking change for everyone already integrated, so the brief asks the agent to decide explicitly and say why.

Inputs

Input	Default	Notes
spec	n/a	Required. Path to your OpenAPI document.
source-dir	.	Directory containing the API source.
language	auto	go, typescript, or detect.
strip-prefix	n/a	Prefix to strip so routes match spec paths.
contract-middleware	n/a	Only audit routes guarded by these middleware, one name per line. Also enables the two guard rules.
exclude-paths	n/a	Globs to leave out of the audit, one per line. Matched after strip-prefix is removed.
api-key	n/a	Omit for deterministic-only: no cost, no secret.
base-url	OpenRouter	Any OpenAI-compatible endpoint.
model	z-ai/glm-5.3-flash	Any model id your endpoint serves.
reasoning	n/a	off, low, medium, high. Slower and dearer; omit for the model’s own default.
workers	8	Concurrent model calls. Raise to shorten wall clock.
fail-on	high	critical, high, medium, low, none.
memory-url	n/a	Turns on self-improvement. Storage you own; nothing is kept in your repo.
memory-key-id	n/a	Access key id, or Cloudinary API key. Pass a secret.
memory-secret	n/a	Secret key, or Cloudinary API secret.
memory-token	n/a	Bearer token for an HTTP store, or an IPFS pinning JWT.
memory-region	n/a	Region for an S3-compatible store.
webhook-url	n/a	Full JSON report POSTed to a data sink.
webhook-secret	n/a	Sent as X-Auditor-Token.
slack-webhook-url	n/a	Formatted Slack message, verified findings only.
telegram-bot-token	n/a	With telegram-chat-id.
notify-min-severity	high	Lowest severity worth interrupting a human for.

Outputs: findings, critical, high, sarif, summary, brief, brief-zip, brief-dir.

Turning on self-improvement

The auditor can learn from its own mistakes across runs, but only if you tell it where to keep the record. There is no default location and no shared store. Leave memory-url out and it keeps no memory at all.

      - uses: samso9th/contract-auditor@v1
        with:
          spec: openapi.json
          api-key: ${{ secrets.OPENROUTER_API_KEY }}
          memory-url: s3://my-bucket/contract-auditor
          memory-key-id: ${{ secrets.MEMORY_KEY_ID }}
          memory-secret: ${{ secrets.MEMORY_SECRET }}
          memory-region: eu-west-1

Any S3-compatible bucket works. For Cloudflare R2, MinIO, Spaces or Backblaze, add the endpoint to the URL:

          memory-url: s3://my-bucket/contract-auditor?endpoint=https://<account>.r2.cloudflarestorage.com

Other backends, same shape:

Store	memory-url	Credentials
Any HTTPS endpoint that takes GET and PUT	https://host/ledger.jsonl	memory-token
Cloudinary raw storage	cloudinary://cloud-name/ledger	memory-key-id, memory-secret
IPFS via a Pinata-compatible pinner	ipfs://ledger	memory-token

Two things change when you set this. Each run reads the ledger before auditing and appends to it afterwards, so the store has to be reachable from the runner. And the verification gate starts running against your repository, because a claim with no verdict teaches nothing: a temporary test is written next to your code, executed, and deleted. A claim its test disproves is dropped from the report. One whose test cannot be built is kept and marked, never silently lost.

Memory only ever adjusts priors. It changes which past mistakes the model is shown and how findings are ranked, and it can never suppress a finding: if the generated test still fails, the finding ships. About one endpoint in twenty is audited with memory disabled on purpose, so a wrong assumption keeps getting tested instead of settling in permanently.

Nothing is written into your checkout, and no ledger ships inside the action’s image: the build deletes any that reaches it and fails if one remains.

Alerting a human

Two different jobs, deliberately separate.

Slack or Telegram

Formatted, verified-only, and silent on a clean run.

    with:
      slack-webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
      notify-min-severity: critical

See notifications for the formatting and filtering rules.

Data sink

Raw JSON, every finding, for Postgres or a warehouse.

    with:
      webhook-url: ${{ secrets.AUDITOR_WEBHOOK_URL }}
      webhook-secret: ${{ secrets.AUDITOR_WEBHOOK_SECRET }}

file and line are structured fields, so a sink never parses prose to place a finding.

{
  "tool": "contract-auditor",
  "repository": "owner/repo",
  "sha": "abc123",
  "run_url": "https://github.com/owner/repo/actions/runs/123",
  "summary": { "total": 3, "by_severity": { "critical": 1, "high": 2 } },
  "findings": [
    {
      "path": "/payouts", "method": "post",
      "kind": "response_type_mismatch", "detail": "amount",
      "severity": "critical",
      "file": "handlers/types.go", "line": 27,
      "evidence": "PayoutResponse.Amount is float64 (JSON number); spec declares string",
      "verdict": "confirmed"
    }
  ]
}

PR comment

    - uses: marocchino/sticky-pull-request-comment@v2
      if: github.event_name == 'pull_request'
      with:
        path: ${{ steps.audit.outputs.summary }}

Needs pull-requests: write in the job’s permissions.

Rolling it out to a team

1. Report only

   fail-on: none with continue-on-error: true. Nothing blocks.

2. Put findings on the diff

   Add the SARIF upload, so they appear where people already look.

3. Clear the backlog

   Fix what is real. Record deliberate differences in auditor/memory/allowlist.json. Each entry needs a reason and a date, so the allowlist does not become a place where findings go to be forgotten.

4. Tighten

   fail-on: critical, then high once the signal is trusted.

Do not block merges on day one. That is the quickest way to get the tool removed in week two.

Scheduled audits

Pull request runs catch drift as it is introduced. A weekly run catches the drift that was already there before you adopted the tool, and drift in the spec itself.

on:
  schedule:
    - cron: "0 9 * * 1"    # Mondays, 09:00 UTC
  pull_request:

Troubleshooting

Every route reported as missing from the spec

strip-prefix is wrong. If code registers /api/v1/payouts and the spec documents /payouts, set strip-prefix: /api/v1.

The report is mostly routes that were never meant to be public

Dashboard session routes, health checks and admin surfaces are registered in code and deliberately absent from an integrator-facing spec, so every audit reports them. Leave them out with exclude-paths, one glob per line, matched against the path as the spec writes it.

manifest unknown when pulling the image

The package is private, or was never linked to its repository. Check with gh api /user/packages/container/contract-auditor --jq '.visibility'. Publishing from a public repository normally yields a public package automatically; if it reports private, set Package settings → visibility → Public.

denied: permission_denied when publishing

Settings → Actions → General → Workflow permissions → Read and write. The workflow requesting packages: write is not enough on its own; the repository default caps what the token can be granted.

Findings show verdict: error

The gate could not compile or run its generated test, usually a Go toolchain mismatch: the image pins Go 1.24. These findings are kept and flagged, never silently dropped, because a broken toolchain must never look like a clean bill of health. Deterministic findings are unaffected; they execute nothing.

The run reported unread_endpoints

The model returned something unparseable for those endpoints even after a retry, so the judgment pass did not read them. That gets reported rather than hidden, because an endpoint nobody read is not the same as an endpoint with no drift.

TypeScript findings show unsupported

Route extraction works for TypeScript; the verification gate does not yet. Unverified claims are not let through, so they are reported as unsupported.