Operations

Mainnet incident response

Playbooks for production mainnet x402 paid beta incidents, with evidence preservation and strict secret-handling boundaries.

Current production mode: Base mainnet USDC billing beta on primitive402.dev.
status.json is the live source Billing policy Mainnet readiness Mainnet monitoring

Do not ask for secrets

Never ask users for private keys, seed phrases, admin tokens, CDP secrets, raw payment headers, wallet recovery material, or screenshots that expose those values.

First response

  1. Check /health, /ready, and /status.json.
  2. Run pnpm monitor:mainnet --baseUrl https://primitive402.dev.
  3. Run discovery and unpaid x402 matrix checks if safe.
  4. Check private admin metrics only with a securely loaded ADMIN_METRICS_TOKEN.
  5. Preserve request id, route, response status, timestamp, tx hash, payer wallet if available, network, asset, and settlement status.

Common incidents

  • CDP/facilitator outage: verify unpaid metadata, check facilitator auth/settlement status, rotate CDP keys if exposed, and consider X402_ENABLED=false if paid traffic is unsafe.
  • Paid route errors: identify route-specific versus global failures, compare safe local route behavior, preserve settlement and request evidence, and prepare support review for settled non-2xx outcomes.
  • Duplicate payment or billing issue: ask for transaction hash, approximate timestamp, route, response status, and request id when available; never ask for wallet secrets.
  • Unsafe URL or SSRF false positive: preserve route, request id, target domain hash when available, response status, and SSRF error code.
  • Rate-limit spike: compare rateLimitedCalls with botProbeCalls and confirm whether real paid users are affected.

Rollback triggers

  • /ready remains critical.
  • /status.json or x402 metadata shows wrong network, asset, payTo, or resource URL.
  • CDP/facilitator auth, verification, or settlement fails repeatedly.
  • PAYMENT-RESPONSE is missing after paid 2xx responses.
  • Paid route errors rise after a deploy.
  • CDP keys, admin token, payment headers, private keys, or seed phrases are exposed.

Set X402_ENABLED=false when paid traffic must stop quickly because payment settlement, facilitator auth, route execution, or reconciliation is unsafe.

Repository docs

Markdown source: docs/ops/mainnet-incident-response.md.

Mainnet monitoring Mainnet readiness Paid x402 test guide Docs home