Secret Rotation Checklist

Checklist for rotating API keys, DB credentials, and HMAC secrets

Secret Rotation Checklist

Operator-facing rotation procedures for every secret QTG holds. Apply when:

• a secret may have been exposed (terminal scrollback, screen share, leaked log, lost device)
• a key reaches its scheduled rotation window
• an operator with key access offboards
• post-incident hardening calls for cycling everything in a class

Each section is a self-contained subprocedure. Run them independently; nothing in this document assumes you are rotating the whole inventory in one window.

Pro boundary. AWS KMS signer lifecycle administration and its operator runbook are Pro. This Free checklist covers generic credential exposure and incident handling only; it does not link to or reproduce the Pro procedure.

---

Principles

1. Issue first, retire after. Never delete an active credential before confirming the replacement works end-to-end. The window where both are accepted is your safety net — keep it explicit, never implicit.
2. Rotation is a runtime change. Every rotation requires a config reload or process restart somewhere. Plan the restart window before you start.
3. Health check is part of the procedure, not optional. A rotation is not complete until the new credential has signed a real call (HMAC verify, exchange order list, callback dispatch, signer health). If you cannot prove the new credential works, you have not rotated — you have created a broken state.
4. Audit trail goes in the incident log. When rotating due to suspected exposure, write an incident entry — <UTC>-secret-rotation-<scope> — into whatever incident log your deployment keeps, alongside the rotation. The log is the only record after the secret values themselves are gone.
5. Local secrets never enter git. If a secret appears in a commit (even a deleted line in a later commit), it is permanently exposed — see § Git history secret audit for triage.

---

Secret inventory

Class	Env var(s)	Storage	Rotation trigger	Restart needed
Callback HMAC v3	MG_CALLBACK_HMAC_SECRET	.env + callback receiver	Exposure suspected, scheduled annual	Yes (callback worker + receiver)
Exchange API keys	MG_{UPBIT,BINANCE,BYBIT,OKX}_{ACCESS,SECRET}_KEY	.env + exchange console	Operator offboard, exposure, scheduled annual	Yes
ApiClientKey HMAC secrets	DB row api_client_keys.hmac_secret	App DB (plaintext)	Operator offboard, dashboard token exposure	No (live)
Local signer key — plain	MG_LOCAL_SIGNER_PRIVATE_KEY_HEX (or named ref)	.env	Any local-disk exposure suspicion	Yes
Local signer key — op	1Password vault item	1Password	Any 1Password account compromise	Yes (signer reloads on next sign)
AWS KMS EVM key	KMS alias alias/qtg/...	AWS KMS	Generic exposure incident handling	n/a
Remote signer auth token	MG_REMOTE_SIGNER_AUTH_TOKEN	.env + remote signer service	Token exposure or remote-signer offboard	Yes
CCIP sidecar auth token	MG_CCIP_SIDECAR_AUTH_TOKEN	.env + sidecar deployment	Sidecar redeployment, exposure	Yes (QTG + sidecar)
Dashboard write token (legacy)	MG_DASHBOARD_WRITE_TOKEN	.env	Should not exist — see HMAC migration runbook	n/a — the variable must be absent; an empty value also blocks boot

---

Callback HMAC v3 secret

MG_CALLBACK_HMAC_SECRET signs every callback outbox dispatch QTG sends to the external receiver. The receiver’s matching secret must rotate atomically.

1. Pre-flight. Confirm the callback receiver is rotation-aware (accepts an N+1 secret in parallel with N for a window). If not, you will need a brief downtime window — coordinate with the receiver owner first.
2. Generate. python3 -c "import secrets; print(secrets.token_hex(32))" — produces a 64-char hex string. Do not reuse old values.
3. Apply to the receiver first. Add the new secret as the secondary accepting key on the receiver side. Do NOT remove the old key yet.
4. Apply to QTG. Edit .env, set MG_CALLBACK_HMAC_SECRET to the new value. Restart QTG (docker compose -f infra/docker-compose.yml restart qtg-mainnet — or qtg-testnet — or the equivalent uvicorn process).
5. Validate. Trigger any callback-producing path (a movement state change is the cheapest — approve any pending movement, or retry a callback that’s already in the outbox). Confirm the receiver returns 200 with the new key.
6. Retire the old key on the receiver. Only after step 5 passes.
7. Document. If rotating due to exposure: incident log, plus a changelog note if it is release-relevant.

Callback contract reference: docs/callback-verification-contract.md. The v3 scheme includes a nonce (replay defense) — see src/qtg/callback_auth.py for the canonical signing surface.

---

Exchange API keys

Per-exchange procedure. Each exchange has its own console UX; the QTG side of the rotation is identical across all of them.

Generic procedure (Upbit / Binance / Bybit / OKX)

1. Generate at the exchange. Create a new API key with the same scope as the existing one (typically: read, withdraw, deposit-address-query, and travel-rule beneficiary if applicable). Note any IP allowlist requirement — re-attach the same allowlist before saving.
2. Whitelist verification. For exchanges that require withdrawal-address whitelist on the key (Bybit V5 universal book, Upbit), confirm that destination addresses are still attached to the new key. If the key inherits the account-level allowlist, no extra step. If per-key, re-add every required destination before retiring the old key.
3. Apply to .env. Replace the matching access/secret pair. Two-line edit: MG_<EX>_ACCESS_KEY=... and MG_<EX>_SECRET_KEY=....
4. Restart QTG. Adapters cache the credentials at construction; a runtime reload is required.
5. Validate (read-only first). Run a non-mutating call against the venue.
   qtg venue probe upbit --asset USDT --network tron
   
   Exit 0 means every read-only probe passed; exit 2 means missing credentials or an unknown venue. Same form for binance / bybit / okx / coinbase. Do not reach for run_cex_live_drill here — it executes a live withdrawal.
6. Validate (tiny mutation, optional but recommended). If rotation is due to suspected exposure, also confirm a withdrawal preflight passes (e.g. submit_withdrawal against a known-whitelisted destination with dry_run semantics where supported). Do not broadcast a real withdrawal just to prove the key works — read calls + preflight are sufficient.
7. Retire the old key. Disable, then delete, in the exchange console. The disable-before-delete gap exists so you can quickly re-enable if step 5 or 6 surfaces a problem.

Exchange-specific notes

• Upbit: secret returns to you only once at creation. Capture into 1Password or the equivalent immediately. IP allowlist is mandatory — match the host.
• Bybit V5: travel-rule beneficiary plumbing (MG_BYBIT_BENEFICIARY_NAME) is not a secret but is required for KR-destination withdrawals; it does not rotate. The vasp/list cache is per-ClassVar and survives key rotation.

---

ApiClientKey HMAC secrets (admin_cli / dashboard_writer / agent / operator)

These keys are stored in api_client_keys.hmac_secret as plaintext text in the application DB. Rotation is live-safe (no restart needed) because the HMAC middleware reads from DB on every request.

Issue a new key alongside the old one

1. Bootstrap a replacement key with a new key_id but the same role and client_name:
   # Dashboard writer:
   MG_CLI_BOOTSTRAP_ALLOWED=true \
   MG_DASHBOARD_WRITER_EXPECTED_KEY_COUNT=<current+1> \
   uv run python -m qtg.interfaces.tools.bootstrap_dashboard_writer_key \
     --key-id <operator>-<YYYY-MM-DD> \
     --database-url "$MG_DATABASE_URL"
   
   # Admin CLI:
   MG_CLI_BOOTSTRAP_ALLOWED=true \
   MG_CLI_AUTH_EXPECTED_KEY_COUNT=<current+1> \
   uv run python -m qtg.interfaces.tools.bootstrap_admin_cli_key \
     --key-id admin-<operator>-<YYYY-MM-DD> \
     --database-url "$MG_DATABASE_URL"
   
2. The CLI prints key_id + hmac_secret once. Capture both into 1Password.
3. Validate with the new key. Run any HMAC-gated request (e.g., a GET /v3/movements/... for admin_cli, or unlock writes in the dashboard with the new key). Confirm dashboard_writer_auth_ok (or the equivalent admin_cli_auth_ok) log line emits with the new key_id.
4. Revoke the old key. Use the committed CLI, not raw SQL:
   uv run python -m qtg.interfaces.tools.revoke_auth_key \
     --key-id <old-key-id> \
     --database-url "$MG_DATABASE_URL"
   
   It exits 0 only once the row reads status = 'revoked'. The HMAC middleware filters on status = 'active', so the key stops working on the very next request — no restart needed.
5. Decrement the expected-count gate (if it was bumped). Update .env: MG_DASHBOARD_WRITER_EXPECTED_KEY_COUNT / MG_CLI_AUTH_EXPECTED_KEY_COUNT back to the live count. The boot gate fail-closes on mismatch at next restart, so leaving it inflated will surface as a startup error on the next restart for any reason.

Operator offboarding

When an operator leaves, revoke every key_id they owned:

for k in <operator>-... <operator>-...; do
  uv run python -m qtg.interfaces.tools.revoke_auth_key \
    --key-id "$k" --database-url "$MG_DATABASE_URL"
done

Then decrement the expected-count gate (see above) on the next planned restart.

When DB read is suspected exposed

Plaintext HMAC secret storage is an accepted residual risk (tracked as docs/reference/dashboard-auth.md known limitation #7). If a DB backup is suspected leaked, every active HMAC key is exposed — issue replacements for every api_client_keys row, then revoke all old keys. Same procedure as above, applied N times.

---

Local signer key material (Tier-3)

Backend plain and op rotate differently because the key lives in fundamentally different storage.

plain backend (MG_LOCAL_SIGNER_KEY_SOURCE=plain)

1. Generate the new key off-device (hardware wallet, air-gapped machine, or eth_keys in a one-shot script you immediately discard).
2. Capture the new signer address. You will need to register this address with whatever on-chain allowlist or whitelist the signer participates in (CCTP burn allowlist, Stargate trusted-pool destination, etc.) before the rotation, not after.
3. Edit .env: replace MG_LOCAL_SIGNER_PRIVATE_KEY_HEX (or whichever env the MG_LOCAL_SIGNER_KEY_REF points to).
4. Restart QTG. The signer rebootstraps and logs the new address; confirm it matches step 2.
5. Validate: any signer health check or preflight that exercises EVM signing. uv run python -m qtg.interfaces.tools.cctp_live_preflight is the most commonly exercised path.
6. Wipe the old key — overwrite the file or env var with zeros and unset it. On macOS/Linux: unset MG_LOCAL_SIGNER_PRIVATE_KEY_HEX then remove the line from .env.

plain is dev-only. If you reach this section in a production rollout, migrate to op or AWS KMS instead. See docs/reference/signers/local-private-key-signer.md § Migration.

op backend (MG_LOCAL_SIGNER_KEY_SOURCE=op)

1. Generate the new key in 1Password (new item under the same vault, distinct ref). Note the new op://Vault/Item/private-key reference.
2. Confirm the new signer address (decrypt and derive locally if you have a secure environment; otherwise let step 4 surface the address through QTG’s signer health log).
3. Update on-chain allowlists for the new address (see plain step 2).
4. Edit .env: update MG_LOCAL_SIGNER_KEY_REF to the new op://... URI. Restart QTG.
5. Validate: signer health log line shows the new address; run a preflight.
6. Wipe: delete the old 1Password item. The on-disk bytearray was already wiped per-sign (refetch_per_sign=true default) — no host cleanup needed.

If you are pinning the op binary via MG_LOCAL_SIGNER_OP_PATH, the binary path is not a secret and does not rotate with the key.

---

CCIP sidecar auth token

MG_CCIP_SIDECAR_AUTH_TOKEN is shared between QTG and the CCIP sidecar deployment. The sidecar verifies the token on every refresh / send.

1. Generate (same secrets.token_hex(32) pattern).
2. Update the sidecar deployment first (e.g. its .env or k8s secret) so it accepts the new token. If the sidecar supports dual-accept, add the new value as secondary; otherwise plan a brief downtime window.
3. Update .env on the QTG side; restart QTG.
4. Validate: uv run python -m qtg.interfaces.tools.refresh_ccip_registry (or whichever sidecar-hitting CLI is available). Confirm 200.
5. Retire the old token on the sidecar.

---

Remote signer auth token

MG_REMOTE_SIGNER_AUTH_TOKEN is the bearer token QTG presents to a RemoteSignerProxy upstream. The upstream’s view of the token must rotate atomically.

Same shape as the CCIP sidecar procedure. The upstream owner is the source of truth — coordinate the window first.

---

Post-rotation validation

After any single rotation, before declaring the procedure complete:

• dashboard_writer_auth_ok / admin_cli_auth_ok log line emits with the new key_id (for ApiClientKey rotations)
• Affected adapter / signer health check passes (signer bootstrap log line, executor health() if exposed)
• At least one read-only call against the rotated surface returns 2xx
• Old credential is disabled (not just retired in memory)
• If rotation was triggered by suspected exposure: incident log committed

---

Git history secret audit

A secret that ever appeared in a commit is permanently exposed via git log -p, even if the file was later deleted or the value overwritten. There is no in-place fix — only rotate and accept the loss.

Triage when you find one

1. Rotate immediately. Run the matching subprocedure above for that secret class. Do this before any source-tree cleanup.
2. Decide on history rewrite.
   • Pre-public repo (this Pro repo, never published): a history rewrite (git filter-repo --replace-text) is technically possible but rarely worth it for a private 1-operator repo. Document the exposure window as an incident entry — <UTC>-git-history-secret-<scope> — and move on.
   • Public/OSS repo (github.com/jephalabs/quant-transfer-guard): the OSS publish is squash-only (scripts/publish_oss.sh), so OSS history never carries Pro history. Confirm by git log --all --source -p -S '<secret-fragment>' against the OSS clone. If the leak made it into the OSS squash baseline, force-publish a new baseline. Pre-1.0 immutable-tag policy still applies to new tags; the affected tag stays as-is for clone provenance.
3. Re-audit. Run a repo-wide scan after rotation to catch sibling leaks:
   gitleaks detect --source . --verbose --redact
   # or, focused:
   git grep -nE '(SECRET|PRIVATE_KEY|ACCESS_KEY|HMAC)[A-Z_]*=[a-zA-Z0-9_+/-]+'
   
4. Lock the door. If the leak vector is a config file ever-tracked, add it to .gitignore (most are already there — .env, .env.local, .env.*.local). If the leak was a paste-in-source mistake, consider a pre-commit gitleaks hook.

Known safe items in git

• MG_CALLBACK_HMAC_SECRET=CHANGE_ME placeholder in .env.example is not a leak — the literal value CHANGE_ME is documented in tests/qtg/test_settings_load.py as the sentinel that the boot gate rejects.
• TEST_PRIVATE_KEY_HEX = "0x" + "a" * 64 in tests/qtg/test_local_private_key_signer.py is a throwaway test key with zero on-chain assets ever attached. Documented in the test file header.

---

See also

• docs/reference/dashboard-auth.md — dashboard HMAC scheme + known limitations (incl. plaintext secret storage)
• docs/runbooks/dashboard-auth-v1-to-hmac.md — historical migration from the legacy bundle-baked token (deprecated)
• docs/callback-verification-contract.md — callback v3 signing contract
• docs/reference/signers/local-private-key-signer.md — Tier-3 signer internals + migration to AWS KMS