> ## Documentation Index > Fetch the complete documentation index at: https://jephalabs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Secret Rotation Checklist > Checklist for rotating API keys, DB credentials, and HMAC secrets # Secret Rotation Checklist Operator-facing rotation procedures for every secret QTG holds. Apply when: * a secret may have been exposed (terminal scrollback, screen share, leaked log, lost device) * a key reaches its scheduled rotation window * an operator with key access offboards * post-incident hardening calls for cycling everything in a class Each section is a self-contained subprocedure. Run them independently; nothing in this document assumes you are rotating the whole inventory in one window. > **Pro boundary.** AWS KMS signer lifecycle administration and its operator > runbook are Pro. This Free checklist covers generic credential exposure and > incident handling only; it does not link to or reproduce the Pro procedure. *** ## Principles 1. **Issue first, retire after.** Never delete an active credential before confirming the replacement works end-to-end. The window where both are accepted is your safety net — keep it explicit, never implicit. 2. **Rotation is a runtime change.** Every rotation requires a config reload or process restart somewhere. Plan the restart window before you start. 3. **Health check is part of the procedure, not optional.** A rotation is not complete until the new credential has signed a real call (HMAC verify, exchange order list, callback dispatch, signer health). If you cannot prove the new credential works, you have not rotated — you have created a broken state. 4. **Audit trail goes in the incident log.** When rotating due to suspected exposure, write an incident entry — `-secret-rotation-` — into whatever incident log your deployment keeps, alongside the rotation. The log is the only record after the secret values themselves are gone. 5. **Local secrets never enter `git`.** If a secret appears in a commit (even a deleted line in a later commit), it is permanently exposed — see [§ Git history secret audit](#git-history-secret-audit) for triage. *** ## Secret inventory | Class | Env var(s) | Storage | Rotation trigger | Restart needed | | ------------------------------ | -------------------------------------------------- | ------------------------------ | ------------------------------------------------- | ------------------------------------------------------------------ | | Callback HMAC v3 | `MG_CALLBACK_HMAC_SECRET` | `.env` + callback receiver | Exposure suspected, scheduled annual | Yes (callback worker + receiver) | | Exchange API keys | `MG_{UPBIT,BINANCE,BYBIT,OKX}_{ACCESS,SECRET}_KEY` | `.env` + exchange console | Operator offboard, exposure, scheduled annual | Yes | | ApiClientKey HMAC secrets | DB row `api_client_keys.hmac_secret` | App DB (plaintext) | Operator offboard, dashboard token exposure | No (live) | | Local signer key — `plain` | `MG_LOCAL_SIGNER_PRIVATE_KEY_HEX` (or named ref) | `.env` | Any local-disk exposure suspicion | Yes | | Local signer key — `op` | 1Password vault item | 1Password | Any 1Password account compromise | Yes (signer reloads on next sign) | | AWS KMS EVM key | KMS alias `alias/qtg/...` | AWS KMS | Generic exposure incident handling | n/a | | Remote signer auth token | `MG_REMOTE_SIGNER_AUTH_TOKEN` | `.env` + remote signer service | Token exposure or remote-signer offboard | Yes | | CCIP sidecar auth token | `MG_CCIP_SIDECAR_AUTH_TOKEN` | `.env` + sidecar deployment | Sidecar redeployment, exposure | Yes (QTG + sidecar) | | Dashboard write token (legacy) | `MG_DASHBOARD_WRITE_TOKEN` | `.env` | **Should not exist** — see HMAC migration runbook | n/a — the variable must be absent; an empty value also blocks boot | *** ## Callback HMAC v3 secret `MG_CALLBACK_HMAC_SECRET` signs every callback outbox dispatch QTG sends to the external receiver. The receiver's matching secret must rotate atomically. 1. **Pre-flight.** Confirm the callback receiver is rotation-aware (accepts an N+1 secret in parallel with N for a window). If not, you will need a brief downtime window — coordinate with the receiver owner first. 2. **Generate.** `python3 -c "import secrets; print(secrets.token_hex(32))"` — produces a 64-char hex string. Do **not** reuse old values. 3. **Apply to the receiver first.** Add the new secret as the secondary accepting key on the receiver side. Do NOT remove the old key yet. 4. **Apply to QTG.** Edit `.env`, set `MG_CALLBACK_HMAC_SECRET` to the new value. Restart QTG (`docker compose -f infra/docker-compose.yml restart qtg-mainnet` — or `qtg-testnet` — or the equivalent `uvicorn` process). 5. **Validate.** Trigger any callback-producing path (a movement state change is the cheapest — approve any pending movement, or retry a callback that's already in the outbox). Confirm the receiver returns 200 with the new key. 6. **Retire the old key on the receiver.** Only after step 5 passes. 7. **Document.** If rotating due to exposure: incident log, plus a changelog note if it is release-relevant. **Callback contract reference**: `docs/callback-verification-contract.md`. The v3 scheme includes a nonce (replay defense) — see `src/qtg/callback_auth.py` for the canonical signing surface. *** ## Exchange API keys Per-exchange procedure. **Each exchange has its own console UX**; the QTG side of the rotation is identical across all of them. ### Generic procedure (Upbit / Binance / Bybit / OKX) 1. **Generate at the exchange.** Create a new API key with the same scope as the existing one (typically: read, withdraw, deposit-address-query, and travel-rule beneficiary if applicable). Note any IP allowlist requirement — re-attach the same allowlist before saving. 2. **Whitelist verification.** For exchanges that require withdrawal-address whitelist on the **key** (Bybit V5 universal book, Upbit), confirm that destination addresses are still attached to the new key. If the key inherits the account-level allowlist, no extra step. If per-key, re-add every required destination before retiring the old key. 3. **Apply to `.env`.** Replace the matching access/secret pair. Two-line edit: `MG__ACCESS_KEY=...` and `MG__SECRET_KEY=...`. 4. **Restart QTG.** Adapters cache the credentials at construction; a runtime reload is required. 5. **Validate (read-only first).** Run a non-mutating call against the venue. ```bash theme={null} qtg venue probe upbit --asset USDT --network tron ``` Exit 0 means every read-only probe passed; exit 2 means missing credentials or an unknown venue. Same form for `binance` / `bybit` / `okx` / `coinbase`. Do **not** reach for `run_cex_live_drill` here — it executes a live withdrawal. 6. **Validate (tiny mutation, optional but recommended).** If rotation is due to suspected exposure, also confirm a withdrawal preflight passes (e.g. `submit_withdrawal` against a known-whitelisted destination with `dry_run` semantics where supported). Do **not** broadcast a real withdrawal just to prove the key works — read calls + preflight are sufficient. 7. **Retire the old key.** Disable, then delete, in the exchange console. The disable-before-delete gap exists so you can quickly re-enable if step 5 or 6 surfaces a problem. ### Exchange-specific notes * **Upbit**: secret returns to you only once at creation. Capture into 1Password or the equivalent immediately. IP allowlist is mandatory — match the host. * **Bybit V5**: travel-rule beneficiary plumbing (`MG_BYBIT_BENEFICIARY_NAME`) is **not** a secret but is required for KR-destination withdrawals; it does not rotate. The `vasp/list` cache is per-`ClassVar` and survives key rotation. *** ## ApiClientKey HMAC secrets (admin\_cli / dashboard\_writer / agent / operator) These keys are stored in `api_client_keys.hmac_secret` as plaintext text in the application DB. Rotation is **live-safe** (no restart needed) because the HMAC middleware reads from DB on every request. ### Issue a new key alongside the old one 1. Bootstrap a replacement key with a new `key_id` but the **same** role and client\_name: ```bash theme={null} # Dashboard writer: MG_CLI_BOOTSTRAP_ALLOWED=true \ MG_DASHBOARD_WRITER_EXPECTED_KEY_COUNT= \ uv run python -m qtg.interfaces.tools.bootstrap_dashboard_writer_key \ --key-id - \ --database-url "$MG_DATABASE_URL" # Admin CLI: MG_CLI_BOOTSTRAP_ALLOWED=true \ MG_CLI_AUTH_EXPECTED_KEY_COUNT= \ uv run python -m qtg.interfaces.tools.bootstrap_admin_cli_key \ --key-id admin-- \ --database-url "$MG_DATABASE_URL" ``` 2. The CLI prints `key_id` + `hmac_secret` once. Capture both into 1Password. 3. **Validate with the new key.** Run any HMAC-gated request (e.g., a `GET /v3/movements/...` for admin\_cli, or unlock writes in the dashboard with the new key). Confirm `dashboard_writer_auth_ok` (or the equivalent `admin_cli_auth_ok`) log line emits with the new `key_id`. 4. **Revoke the old key.** Use the committed CLI, not raw SQL: ```bash theme={null} uv run python -m qtg.interfaces.tools.revoke_auth_key \ --key-id \ --database-url "$MG_DATABASE_URL" ``` It exits 0 only once the row reads `status = 'revoked'`. The HMAC middleware filters on `status = 'active'`, so the key stops working on the very next request — no restart needed. 5. **Decrement the expected-count gate (if it was bumped).** Update `.env`: `MG_DASHBOARD_WRITER_EXPECTED_KEY_COUNT` / `MG_CLI_AUTH_EXPECTED_KEY_COUNT` back to the live count. The boot gate fail-closes on mismatch at next restart, so leaving it inflated will surface as a startup error on the next restart for any reason. ### Operator offboarding When an operator leaves, revoke every `key_id` they owned: ```bash theme={null} for k in -... -...; do uv run python -m qtg.interfaces.tools.revoke_auth_key \ --key-id "$k" --database-url "$MG_DATABASE_URL" done ``` Then decrement the expected-count gate (see above) on the next planned restart. ### When DB read is suspected exposed Plaintext HMAC secret storage is an accepted residual risk (tracked as `docs/reference/dashboard-auth.md` known limitation #7). If a DB backup is suspected leaked, **every active HMAC key is exposed** — issue replacements for every `api_client_keys` row, then revoke all old keys. Same procedure as above, applied N times. *** ## Local signer key material (Tier-3) Backend `plain` and `op` rotate differently because the key lives in fundamentally different storage. ### `plain` backend (`MG_LOCAL_SIGNER_KEY_SOURCE=plain`) 1. Generate the new key off-device (hardware wallet, air-gapped machine, or `eth_keys` in a one-shot script you immediately discard). 2. Capture the **new** signer address. You will need to register this address with whatever on-chain allowlist or whitelist the signer participates in (CCTP burn allowlist, Stargate trusted-pool destination, etc.) **before** the rotation, not after. 3. Edit `.env`: replace `MG_LOCAL_SIGNER_PRIVATE_KEY_HEX` (or whichever env the `MG_LOCAL_SIGNER_KEY_REF` points to). 4. Restart QTG. The signer rebootstraps and logs the new address; confirm it matches step 2. 5. Validate: any signer health check or preflight that exercises EVM signing. `uv run python -m qtg.interfaces.tools.cctp_live_preflight` is the most commonly exercised path. 6. Wipe the old key — overwrite the file or env var with zeros and unset it. On macOS/Linux: `unset MG_LOCAL_SIGNER_PRIVATE_KEY_HEX` then remove the line from `.env`. > **`plain` is dev-only.** If you reach this section in a production rollout, > migrate to `op` or AWS KMS instead. See > `docs/reference/signers/local-private-key-signer.md` § Migration. ### `op` backend (`MG_LOCAL_SIGNER_KEY_SOURCE=op`) 1. Generate the new key in 1Password (new item under the same vault, distinct ref). Note the new `op://Vault/Item/private-key` reference. 2. Confirm the new signer address (decrypt and derive locally if you have a secure environment; otherwise let step 4 surface the address through QTG's signer health log). 3. Update on-chain allowlists for the new address (see `plain` step 2). 4. Edit `.env`: update `MG_LOCAL_SIGNER_KEY_REF` to the new `op://...` URI. Restart QTG. 5. Validate: signer health log line shows the new address; run a preflight. 6. **Wipe**: delete the old 1Password item. The on-disk `bytearray` was already wiped per-sign (`refetch_per_sign=true` default) — no host cleanup needed. If you are pinning the `op` binary via `MG_LOCAL_SIGNER_OP_PATH`, the binary path is not a secret and does not rotate with the key. *** ## CCIP sidecar auth token `MG_CCIP_SIDECAR_AUTH_TOKEN` is shared between QTG and the CCIP sidecar deployment. The sidecar verifies the token on every refresh / send. 1. Generate (same `secrets.token_hex(32)` pattern). 2. Update the sidecar deployment first (e.g. its `.env` or k8s secret) so it accepts the new token. If the sidecar supports dual-accept, add the new value as secondary; otherwise plan a brief downtime window. 3. Update `.env` on the QTG side; restart QTG. 4. Validate: `uv run python -m qtg.interfaces.tools.refresh_ccip_registry` (or whichever sidecar-hitting CLI is available). Confirm 200. 5. Retire the old token on the sidecar. *** ## Remote signer auth token `MG_REMOTE_SIGNER_AUTH_TOKEN` is the bearer token QTG presents to a `RemoteSignerProxy` upstream. The upstream's view of the token must rotate atomically. Same shape as the CCIP sidecar procedure. The upstream owner is the source of truth — coordinate the window first. *** ## Post-rotation validation After any single rotation, before declaring the procedure complete: * [ ] `dashboard_writer_auth_ok` / `admin_cli_auth_ok` log line emits with the **new** `key_id` (for ApiClientKey rotations) * [ ] Affected adapter / signer health check passes (signer bootstrap log line, executor `health()` if exposed) * [ ] At least one read-only call against the rotated surface returns 2xx * [ ] Old credential is **disabled** (not just retired in memory) * [ ] If rotation was triggered by suspected exposure: incident log committed *** ## Git history secret audit A secret that ever appeared in a commit is permanently exposed via `git log -p`, even if the file was later deleted or the value overwritten. There is no in-place fix — only **rotate and accept the loss**. ### Triage when you find one 1. **Rotate immediately.** Run the matching subprocedure above for that secret class. Do this **before** any source-tree cleanup. 2. **Decide on history rewrite.** * **Pre-public repo** (this Pro repo, never published): a history rewrite (`git filter-repo --replace-text`) is technically possible but rarely worth it for a private 1-operator repo. Document the exposure window as an incident entry — `-git-history-secret-` — and move on. * **Public/OSS repo** (`github.com/jephalabs/quant-transfer-guard`): the OSS publish is squash-only (`scripts/publish_oss.sh`), so OSS history never carries Pro history. Confirm by `git log --all --source -p -S ''` against the OSS clone. If the leak made it into the OSS squash baseline, force-publish a new baseline. Pre-1.0 immutable-tag policy still applies to **new** tags; the affected tag stays as-is for clone provenance. 3. **Re-audit.** Run a repo-wide scan after rotation to catch sibling leaks: ```bash theme={null} gitleaks detect --source . --verbose --redact # or, focused: git grep -nE '(SECRET|PRIVATE_KEY|ACCESS_KEY|HMAC)[A-Z_]*=[a-zA-Z0-9_+/-]+' ``` 4. **Lock the door.** If the leak vector is a config file ever-tracked, add it to `.gitignore` (most are already there — `.env`, `.env.local`, `.env.*.local`). If the leak was a paste-in-source mistake, consider a pre-commit `gitleaks` hook. ### Known safe items in git * `MG_CALLBACK_HMAC_SECRET=CHANGE_ME` placeholder in `.env.example` is **not** a leak — the literal value `CHANGE_ME` is documented in `tests/qtg/test_settings_load.py` as the sentinel that the boot gate rejects. * `TEST_PRIVATE_KEY_HEX = "0x" + "a" * 64` in `tests/qtg/test_local_private_key_signer.py` is a throwaway test key with zero on-chain assets ever attached. Documented in the test file header. *** ## See also * `docs/reference/dashboard-auth.md` — dashboard HMAC scheme + known limitations (incl. plaintext secret storage) * `docs/runbooks/dashboard-auth-v1-to-hmac.md` — historical migration from the legacy bundle-baked token (deprecated) * `docs/callback-verification-contract.md` — callback v3 signing contract * `docs/reference/signers/local-private-key-signer.md` — Tier-3 signer internals + migration to AWS KMS