> ## Documentation Index > Fetch the complete documentation index at: https://jephalabs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # KMS Signer Rotation > Step-by-step runbook for rotating AWS KMS signing keys # KMS Signer Rotation Runbook **Scope**: Scheduled rotation of a production EVM `AwsKmsEvmSigner` key. Covers preflight → drill → cut-over → KMS key deletion → exception handling. See the [Pro signer lifecycle reference](/pro/reference/signer-lifecycle) for the lifecycle API and CLI contract. > **This runbook covers prod EVM `AwsKmsEvmSigner` only.** > For Callback HMAC v3, CEX API keys, or Tier-3 local signer rotation, see > [§ 7 Related rotation procedures](#7-related-rotation-procedures). *** ## 1. Overview & Control Objective Mapping **Control objective**: > KMS asymmetric key rotation is a scheduled operator-run procedure. > The procedure, actor, asset movement, and completion time leave an evidence > chain in the audit log and rotation snapshot. | Evidence item | Storage location | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Action record | `audit_events` row (INSERT-only, PG TRIGGER enforced) | | Detailed payload | `signer_rotation_events.snapshot` (FK `audit_event_id` NOT NULL) | | Actor identity | `audit_events.actor_client_id` + `actor_key_id` (RBAC v2 admin role) | | Failure/block record | Retirement hard-gate denials only: `retire_denied` records lifecycle audit/event evidence. Identity and health rotation denials occur before lifecycle evidence and create **no** lifecycle audit event or `SignerRotationEvent`; they return only a code-only HTTP error. | **Control limit**: this covers app-role compromise protection. It does not cover a broad AWS account compromise or IAM root takeover. ### Evidence chain quick-look query ```sql theme={null} SELECT ae.created_at, ae.entity_type, ae.entity_key, ae.action, ae.outcome, ae.actor_client_id, ae.actor_key_id, ae.namespace, ae.new_value, sre.event_type, sre.snapshot FROM audit_events ae LEFT JOIN signer_rotation_events sre ON sre.audit_event_id = ae.id WHERE ae.entity_type = 'signer' AND ae.created_at >= NOW() - INTERVAL '12 months' ORDER BY ae.created_at; ``` *** ## 2. Preflight ### 2-1. Resolve the current signer identity (required — Gate 3 backstop) The Gate 3 nonterminal-node scan also matches `signer_binding['signer_address_snapshot']`. Record `old_address` before rotation. Use the supported synchronous bootstrap method; do not adapt an older async probe: ```bash theme={null} uv run python -c ' from qtg.infrastructure.signers.aws_kms_evm import AwsKmsEvmSigner signer = AwsKmsEvmSigner( signer_key="", kms_key_id="", aws_region="", aws_profile=None, ) print(signer.bootstrap_signer_address()) ' ``` `bootstrap_signer_address()` is synchronous. Record only the resulting address in the operator note, never KMS credentials or provider response payloads. ### 2-2. Create a new KMS asymmetric key ```bash theme={null} aws kms create-key \ --key-usage SIGN_VERIFY \ --key-spec ECC_SECG_P256K1 \ --description "qtg-signer--" \ --region ``` Record the output `KeyMetadata.KeyId` and `KeyMetadata.Arn`. #### Minimum IAM policy (grant to the QTG runtime role) ```json theme={null} { "Effect": "Allow", "Action": ["kms:DescribeKey", "kms:GetPublicKey", "kms:Sign"], "Resource": "arn:aws:kms:::key/" } ``` ### 2-3. Confirm the new signer EVM address ```bash theme={null} uv run python -c ' from qtg.infrastructure.signers.aws_kms_evm import AwsKmsEvmSigner signer = AwsKmsEvmSigner( signer_key="", kms_key_id="", aws_region="", aws_profile=None, ) print(signer.bootstrap_signer_address()) ' ``` Record the output as `candidate_live_address`. It is the fresh live identity used by the defense-in-depth verification below. ### 2-4. Enroll the replacement through runtime bootstrap Add an `aws_kms_evm` replacement entry to `MG_EXTRA_SIGNERS_JSON` using the same `signer_key`, `kms_key_id`, `aws_region`, and optional `aws_profile` / `aws_endpoint_url` fields used by the runtime bootstrap. Treat the JSON value as secret-bearing configuration: do not paste it into tickets, logs, transcripts, or the operator evidence packet. Restart the Pro application. Startup registers the configured signer and runs `sync_registry_to_db()` to materialize its registry row. Do **not** create a replacement runtime enrollment with direct SQL: a database row alone neither registers the in-memory signer nor proves its KMS identity. ### 2-5. Verify roster enrollment and pin expected identity Keep workers enabled with `MG_WORKERS_ENABLED=true` and wait for the executor-health worker to complete a successful signer health refresh. That worker persists the EVM `observed_signer_address`; a roster GET alone does not materialize the observed identity. Before pinning, confirm `GET /v3/signers` includes `` as active with a fresh observed identity and the deliberately unpinned state: ```json theme={null} { "signer_key": "", "status": "active", "observed_signer_address": "", "identity_state": "missing_expected" } ``` This first roster read proves runtime enrollment and health refresh, not identity approval. If the signer is not active, has no observed identity, or does not show `identity_state: "missing_expected"`, STOP and fix enrollment before the expected-identity request. `observed_signer_address` is diagnostic and non-authoritative; it is not a substitute for the server's fresh rotation-time comparison. Pin the independently recorded candidate identity with the shared signed HTTP client. `QTG_KEY_ID`, `QTG_HMAC_SECRET_FILE` (or `QTG_HMAC_SECRET`), and `QTG_API_URL` must already be configured; do not print any of their values. `QtgHttpClient.request("PUT", ...)` provides the HMAC-signed transport; there is no dedicated expected-identity CLI command. ```python theme={null} from qtg.clients.qtg_http import build_default_client signer_key = "" with build_default_client() as client: result = client.request("PUT", f"/v3/signers/{signer_key}/expected-identity", json_body={ "expected_address": "", "reason": "scheduled enrollment ", }, ) print({key: result[key] for key in ("signer_key", "outcome", "identity_state")}) ``` Re-read `GET /v3/signers` after the successful PUT. Require the pinned `expected_signer_address`, the fresh `observed_signer_address`, and `identity_state: "match"`. Any absence, invalid address, or mismatch is a STOP; do not proceed to drill or cut-over. ### 2-6. Defense-in-depth identity verification (required) Rotation-time identity enforcement runs inside the Pro server. For every `--drill` request and every live `--ack-impact` rotation, the server automatically resolves the replacement signer's fresh runtime identity and compares it with the persisted `expected_signer_address`. The CLI sends the rotation request; it does not perform this comparison locally. The server fails closed before impact calculation or any audit event, rotation event, metadata, or authority-remap write. `observed_signer_address` is diagnostic and non-authoritative; `identity_state: "match"` remains useful roster and health-refresh evidence but cannot authorize rotation. Identity denial returns HTTP 409 with one stable code and no address or backend-cause detail: | Error | Meaning | | ------------------------------------- | ----------------------------------------------------------------------- | | `signer_expected_identity_missing` | No operator pin exists. | | `signer_expected_identity_invalid` | The persisted pin is malformed or zero. | | `signer_runtime_identity_unavailable` | Fresh identity cannot be resolved or runtime signer continuity is lost. | | `signer_runtime_identity_invalid` | The fresh runtime value is malformed or zero. | | `signer_identity_mismatch` | Canonical expected and fresh identities differ. | Keep the independent KMS probe from Step 2-3 and compare its result with the pinned expected value before drill. This manual probe remains a defense-in-depth check; missing, invalid, or mismatched values are a **STOP** even though the server repeats the fresh resolution and comparison for the request. ### 2-7. Check new signer health (drill mode) ```bash theme={null} uv run qtg signer rotate \ --replace-with \ --drill \ --reason "preflight-health-" ``` Drill mode runs validation, persists drill evidence, and produces an impact report without changing signer bindings. Before calculating impact or persisting that evidence, the server freshly resolves and verifies the replacement identity. Confirm `outcome: "drilled"` in the output JSON. *** ## 3. Run Drill Mode (recommended before cut-over) ```bash theme={null} uv run qtg signer rotate \ --replace-with \ --drill \ --reason "drill-" ``` #### Impact report review checklist | Field | What to check | | ---------------------------------- | -------------------------------------------------------------------------------------------- | | `affected_authority_mappings` | Matches the expected number of authorities | | `remap_action` per row | Correct `update_in_place` vs `delete_then_keep_existing` branch | | `collision_existing_row_id` | Whether an existing row collision occurred; if present, an idempotent re-run may be possible | | `existing_account_address_changed` | If `true`, the existing row address is stale and will change after cut-over | Drill mode does not change signer bindings. Record the audit/event identifiers and the impact report in the sanitized operator evidence. *** ## 4. Live Cut-over Follow the 9-step sequence. Complete each step before moving to the next. ### Step 1 — Re-resolve replacement identity immediately before rotation Do not reuse `candidate_live_address` from the earlier drill preparation. Immediately before the state-changing command, resolve the replacement KMS identity again and compare it with the pinned value from the latest roster GET: ```bash theme={null} uv run python -c ' from qtg.infrastructure.signers.aws_kms_evm import AwsKmsEvmSigner signer = AwsKmsEvmSigner( signer_key="", kms_key_id="", aws_region="", aws_profile=None, ) pre_cutover_live_address = signer.bootstrap_signer_address() pinned_expected_address = "" if pre_cutover_live_address.lower() != pinned_expected_address.lower(): raise SystemExit("STOP: fresh KMS identity does not match pinned expected identity") print(pre_cutover_live_address) ' ``` An absent, invalid, or mismatched fresh identity is a STOP. Do not rotate until the pinned `expected_signer_address` and `pre_cutover_live_address` match; this is a second independent resolution, not a reuse of the drill probe. It is defense in depth: the server performs its own fresh resolution and comparison again when it handles the live rotation request. From this fresh probe through completion of the rotation response, **STOP** if any operator or deployment action would repoint or mutate the replacement KMS alias/key or replace the runtime signer object. Do not continue the in-flight request; restore a stable configuration, repeat the fresh probe, and submit a new rotation request. The automatic gate validates one fresh identity snapshot plus continuity of that same runtime signer object for the request; it does not perform a second in-lock KMS identity resolution. Keep the replacement configuration stable until the rotation response completes. ### Step 2 — Rotate (deprecate old, remap authorities) ```bash theme={null} uv run qtg signer rotate \ --replace-with \ --ack-impact \ --reason "scheduled-" ``` The server, not the CLI, repeats the fresh replacement-identity resolution and expected-pin comparison before it calculates impact or writes lifecycle evidence. On success, `old_signer_key.rotation_metadata.deprecated_from` is set. The OLD key is then excluded from new movement and template-compile candidates. **In-flight nodes can continue to sign, observe, and recover with the OLD key. This is expected.** ### Step 3 — Capture pre-drain inventory ```bash theme={null} uv run qtg signer inventory ``` Attach the output snapshot to the incident log. Check native gas, USDC, and USDT balances per chain. ### Step 4 — Manual drain (per chain) For each chain, use this order: **(1) provision gas -> (2) move ERC-20 assets -> (3) move native dust last**. Use a QTG movement or direct calldata. A retire attempt before drain completion is blocked by Gate 2. ### Step 5 — Scan allowances ```bash theme={null} uv run qtg signer allowances ``` Record the non-zero allowance list. Revoke per protocol: CCTP -> Circle bridge UI, Stargate -> Stargate UI / Etherscan write, Circle Gateway -> `approve(0)` calldata. ### Step 6 — Manual revoke Revoke each non-zero allowance from Step 5 in the protocol UI. Record revoke txids. ### Step 7 — Recheck post-drain inventory ```bash theme={null} uv run qtg signer inventory ``` Confirm all balances are at or below the dust level. If any balance remains, rerun Step 4. ### Step 8 — Recheck allowances ```bash theme={null} uv run qtg signer allowances ``` If non-zero allowances remain, complete revoke before proceeding. Use `--ack-allowances` only when there is a reason revoke cannot be completed, and record that reason in the operator note. ### Step 9 — Retire after a zero-allowance scan ```bash theme={null} uv run qtg signer retire \ --dust-threshold-usd 1.0 ``` Use `--ack-allowances` only when non-zero allowances deliberately remain and the operator has recorded why they cannot be revoked; then use the exceptional acknowledged-residual command: ```bash theme={null} uv run qtg signer retire \ --ack-allowances \ --dust-threshold-usd 1.0 ``` > **Unit warning for `--dust-threshold-usd`** > > This parameter is **not USD; it is token units in the smallest denomination**. > The Phase 7 v1 implementation compares raw balances without real USD conversion. > In production, conservatively **drain to actual zero** before retiring. > `1.0` means "1 token unit" (for example, 0.000001 USDC for USDC). All four gates must pass for retire to succeed: | Gate | Condition | | ------ | ------------------------------------------------------------------------------------------------ | | Gate 1 | `deprecated_from is not null` (confirms Step 2 completed) | | Gate 2 | All balances \<= `dust_threshold_usd` (token units) | | Gate 3 | No nonterminal `MovementRequestNode` references (`signer_key` + `signer_address_snapshot` match) | | Gate 4 | `--ack-allowances` is required if non-zero allowances remain | There is no `--force` flag. If a gate fails, see [§ 6 Exception scenarios](#6-exception-scenarios). *** ## 5. Delete the AWS KMS Key (post cut-over, out of band) This section is outside QTG. Run it after QTG retire completes. 1. Remove the `kms:Sign` / `kms:GetPublicKey` policy for the OLD key through the IAM console or CLI. 2. Schedule deletion. It can be canceled with `cancel-key-deletion` during the waiting period: ```bash theme={null} # Development/test: 7 days aws kms schedule-key-deletion --key-id --pending-window-in-days 7 # Production recommendation: 30 days aws kms schedule-key-deletion --key-id --pending-window-in-days 30 ``` *** ## 6. Exception scenarios ### Drain does not complete | Cause | Action | | ------------------------ | ------------------------------------------------------ | | Insufficient gas | Send a small amount of native token to the OLD address | | RPC failure | Retry only the affected chain with `--chains ` | | Protocol withdrawal lock | Wait for withdrawal completion, then retry | | Stuck allowance | etherscan write → `approve(spender, 0)` | ### In-flight node breaks Deprecated signers can still sign, observe, and recover in-flight nodes — deprecation stops new assignments, it does not revoke an in-flight one. If a node actually breaks, check the recovery worker logs and then run: ```sql theme={null} SELECT id, state, signer_binding, updated_at FROM movement_request_nodes WHERE state NOT IN ('completed','failed','cancelled') AND signer_binding::text LIKE '%%'; ``` If recovery is impossible, use the supported movement recovery actions (cancel, retry, or resume) and the ordinary operator recovery process. Do not mutate movement state directly or invent a failure endpoint. ### Follow-up checks after drill failure Drill failure means no state mutation occurred. Check whether the new signer passes `health()`, whether `signer_registry.status = 'active'`, and whether `signer_key` has a typo. ### Gate 2 dust threshold exceeded ``` retire denied: Gate 2 — balance exceeds dust_threshold_usd ``` Rerun Step 4 (Manual drain). After drain completes, rerun Steps 5–8, then retry Step 9 retirement. ### Gate 3 finds nonterminal nodes ``` retire denied: Gate 3 — nonterminal nodes reference this signer ``` ```sql theme={null} SELECT mrn.id, mrn.state, m.id as movement_id, m.status as movement_status FROM movement_request_nodes mrn JOIN movements m ON m.id = mrn.movement_id WHERE mrn.state NOT IN ('completed','failed','cancelled') AND ( mrn.signer_binding->>'signer_key' = '' OR mrn.signer_binding->>'signer_address_snapshot' = '' ); ``` Options: * **Wait**: wait until the node reaches a terminal state (recommended). * **Recover**: use the supported cancel, retry, or resume action as appropriate, then retry retire only after the node is terminal. ### Gate 4 allowance ack incomplete After revoke completes, retry retire without `--ack-allowances`. Proceed with `--ack-allowances` only when there is a reason UI revoke cannot be completed, and record the incomplete-revoke reason in the operator note. *** ## 7. Related rotation procedures | Procedure | Document | | --------------------------------- | ------------------------------------------------------------------------ | | Callback HMAC v3 rotation | `docs/runbooks/secret-rotation-checklist.md` § Callback HMAC v3 | | CEX API key rotation | `docs/runbooks/secret-rotation-checklist.md` § Exchange API keys | | Tier-3 local signer rotation | `docs/runbooks/secret-rotation-checklist.md` § Local signer key material | | ApiClientKey HMAC secret rotation | `docs/runbooks/secret-rotation-checklist.md` § ApiClientKey HMAC secrets | > **This runbook covers prod EVM `AwsKmsEvmSigner` only.** > The Pro KMS Signing Gateway (post-GA Pro spec), including runtime app-role > compromise protection, is handled by a separate spec. > > The AWS KMS rotation row in `docs/runbooks/secret-rotation-checklist.md` > forward-references this runbook from the secret inventory table. *** ## 8. SOC2/ISO27001 Evidence Query ### 8-1. All rotation actions in the last 12 months ```sql theme={null} SELECT ae.created_at, ae.entity_type, ae.entity_key, ae.action, ae.outcome, ae.actor_client_id, ae.actor_key_id, ae.namespace, ae.new_value, sre.event_type, sre.snapshot FROM audit_events ae LEFT JOIN signer_rotation_events sre ON sre.audit_event_id = ae.id WHERE ae.entity_type = 'signer' AND ae.created_at >= NOW() - INTERVAL '12 months' ORDER BY ae.created_at; ``` `event_type`: `rotate` / `rotate_drill` / `inventory` / `allowance_scan` / `retire` / `retire_denied`. `retire_denied` is reserved for retirement hard-gate denial evidence. Identity and health rotation denials do not create an event. ### 8-2. Actor role attribution Use `actor_client_id` + `actor_key_id` to look up the executor's role: ```sql theme={null} SELECT ae.*, ack.role FROM audit_events ae LEFT JOIN api_clients ac ON ac.id::text = ae.actor_client_id LEFT JOIN api_client_keys ack ON ack.client_id = ac.id AND ack.key_id = ae.actor_key_id WHERE ae.entity_type = 'signer' ORDER BY ae.created_at; ``` > RBAC v2 alembic 0003 swapped the `purpose` column to `role`. > Environments migrated from the old schema may have `ack.role` as NULL. ### 8-3. Query the full snapshot for a specific signer ```sql theme={null} SELECT ae.created_at, ae.action, ae.outcome, sre.event_type, sre.snapshot FROM audit_events ae JOIN signer_rotation_events sre ON sre.audit_event_id = ae.id WHERE ae.entity_key = '' ORDER BY ae.created_at; ``` `sre.snapshot`: drill/rotate -> impact report JSON, inventory/retire -> balance snapshot. *** ## References * Pro runtime CLI: `qtg signer` * Signer protocol and the local AWS KMS signer: [`reference/signers/signer-protocol.md`](/reference/signers/signer-protocol) * Control objectives for this rotation: section 1 of this runbook