Auth Rollout

HMAC authentication rollout plan and migration guide

Auth Enable Rollout Runbook

Purpose: an operator checklist to close easy-to-miss preparation items in order before switching MG_AUTH_ENABLED=true

---

1. What this document covers

This document organizes the preparation and verification procedure required when enabling v3 inbound HMAC auth for the first time.

Covered scope:

• caller inventory cleanup
• role assignment
• client/key bootstrap with the auth seed CLI
• verification of caller-side HMAC signing implementation
• staging enable sequence
• smoke test
• rollback

Out of scope:

• revoke/rotate lifecycle
• mTLS

---

2. Final target state

The final state should be the following.

• All /v3/** requests except /healthz require HMAC signatures
• Only registered callers can send requests
• Each caller is assigned one of three roles (admin / operator / agent)
• Replay nonces are recorded/rejected
• Operators can trace which caller uses which key

2-1. Current verification state (2026-03-21)

The following has already been verified.

• staging DB qtg_v3 created
• five staging callers seeded
  • dashboard-staging → operator
  • ops-cli-staging → operator
  • admin-cli-staging → admin
  • read-monitor-staging → operator
  • strategy-engine-staging → operator (or agent if it only files template proposals)
• local real HTTP smoke test completed with MG_AUTH_ENABLED=true
  • /healthz → 200
  • unsigned GET /v3/signers → 401
  • operator key GET /v3/signers → 200
  • operator key POST /v3/movements → 201
  • agent key against an admin-only route (e.g. PATCH /v3/executors/{executor_key}) → 403
  • operator key POST /v3/movements/{id}/approve → 200
  • admin key on an admin route -> expected 404 after auth passes (route reached, target row absent)

Local artifacts:

• .artifacts/auth-staging/staging-auth-seeds.json
• .artifacts/auth-staging/staging-auth-smoke-results.json
• .artifacts/auth-staging/staging-auth-external-http-smoke.json

Notes:

• seed artifacts contain real secrets, so they must be treated as local secrets.
• Secret distribution to actual external caller processes (dashboard/ops-cli/strategy-engine) is not yet complete.

---

3. Preflight checklist

A. Caller inventory

First, list every system that actually calls /v3/**.

Caller	Description	Assigned role	Actual route usage	Secret storage location	Done
dashboard	operator UI	operator	/approve, /reject, GET /v3/*		[ ]
ops-cli	operator CLI	operator	/actions/*, POST /v3/movements		[ ]
admin-cli	admin tasks	admin	PATCH /v3/executors/*, PATCH /v3/signers/*		[ ]
read-monitor	status monitoring	operator	GET /v3/*		[ ]
strategy-engine	movement creation	operator (or agent)	POST /v3/movements, agent template proposals		[ ]

If even one caller is missing, 401/403 errors can occur immediately after enable.

B. Verify role mapping

The three roles are:

• admin — full access, including template / signer / executor mutation
• operator — write and operate: create movements, approve/reject/retry/cancel, and all read surfaces
• agent — network-scoped automation: read surfaces plus its own agent-authority template proposals and bridge attempts (bound 1:1 to a AgentAuthority)

There is no read role; read surfaces are reachable by operator (and by admin/agent where the route allows it). The route → allowed-roles map lives in ROLE_ALLOWED_ROUTES (src/qtg/interfaces/api/middleware/auth/hmac.py). Current meanings:

• POST /v3/movements → any role (admin / operator / agent)
• GET /v3/movements, GET /v3/movements/{id}, GET /v3/movements/{id}/timeline → any role
• GET /v3/whoami, GET /v3/balances, GET /v3/route-catalog → any role
• GET /v3/plan-templates/{template_id} → any role
• POST /v3/movements/{id}/approve|reject → admin / operator
• POST /v3/movements/{id}/actions/resume|retry|cancel → admin / operator
• GET /v3/executors, GET /v3/signers → admin / operator
• POST /v3/plan-templates, PATCH /v3/plan-templates/{template_key} → admin
• PATCH /v3/executors/{executor_key} → admin
• PATCH /v3/signers/{signer_key} → admin
• POST /v3/admin/cutover-mode → admin
• POST /v3/registry/verify-drift → admin
• POST /v3/agent-authorities/{authority_key}/template-proposals → agent
• POST /v3/agent-authorities/{authority_key}/bridge-attempt → agent

An agent key is additionally namespace-scoped: AgentNamespaceMiddleware rejects requests whose path does not match the key’s bound AgentAuthority namespace with 403 agent_namespace_mismatch.

---

4. Key bootstrap

Seed CLI

Base command:

uv run python -m qtg.interfaces.tools.seed_auth_client \
  --client-name dashboard \
  --key-id dashboard-staging \
  --role operator

Options:

• --database-url — explicit target DB
• --hmac-secret — set the secret directly
• --cert-fingerprint — initial client metadata value

Default behavior:

• Reuse the active client if one with the same name exists
• Otherwise create a new ApiClient
• Create a new ApiClientKey
• Generate a random secret if none is specified
• Print the result once to JSON stdout

Checks during bootstrap

• agree on the client_name naming rule (dashboard, ops-cli, strategy-engine, etc.)
• agree on the key_id naming rule ({caller}-{env} recommended)
• retrieve hmac_secret from JSON stdout safely
• prevent the secret from remaining in shell history / CI logs / shared paste
• place the secret into the actual caller’s secret store

5. Caller implementation checklist

Each caller must implement/verify the following.

• X-QTG-Key-Id
• X-QTG-Timestamp
• X-QTG-Nonce
• X-QTG-Signature
• canonical string matches the server

Canonical string:

METHOD
PATH
QUERY
SHA256(raw_body_bytes)
TIMESTAMP
NONCE

• PATH is the routed, percent-decoded path the server dispatches on — no origin, no query string.
• QUERY is the query exactly as sent: no leading ?, no sorting, empty string if absent. Sorting before signing transmits bytes you did not sign, and every such request 401s.

Additional checks:

• nonce is a new value for every request
• timestamp uses the current epoch seconds
• body hash is computed from raw bytes
• GET requests also require auth (except /healthz)

6. Staging enable sequence

Phase 1 — verify wiring with auth off

• MG_AUTH_ENABLED=false
• deploy so that each caller can create signed requests
• prepare a smoke helper that can send signed requests in advance

Phase 2 — seed complete

• all required callers are seeded in the staging DB
• roles match the expected values
• secret distribution is complete for each caller

Phase 3 — smoke test

positive test:

• operator key → GET /v3/signers
• operator key → POST /v3/movements
• operator key → POST /v3/movements/{id}/approve
• operator key -> POST /v3/movements/{id}/actions/retry or /cancel
• admin key -> PATCH /v3/executors/{executor_key} or PATCH /v3/signers/{signer_key}
• agent key → POST /v3/agent-authorities/{authority_key}/template-proposals (within its namespace)

negative test:

• unsigned request → 401
• wrong secret → 401
• stale timestamp → 401
• replay nonce → 401
• role not allowed for the route → 403
• agent key outside its bound namespace → 403 (agent_namespace_mismatch)

Phase 4 — enable

• MG_AUTH_ENABLED=true in the staging env
• app restart / redeploy
• rerun the smoke test

7. Checks immediately after enable

• /healthz is still unauthenticated
• verify that unsigned /v3/** requests are blocked
• expected callers work normally
• auth failure reasons are logged
• verify NonceRegistry entries are recorded

Most common operator mistakes:

• a read-only monitor forgets GET auth
• an ad-hoc curl script is unsigned
• timestamp/nonce generation differs by caller, causing canonical mismatch

8. Rollback

If a problem occurs after auth enable, the simplest rollback is:

1. MG_AUTH_ENABLED=false
2. app restart / redeploy

Notes:

• Seeded ApiClient / ApiClientKey rows do not need to be deleted during rollback
• After identifying the root cause, fix the signed caller side and retry

9. Completion criteria

If all of the following are satisfied, consider “auth enable preparation complete.”

• caller inventory documented
• each caller role finalized
• key bootstrap completed with the seed CLI
• secret distribution completed
• staging positive/negative smoke tests passed
• rollback procedure verified

Only after that should MG_AUTH_ENABLED=true be enabled.

---

10. Key Rotation Procedure

When a key must be replaced after auth enable (periodic rotation, suspected exposure, etc.).

Principle

• Order: Seed new -> Deploy -> Verify -> Revoke old.
• If a new key_id is issued for the same client_name, it can stay active at the same time as the old key.
• Revoke the old key only after verifying that the caller works normally with the new key.

Procedure

Step 1. Check the current key

uv run python -m qtg.interfaces.tools.list_auth_keys \
  --database-url "$MG_DATABASE_URL"

Step 2. Issue a new key

uv run python -m qtg.interfaces.tools.seed_auth_client \
  --client-name dashboard \
  --key-id dashboard-staging-v2 \
  --role operator \
  --database-url "$MG_DATABASE_URL"

• Retrieve the hmac_secret value from the output safely.
• To avoid leaving it in shell history, use history -d or HISTCONTROL=ignorespace.

Step 3. Deploy the new secret

Set the new key_id and hmac_secret in the caller’s environment variables or .env file.

QTG_KEY_ID=dashboard-staging-v2
QTG_HMAC_SECRET=<new-secret>

Restart the caller process.

Step 4. Smoke test

Verify that normal requests succeed with the new key (see §6 Smoke test).

Step 5. Revoke the old key

uv run python -m qtg.interfaces.tools.revoke_auth_key \
  --key-id dashboard-staging-v1 \
  --database-url "$MG_DATABASE_URL"

• Immediately after revoke, all requests using the old key are rejected with 401.
• The DB row is not deleted and remains with status=revoked for auditability.

Step 6. Verify

uv run python -m qtg.interfaces.tools.list_auth_keys \
  --database-url "$MG_DATABASE_URL"

• old key: key_status=revoked
• new key: key_status=active

Emergency revoke (suspected exposure)

If exposure is suspected, run Step 5 immediately, then recover in order from Step 2. In this case the order is old-key revoke -> new-key issue -> deployment, so API access for that caller is interrupted from the revoke point until deployment of the new key is complete.

---

11. Secret deployment model

Current model: env-based deployment

QTG uses a single-host Docker Compose + Tailscale access model. Secrets are injected per caller through each process’s environment variables.

Deployment location	Method
QTG server	.env file -> Docker Compose env_file
Dashboard (same host)	.env or Docker env
Hummingbot controller (same host)	.env or config yaml
ops-cli (operator local)	~/.qtg/credentials or environment variables

Secret protection rules

• .env files use chmod 600 and are included in .gitignore
• The seed CLI output is shown only once, so save it immediately in a safe location
• Prevent secrets from remaining in CI/CD logs, shell history, or clipboard
• If later migrating to AWS Secrets Manager, Vault, and so on, keep env var names unchanged

---

Rate Limiting

After HMAC auth passes, an in-memory fixed-window rate check runs before call_next.

The limiter picks one of two per-key tiers from the caller’s role class (see RateLimiter.check in src/qtg/infrastructure/security/rate_limit.py):

• write tier — keys whose role is admin or operator (the write/operate-class roles)
• read tier — agent keys (read-class)

Setting	Default	Description
MG_RATE_LIMIT_ENABLED	true	enables rate limiting (requires MG_AUTH_ENABLED=true)
MG_RATE_LIMIT_READ_RPM	300	per-minute request limit for read-class (agent) keys
MG_RATE_LIMIT_WRITE_RPM	60	per-minute request limit for write-class (admin / operator) keys

• The tier is selected by the key’s role class, not by the individual route. A higher write limit therefore covers every route an operator or admin key can reach.
• When the limit is exceeded: 429 {"detail": "rate limit exceeded"} + Retry-After header
• /healthz and /dashboard/* are exempt from rate limiting
• It does not operate when rate_limit_enabled=True is set while auth_enabled=False