mirror of
https://github.com/block/buzz.git
synced 2026-11-07 01:41:06 +00:00
a17243d846
## Goal Give operators a way to relieve database pressure in seconds, without restarting relay pods. Today every concurrency limit is an environment variable, so changing one means a rollout. During an incident, that rollout's restart sets off a reconnect storm of its own. ## What changes **Two runtime-tunable caps.** Each is read from an integer feature flag in a new process-wide evaluation context, and re-read every 5 s: | Flag | Caps | Ceiling | |---|---|---| | `buzz.relay.concurrency.history-reads` | subscription-history reads holding a reader connection (the #9132 gate) | the existing `pool − pool/5` share, which keeps readers free for authorization | | `buzz.relay.concurrency.auth` | verified NIP-42 AUTH frames running their database checks | none (unbounded today) | - **Flags can only tighten.** A positive value is clamped to the ceiling. Zero, a negative value, no value, or a provider that never initialized leaves the gate at its ceiling, which is today's behavior. Once LaunchDarkly has delivered values, they stay in force through a later outage, so a cap tightened during an incident survives a provider blip. - **Lowering never cancels admitted work.** A lowered gate admits nothing new until the work in flight falls under the new limit. - **Visibility:** `buzz_concurrency_limit{gate}` reports the applied limit (`0` means no limit), and each change logs a line. **One resizable gate.** `HistoryReadGate` becomes `buzz_db::ConcurrencyGate`, a FIFO semaphore whose limit can move anywhere in `1..=ceiling`: - Lowering it forgets free permits and records any shortfall as debt. - A permit released while there is debt is retired instead of returned. - Raising it repays debt first, then adds permits. All of this, and admission itself, settles against the debt under one lock, so a newer limit always supersedes an older one; there is no background resize task to race. Tokio hands a released permit to a queued waiter before it runs; if that waiter is then cancelled, tokio returns the permit to the semaphore directly, so admission retires any permit it gets while debt is outstanding. The AUTH cap uses the same type. **AUTH shedding.** The permit is taken after signature verification, so only real logins queue for the database checks. The wait is bounded at 2 s. A shed login gets a bare close with no `OK`, exactly like an auth timeout. I chose this from what each client does on main today: - **`OK false`:** the desktop app retries on an unjittered 60 s ladder. Mobile parks the session until the app resumes. The agent harness fails at startup. - **Bare close:** the desktop app, mobile, the agent harness and the CLI all reconnect on their jittered, capped backoff. On the relay, the shed is visible as access line `denied/concurrency_limit` and as `buzz_auth_outcomes_total{outcome="concurrency_limit"}`. **The feature-flag provider is chosen at startup.** `BUZZ_FEATURE_FLAGS_PROVIDER` takes `static` (the default), `environment` or `launchdarkly`, with `BUZZ_LAUNCHDARKLY_SDK_KEY` and an optional `BUZZ_LAUNCHDARKLY_RELAY_PROXY_URL`: - **Exactly one provider.** The value is parsed into a single provider, so there is no stacking and no fallback, and the evaluator never changes for the life of the process. The variable sits outside the `BUZZ_FEATURE_FLAG_` namespace, so it can never be read as a flag. - **Startup failures.** Local misconfiguration fails startup: an unknown provider, a missing key, or a malformed proxy URL. LaunchDarkly being unreachable, or rejecting the key, does not. The relay starts on declared defaults, the SDK keeps connecting, and `buzz_feature_flags_ready{provider}` shows whether the provider has initialized. A revoked key must not crash-loop every pod over a tuning knob. - **Every build carries the LaunchDarkly adapter.** It stays inert unless selected, so one published image serves every deployment and each deployment's configuration pins its provider. ## Design-doc amendments (`crates/buzz-feature-flags/README.md`) - **Provider selection.** The compile-time "exactly one provider" contract becomes the startup selection above. The only-one guarantee now holds per process instead of per build. - **Startup.** Failing closed now covers local configuration only. - **Guardrails.** "Flags may choose between equivalent implementations only" gains one exception: a flag may set an operational limit as long as it can only tighten it, and shedding under that limit stays retryable. - **Evaluation context.** `EvaluationContext::for_process()` adds a process context, which LaunchDarkly sees as a single `relay`-kind context keyed `relay`. ## What got simpler - **Removed the compile-time provider contract:** the nested fixture crate (with its own 2.8k-line `Cargo.lock`), its 229-line contract test, and the exact-one `compile_error!` guards it required consumers to copy. One startup parse plus a unit test replaces them. - **One gate type serves both caps.** The history-read gate became that generic type instead of gaining a sibling. - **Net change:** about −2.3k lines. ## Review-Proven Rules applied - **#1:** shedding is retryable (`PoolTimedOut` → CLOSED for history, a bare close for AUTH) and never an empty success. Provider failures are logged and surfaced through a gauge. - **#2:** the gate settles every limit change and permit release under one lock. The table test covers lowering while idle, lowering below the count held, lowering then raising before holders leave, lowering then raising past the count held, and restoring the ceiling. - **#3:** the tests drive the production call sites: `Db::get_channel_window_with_session` under a lowered and raised cap (Postgres), and `handle_auth` under a full AUTH gate. The controller test runs the real refresh loop. - **#4:** values are clamped, waits are bounded, and the refresh runs on a fixed interval with skipped missed ticks. - **#9:** AUTH shedding sends clients down their jittered reconnect backoff rather than an unjittered or terminal path (see above). ## Review rework (`92a975c8`, `3f09cb33`) - **Keys renamed** to `buzz.relay.concurrency.*`: LaunchDarkly keys cannot be renamed later, and the flag project may be shared with other services. - **Shutdown:** LaunchDarkly's final flush has no deadline of its own, so `close()` now runs it on a detached thread and waits at most 2 s, after the audit drain. (The blocking pool would not bound it: the runtime waits for blocking tasks when it drops.) - **Refresh loop:** a panicking refresh is caught, logged and counted (`buzz_concurrency_limit_refresh_panics_total`), and the next tick retries, so the caps cannot freeze silently. - **Runbook:** `docs/operator-concurrency-caps.md`, covering how to tighten a cap and the cases in which a tightened cap returns to its ceiling (mistyped or deleted flag, pod started during a provider outage). - **AUTH test:** asserts the second login passes the gate and returns its permit; the gate permit is renamed so it no longer shares a name with the NIP-FI effect permit. Gated on a Linux workstation at `92a975c8` (`3f09cb33` is rustfmt only; `cargo fmt --check` passes there): clippy `-D warnings`, `buzz-feature-flags` (50), `buzz-relay --lib --bins` (1740), the new modules under parallel `cargo test`, and the Postgres AUTH tests (27). Each new test fails under its mutant: the refresh without `catch_unwind`, `close()` with no deadline (hangs until killed), and the AUTH handler with the gate skipped. ## Testing Re-run in full at `3c289134` after review rework, on a Linux workstation: - `cargo fmt --check`; `cargo clippy -p buzz-feature-flags -p buzz-db -p buzz-relay --all-targets --all-features -D warnings`. - `buzz-feature-flags` with and without `launchdarkly` (50 and 24 tests), `buzz-db --lib` (209), and `buzz-relay --lib --bins` (1738), plus a plain parallel `cargo test` of the new relay modules. - Postgres: the `buzz-db` routed-read and history-gate tests (55), and the `buzz-relay` AUTH and connection tests (38). - `a_cancelled_waiter_cannot_reopen_a_lowered_gate` fails with the debt check in `admit` removed. - The new infra-free relay tests are added to the CI unit lane (`Justfile` and `scripts/run-tests.sh`). - E2E against a local relay at `06c39d64`: cap gauges, sequential logins and history under cap 1, a 20-login burst (19 shed with a bare close, all reconnected), and LaunchDarkly misconfiguration and unreachability. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
61 lines
2.7 KiB
Markdown
61 lines
2.7 KiB
Markdown
# Relay Concurrency Caps
|
|
|
|
Two per-pod caps can be tightened while the relay runs, through integer
|
|
feature flags evaluated in the process context (see
|
|
`crates/buzz-relay/src/concurrency_limits.rs`):
|
|
|
|
| Flag | Caps | Ceiling |
|
|
|------|------|---------|
|
|
| `buzz.relay.concurrency.history-reads` | subscription-history reads holding a reader-pool connection | the reader-pool share reserved for history |
|
|
| `buzz.relay.concurrency.auth` | verified NIP-42 AUTH frames in their database checks | none |
|
|
|
|
A positive value sets the cap, clamped to the ceiling. Zero, a negative value
|
|
or no value leaves the cap at its ceiling. A flag can only tighten a cap.
|
|
Each pod re-reads the flags every 5 s.
|
|
|
|
Past the history-read cap, a read waits up to 1 s and is then refused as
|
|
retryable. Past the AUTH cap, a login waits up to 2 s and then gets a bare
|
|
close, and the client reconnects on its own backoff.
|
|
|
|
## Tightening a cap
|
|
|
|
1. Set the flag to a positive value for the deployment's environment.
|
|
2. Within about 5 s, check `buzz_concurrency_limit{gate}` on every pod
|
|
(`history_reads` or `auth`). It shows the applied limit, with `0` meaning
|
|
no limit.
|
|
3. Watch the sheds: `buzz_auth_outcomes_total{outcome="concurrency_limit"}`
|
|
for AUTH.
|
|
|
|
To release, set the flag back to `0`.
|
|
|
|
## When a tightened cap loosens
|
|
|
|
A cap returns to its ceiling, without an error, when:
|
|
|
|
- **The flag value cannot be read as an integer.** A fractional, string or
|
|
otherwise mistyped value evaluates to the default `0`. Change values only
|
|
among the flag's defined numeric variations.
|
|
- **The flag is deleted or turned off with an off variation of `0`.**
|
|
- **A pod starts while the flag provider is unreachable.** The relay waits up
|
|
to 3 s for flag data, then serves defaults until the provider connects. A
|
|
pod restarted during a provider outage therefore comes up at the ceiling,
|
|
even while the cap is tightened for the pods that were already running.
|
|
`buzz_feature_flags_ready{provider} == 0` marks such a pod.
|
|
|
|
A pod that has already received flag data keeps its last-known values through
|
|
a later provider outage, so a running pod's cap does not loosen when the
|
|
connection drops.
|
|
|
|
During an incident in which the provider may be down, confirm
|
|
`buzz_concurrency_limit` on each pod after every restart or scale-up, not just
|
|
after the flag change.
|
|
|
|
## Health signals
|
|
|
|
- `buzz_feature_flags_ready{provider}`: `1` once the provider has delivered
|
|
data (always `1` for the `static` and `environment` providers).
|
|
- `buzz_concurrency_limit_refresh_panics_total`: a refresh that panicked. The
|
|
next refresh retries; a rising count means the caps are not following the
|
|
flags.
|
|
- Shutdown waits at most 2 s for the provider's final analytics flush.
|