## Goal Give operators a way to relieve database pressure in seconds, without restarting relay pods. Today every concurrency limit is an environment variable, so changing one means a rollout. During an incident, that rollout's restart sets off a reconnect storm of its own. ## What changes **Two runtime-tunable caps.** Each is read from an integer feature flag in a new process-wide evaluation context, and re-read every 5 s: | Flag | Caps | Ceiling | |---|---|---| | `buzz.relay.concurrency.history-reads` | subscription-history reads holding a reader connection (the #9132 gate) | the existing `pool − pool/5` share, which keeps readers free for authorization | | `buzz.relay.concurrency.auth` | verified NIP-42 AUTH frames running their database checks | none (unbounded today) | - **Flags can only tighten.** A positive value is clamped to the ceiling. Zero, a negative value, no value, or a provider that never initialized leaves the gate at its ceiling, which is today's behavior. Once LaunchDarkly has delivered values, they stay in force through a later outage, so a cap tightened during an incident survives a provider blip. - **Lowering never cancels admitted work.** A lowered gate admits nothing new until the work in flight falls under the new limit. - **Visibility:** `buzz_concurrency_limit{gate}` reports the applied limit (`0` means no limit), and each change logs a line. **One resizable gate.** `HistoryReadGate` becomes `buzz_db::ConcurrencyGate`, a FIFO semaphore whose limit can move anywhere in `1..=ceiling`: - Lowering it forgets free permits and records any shortfall as debt. - A permit released while there is debt is retired instead of returned. - Raising it repays debt first, then adds permits. All of this, and admission itself, settles against the debt under one lock, so a newer limit always supersedes an older one; there is no background resize task to race. Tokio hands a released permit to a queued waiter before it runs; if that waiter is then cancelled, tokio returns the permit to the semaphore directly, so admission retires any permit it gets while debt is outstanding. The AUTH cap uses the same type. **AUTH shedding.** The permit is taken after signature verification, so only real logins queue for the database checks. The wait is bounded at 2 s. A shed login gets a bare close with no `OK`, exactly like an auth timeout. I chose this from what each client does on main today: - **`OK false`:** the desktop app retries on an unjittered 60 s ladder. Mobile parks the session until the app resumes. The agent harness fails at startup. - **Bare close:** the desktop app, mobile, the agent harness and the CLI all reconnect on their jittered, capped backoff. On the relay, the shed is visible as access line `denied/concurrency_limit` and as `buzz_auth_outcomes_total{outcome="concurrency_limit"}`. **The feature-flag provider is chosen at startup.** `BUZZ_FEATURE_FLAGS_PROVIDER` takes `static` (the default), `environment` or `launchdarkly`, with `BUZZ_LAUNCHDARKLY_SDK_KEY` and an optional `BUZZ_LAUNCHDARKLY_RELAY_PROXY_URL`: - **Exactly one provider.** The value is parsed into a single provider, so there is no stacking and no fallback, and the evaluator never changes for the life of the process. The variable sits outside the `BUZZ_FEATURE_FLAG_` namespace, so it can never be read as a flag. - **Startup failures.** Local misconfiguration fails startup: an unknown provider, a missing key, or a malformed proxy URL. LaunchDarkly being unreachable, or rejecting the key, does not. The relay starts on declared defaults, the SDK keeps connecting, and `buzz_feature_flags_ready{provider}` shows whether the provider has initialized. A revoked key must not crash-loop every pod over a tuning knob. - **Every build carries the LaunchDarkly adapter.** It stays inert unless selected, so one published image serves every deployment and each deployment's configuration pins its provider. ## Design-doc amendments (`crates/buzz-feature-flags/README.md`) - **Provider selection.** The compile-time "exactly one provider" contract becomes the startup selection above. The only-one guarantee now holds per process instead of per build. - **Startup.** Failing closed now covers local configuration only. - **Guardrails.** "Flags may choose between equivalent implementations only" gains one exception: a flag may set an operational limit as long as it can only tighten it, and shedding under that limit stays retryable. - **Evaluation context.** `EvaluationContext::for_process()` adds a process context, which LaunchDarkly sees as a single `relay`-kind context keyed `relay`. ## What got simpler - **Removed the compile-time provider contract:** the nested fixture crate (with its own 2.8k-line `Cargo.lock`), its 229-line contract test, and the exact-one `compile_error!` guards it required consumers to copy. One startup parse plus a unit test replaces them. - **One gate type serves both caps.** The history-read gate became that generic type instead of gaining a sibling. - **Net change:** about −2.3k lines. ## Review-Proven Rules applied - **#1:** shedding is retryable (`PoolTimedOut` → CLOSED for history, a bare close for AUTH) and never an empty success. Provider failures are logged and surfaced through a gauge. - **#2:** the gate settles every limit change and permit release under one lock. The table test covers lowering while idle, lowering below the count held, lowering then raising before holders leave, lowering then raising past the count held, and restoring the ceiling. - **#3:** the tests drive the production call sites: `Db::get_channel_window_with_session` under a lowered and raised cap (Postgres), and `handle_auth` under a full AUTH gate. The controller test runs the real refresh loop. - **#4:** values are clamped, waits are bounded, and the refresh runs on a fixed interval with skipped missed ticks. - **#9:** AUTH shedding sends clients down their jittered reconnect backoff rather than an unjittered or terminal path (see above). ## Review rework (`92a975c8`, `3f09cb33`) - **Keys renamed** to `buzz.relay.concurrency.*`: LaunchDarkly keys cannot be renamed later, and the flag project may be shared with other services. - **Shutdown:** LaunchDarkly's final flush has no deadline of its own, so `close()` now runs it on a detached thread and waits at most 2 s, after the audit drain. (The blocking pool would not bound it: the runtime waits for blocking tasks when it drops.) - **Refresh loop:** a panicking refresh is caught, logged and counted (`buzz_concurrency_limit_refresh_panics_total`), and the next tick retries, so the caps cannot freeze silently. - **Runbook:** `docs/operator-concurrency-caps.md`, covering how to tighten a cap and the cases in which a tightened cap returns to its ceiling (mistyped or deleted flag, pod started during a provider outage). - **AUTH test:** asserts the second login passes the gate and returns its permit; the gate permit is renamed so it no longer shares a name with the NIP-FI effect permit. Gated on a Linux workstation at `92a975c8` (`3f09cb33` is rustfmt only; `cargo fmt --check` passes there): clippy `-D warnings`, `buzz-feature-flags` (50), `buzz-relay --lib --bins` (1740), the new modules under parallel `cargo test`, and the Postgres AUTH tests (27). Each new test fails under its mutant: the refresh without `catch_unwind`, `close()` with no deadline (hangs until killed), and the AUTH handler with the gate skipped. ## Testing Re-run in full at `3c289134` after review rework, on a Linux workstation: - `cargo fmt --check`; `cargo clippy -p buzz-feature-flags -p buzz-db -p buzz-relay --all-targets --all-features -D warnings`. - `buzz-feature-flags` with and without `launchdarkly` (50 and 24 tests), `buzz-db --lib` (209), and `buzz-relay --lib --bins` (1738), plus a plain parallel `cargo test` of the new relay modules. - Postgres: the `buzz-db` routed-read and history-gate tests (55), and the `buzz-relay` AUTH and connection tests (38). - `a_cancelled_waiter_cannot_reopen_a_lowered_gate` fails with the debt check in `admit` removed. - The new infra-free relay tests are added to the CI unit lane (`Justfile` and `scripts/run-tests.sh`). - E2E against a local relay at `06c39d64`: cap gauges, sequential logins and history under cap 1, a 20-login burst (19 shed with a bare close, all reconnected), and LaunchDarkly misconfiguration and unreachability. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
buzz-feature-flags
Provider-neutral, typed feature-flag contracts for Buzz server crates.
This crate is infrastructure-only. It defines a small API for flag evaluation, plus a static evaluator and an optional LaunchDarkly adapter behind a compile feature.
Purpose
buzz-feature-flags centralizes typed flag evaluation so server crates can use
flags without importing vendor SDK types.
Provider-neutral types:
BooleanFlagIntegerFlag(i64, including negative values)EvaluationContext(aCommunityIdwith an optional actorPublicKey, orfor_process()for process-wide settings such as concurrency caps)FlagEvaluator(typedevaluate_boolandevaluate_int)StaticEvaluator(always returns declared defaults)EnvironmentEvaluator(process-wide immutable environment snapshot)EnvironmentDiagnostic(sanitized malformed-value report)
LaunchDarkly support is optional and compile-gated behind
buzz-feature-flags/launchdarkly.
EnvironmentEvaluator is provider-neutral and always available (no Cargo
feature and no external dependency).
Environment Variable Naming
EnvironmentEvaluator reads only the fixed namespace
BUZZ_FEATURE_FLAG_.
For each flag key:
- uppercase ASCII letters,
- replace each run of non-ASCII-alphanumeric characters with
_, - trim leading/trailing
_, - prepend
BUZZ_FEATURE_FLAG_.
Example: relay.feature.query-v2 → BUZZ_FEATURE_FLAG_RELAY_FEATURE_QUERY_V2.
Normalized names must be unique across declared flags. Collisions (for example
a-b and a_b) intentionally map to the same environment variable. Enforcing
uniqueness belongs in a future flag registry; this crate does not add a
registry yet.
Relay Provider Selection
buzz-relay builds exactly one evaluator at startup, chosen by
BUZZ_FEATURE_FLAGS_PROVIDER (outside the BUZZ_FEATURE_FLAG_ namespace, so
it can never be read as a flag):
| Value | Evaluator |
|---|---|
unset, static |
StaticEvaluator |
environment |
EnvironmentEvaluator::from_process_environment() |
launchdarkly |
LaunchDarklyEvaluator, from BUZZ_LAUNCHDARKLY_SDK_KEY and the optional Relay Proxy URL BUZZ_LAUNCHDARKLY_RELAY_PROXY_URL |
The value is parsed into one provider; there is no stacking, no precedence and
no fallback from one provider to another, and the evaluator never changes for
the life of the process. Every relay build carries the LaunchDarkly adapter
(buzz-feature-flags/launchdarkly), which stays inert unless selected, so one
published artifact serves every deployment and the deployment's configuration
pins its provider.
The composition root lives in crates/buzz-relay/src/feature_flags.rs. It
constructs the evaluator once and hands consumers the shared
Arc<dyn FlagEvaluator>, keeping the concrete owner to drain environment
diagnostics and to call LaunchDarkly's close() at shutdown.
Startup fails on a local configuration error: an unknown provider,
launchdarkly without an SDK key, or a malformed Relay Proxy URL. It does not
fail when LaunchDarkly is unreachable: the relay waits briefly for flag data,
then serves declared defaults while the SDK keeps connecting, and
buzz_feature_flags_ready{provider} reports whether the provider has
initialized. Once initialized, LaunchDarkly serves its last-known values
through a later outage, so the gauge stays 1 and flag values do not revert to
defaults until the SDK reconnects and receives new data.
buzz-db Boundary (approved correction)
buzz-db may evaluate flags internally when choosing between equivalent query
implementations. It should depend on buzz-feature-flags without enabling the
LaunchDarkly feature, and it should never construct or import provider types.
Concise example (query path selection only):
use buzz_core::CommunityId;
use std::sync::Arc;
use buzz_feature_flags::{EvaluationContext, FlagEvaluator, IntegerFlag};
pub struct Db {
feature_flags: Arc<dyn FlagEvaluator>,
}
impl Db {
pub async fn list_events(
&self,
community: CommunityId,
) -> anyhow::Result<Vec<EventRow>> {
let context = EvaluationContext::for_community(community);
let query_version = self.feature_flags.evaluate_int(
IntegerFlag::new("db.events.query-version", 1),
&context,
);
if query_version >= 2 {
self.list_events_v2_sql(community).await
} else {
self.list_events_v1_sql(community).await
}
}
}
SQL details are intentionally omitted in this example; the key point is that
public Db methods accept only domain inputs, while feature-flag evaluator
wiring remains internal to Db construction.
EvaluationContext::for_actor is appropriate only when the existing DB
operation already receives an authoritative authenticated actor for domain
behavior. Feature targeting must not add actor/pubkey parameters to otherwise
actor-free DB APIs.
For LaunchDarkly, a community evaluation always includes kind community plus
optional global kind pubkey; a process evaluation is a single kind relay
context keyed relay. Percentage rollouts must explicitly choose a
kind present in evaluation: rollouts must set contextKind to a kind present
in evaluation (community or pubkey) when they are intended to bucket Buzz
traffic. LaunchDarkly rollouts where configuration omits contextKind defaults
to user; Buzz does not synthesize a user alias, so those rollouts bucket as
zero and pick the first positive-weight variation.
Guardrails
- Evaluation context inputs are authoritative server-resolved values
(
community, optionalactor), not client-supplied targeting attributes. - Flags may choose between equivalent implementations only, with one exception: an operational limit (a concurrency cap) may be set by a flag when the flag can only tighten it. The consumer clamps the value to a ceiling fixed at startup, and a missing, non-positive or unavailable value means the ceiling. Shedding under such a limit must be retryable.
- Flags must not weaken authorization, tenant isolation, ordering guarantees, transaction/consistency behavior, or schema invariants.
- Declared defaults choose the established safe path.
- Integer values must be validated at the consumer boundary before affecting behavior.
Fallback, Startup, and Lifecycle
StaticEvaluatoralways returns each flag's declared default.EnvironmentEvaluatorreturns declared defaults when values are missing, empty, invalid, non-Unicode, or the normalized flag key is unusable. Present malformed values also produce a sanitized diagnostic; missing values do not.- LaunchDarkly adapter returns declared defaults when a flag is missing, wrong type, or evaluation fails.
- LaunchDarkly numbers are SDK
f64values. Integer evaluation accepts only mathematically integral values that convert toi64exactly from thatf64representation; otherwise it returns each flag's declared default. - Startup chooses one evaluator from configuration (
StaticEvaluator,EnvironmentEvaluator, or LaunchDarkly); provider precedence/stacking is out of scope. - If LaunchDarkly is used, call evaluator
close()during process shutdown.close()blocks the calling thread while it flushes analytics and has no deadline of its own, so async shutdown code should run it off the runtime and bound the wait (the relay gives it a detached thread and 2 s). - Safety invariants must never rely on remote-flag availability.
Build and Test Expectations
Build modes:
# Provider-neutral (default): no LaunchDarkly dependency activated
cargo build -p buzz-feature-flags
# Provider-neutral environment evaluator is included in the default build
cargo test -p buzz-feature-flags --test environment_evaluator
# LaunchDarkly adapter enabled
cargo build -p buzz-feature-flags --features launchdarkly
Dependency-graph expectation checks:
# Default graph should exclude launchdarkly-server-sdk
cargo tree -p buzz-feature-flags
# Feature graph should include launchdarkly-server-sdk
cargo tree -p buzz-feature-flags --features launchdarkly
Testing expectations:
- Run provider-neutral tests and LaunchDarkly-feature tests for this crate.
- When
buzz-dbadopts flag-gated query selection, add parity tests proving old/new query paths return equivalent rows, ordering, and transactional behavior for the same inputs.