Files
buzz/crates/buzz-feature-flags/README.md
Luke Tornquist a17243d846 feat(relay): runtime-tunable history-read and AUTH concurrency caps via feature flags (#9170)
## Goal

Give operators a way to relieve database pressure in seconds, without
restarting relay pods. Today every concurrency limit is an environment
variable, so changing one means a rollout. During an incident, that
rollout's restart sets off a reconnect storm of its own.

## What changes

**Two runtime-tunable caps.** Each is read from an integer feature flag
in a new process-wide evaluation context, and re-read every 5 s:

| Flag | Caps | Ceiling |
|---|---|---|
| `buzz.relay.concurrency.history-reads` | subscription-history reads
holding a reader connection (the #9132 gate) | the existing `pool −
pool/5` share, which keeps readers free for authorization |
| `buzz.relay.concurrency.auth` | verified NIP-42 AUTH frames running
their database checks | none (unbounded today) |

- **Flags can only tighten.** A positive value is clamped to the
ceiling. Zero, a negative value, no value, or a provider that never
initialized leaves the gate at its ceiling, which is today's behavior.
Once LaunchDarkly has delivered values, they stay in force through a
later outage, so a cap tightened during an incident survives a provider
blip.
- **Lowering never cancels admitted work.** A lowered gate admits
nothing new until the work in flight falls under the new limit.
- **Visibility:** `buzz_concurrency_limit{gate}` reports the applied
limit (`0` means no limit), and each change logs a line.

**One resizable gate.** `HistoryReadGate` becomes
`buzz_db::ConcurrencyGate`, a FIFO semaphore whose limit can move
anywhere in `1..=ceiling`:
- Lowering it forgets free permits and records any shortfall as debt.
- A permit released while there is debt is retired instead of returned.
- Raising it repays debt first, then adds permits.

All of this, and admission itself, settles against the debt under one
lock, so a newer limit always supersedes an older one; there is no
background resize task to race. Tokio hands a released permit to a
queued waiter before it runs; if that waiter is then cancelled, tokio
returns the permit to the semaphore directly, so admission retires any
permit it gets while debt is outstanding. The AUTH cap uses the same
type.

**AUTH shedding.** The permit is taken after signature verification, so
only real logins queue for the database checks. The wait is bounded at 2
s. A shed login gets a bare close with no `OK`, exactly like an auth
timeout. I chose this from what each client does on main today:
- **`OK false`:** the desktop app retries on an unjittered 60 s ladder.
Mobile parks the session until the app resumes. The agent harness fails
at startup.
- **Bare close:** the desktop app, mobile, the agent harness and the CLI
all reconnect on their jittered, capped backoff.

On the relay, the shed is visible as access line
`denied/concurrency_limit` and as
`buzz_auth_outcomes_total{outcome="concurrency_limit"}`.

**The feature-flag provider is chosen at startup.**
`BUZZ_FEATURE_FLAGS_PROVIDER` takes `static` (the default),
`environment` or `launchdarkly`, with `BUZZ_LAUNCHDARKLY_SDK_KEY` and an
optional `BUZZ_LAUNCHDARKLY_RELAY_PROXY_URL`:
- **Exactly one provider.** The value is parsed into a single provider,
so there is no stacking and no fallback, and the evaluator never changes
for the life of the process. The variable sits outside the
`BUZZ_FEATURE_FLAG_` namespace, so it can never be read as a flag.
- **Startup failures.** Local misconfiguration fails startup: an unknown
provider, a missing key, or a malformed proxy URL. LaunchDarkly being
unreachable, or rejecting the key, does not. The relay starts on
declared defaults, the SDK keeps connecting, and
`buzz_feature_flags_ready{provider}` shows whether the provider has
initialized. A revoked key must not crash-loop every pod over a tuning
knob.
- **Every build carries the LaunchDarkly adapter.** It stays inert
unless selected, so one published image serves every deployment and each
deployment's configuration pins its provider.

## Design-doc amendments (`crates/buzz-feature-flags/README.md`)

- **Provider selection.** The compile-time "exactly one provider"
contract becomes the startup selection above. The only-one guarantee now
holds per process instead of per build.
- **Startup.** Failing closed now covers local configuration only.
- **Guardrails.** "Flags may choose between equivalent implementations
only" gains one exception: a flag may set an operational limit as long
as it can only tighten it, and shedding under that limit stays
retryable.
- **Evaluation context.** `EvaluationContext::for_process()` adds a
process context, which LaunchDarkly sees as a single `relay`-kind
context keyed `relay`.

## What got simpler

- **Removed the compile-time provider contract:** the nested fixture
crate (with its own 2.8k-line `Cargo.lock`), its 229-line contract test,
and the exact-one `compile_error!` guards it required consumers to copy.
One startup parse plus a unit test replaces them.
- **One gate type serves both caps.** The history-read gate became that
generic type instead of gaining a sibling.
- **Net change:** about −2.3k lines.

## Review-Proven Rules applied

- **#1:** shedding is retryable (`PoolTimedOut` → CLOSED for history, a
bare close for AUTH) and never an empty success. Provider failures are
logged and surfaced through a gauge.
- **#2:** the gate settles every limit change and permit release under
one lock. The table test covers lowering while idle, lowering below the
count held, lowering then raising before holders leave, lowering then
raising past the count held, and restoring the ceiling.
- **#3:** the tests drive the production call sites:
`Db::get_channel_window_with_session` under a lowered and raised cap
(Postgres), and `handle_auth` under a full AUTH gate. The controller
test runs the real refresh loop.
- **#4:** values are clamped, waits are bounded, and the refresh runs on
a fixed interval with skipped missed ticks.
- **#9:** AUTH shedding sends clients down their jittered reconnect
backoff rather than an unjittered or terminal path (see above).

## Review rework (`92a975c8`, `3f09cb33`)

- **Keys renamed** to `buzz.relay.concurrency.*`: LaunchDarkly keys
cannot be renamed later, and the flag project may be shared with other
services.
- **Shutdown:** LaunchDarkly's final flush has no deadline of its own,
so `close()` now runs it on a detached thread and waits at most 2 s,
after the audit drain. (The blocking pool would not bound it: the
runtime waits for blocking tasks when it drops.)
- **Refresh loop:** a panicking refresh is caught, logged and counted
(`buzz_concurrency_limit_refresh_panics_total`), and the next tick
retries, so the caps cannot freeze silently.
- **Runbook:** `docs/operator-concurrency-caps.md`, covering how to
tighten a cap and the cases in which a tightened cap returns to its
ceiling (mistyped or deleted flag, pod started during a provider
outage).
- **AUTH test:** asserts the second login passes the gate and returns
its permit; the gate permit is renamed so it no longer shares a name
with the NIP-FI effect permit.

Gated on a Linux workstation at `92a975c8` (`3f09cb33` is rustfmt only;
`cargo fmt --check` passes there): clippy `-D warnings`,
`buzz-feature-flags` (50), `buzz-relay --lib --bins` (1740), the new
modules under parallel `cargo test`, and the Postgres AUTH tests (27).
Each new test fails under its mutant: the refresh without
`catch_unwind`, `close()` with no deadline (hangs until killed), and the
AUTH handler with the gate skipped.

## Testing

Re-run in full at `3c289134` after review rework, on a Linux
workstation:

- `cargo fmt --check`; `cargo clippy -p buzz-feature-flags -p buzz-db -p
buzz-relay --all-targets --all-features -D warnings`.
- `buzz-feature-flags` with and without `launchdarkly` (50 and 24
tests), `buzz-db --lib` (209), and `buzz-relay --lib --bins` (1738),
plus a plain parallel `cargo test` of the new relay modules.
- Postgres: the `buzz-db` routed-read and history-gate tests (55), and
the `buzz-relay` AUTH and connection tests (38).
- `a_cancelled_waiter_cannot_reopen_a_lowered_gate` fails with the debt
check in `admit` removed.
- The new infra-free relay tests are added to the CI unit lane
(`Justfile` and `scripts/run-tests.sh`).
- E2E against a local relay at `06c39d64`: cap gauges, sequential logins
and history under cap 1, a 20-login burst (19 shed with a bare close,
all reconnected), and LaunchDarkly misconfiguration and unreachability.


🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:48:06 -05:00

8.4 KiB

buzz-feature-flags

Provider-neutral, typed feature-flag contracts for Buzz server crates.

This crate is infrastructure-only. It defines a small API for flag evaluation, plus a static evaluator and an optional LaunchDarkly adapter behind a compile feature.

Purpose

buzz-feature-flags centralizes typed flag evaluation so server crates can use flags without importing vendor SDK types.

Provider-neutral types:

  • BooleanFlag
  • IntegerFlag (i64, including negative values)
  • EvaluationContext (a CommunityId with an optional actor PublicKey, or for_process() for process-wide settings such as concurrency caps)
  • FlagEvaluator (typed evaluate_bool and evaluate_int)
  • StaticEvaluator (always returns declared defaults)
  • EnvironmentEvaluator (process-wide immutable environment snapshot)
  • EnvironmentDiagnostic (sanitized malformed-value report)

LaunchDarkly support is optional and compile-gated behind buzz-feature-flags/launchdarkly.

EnvironmentEvaluator is provider-neutral and always available (no Cargo feature and no external dependency).

Environment Variable Naming

EnvironmentEvaluator reads only the fixed namespace BUZZ_FEATURE_FLAG_.

For each flag key:

  • uppercase ASCII letters,
  • replace each run of non-ASCII-alphanumeric characters with _,
  • trim leading/trailing _,
  • prepend BUZZ_FEATURE_FLAG_.

Example: relay.feature.query-v2 → BUZZ_FEATURE_FLAG_RELAY_FEATURE_QUERY_V2.

Normalized names must be unique across declared flags. Collisions (for example a-b and a_b) intentionally map to the same environment variable. Enforcing uniqueness belongs in a future flag registry; this crate does not add a registry yet.

Relay Provider Selection

buzz-relay builds exactly one evaluator at startup, chosen by BUZZ_FEATURE_FLAGS_PROVIDER (outside the BUZZ_FEATURE_FLAG_ namespace, so it can never be read as a flag):

Value Evaluator
unset, static StaticEvaluator
environment EnvironmentEvaluator::from_process_environment()
launchdarkly LaunchDarklyEvaluator, from BUZZ_LAUNCHDARKLY_SDK_KEY and the optional Relay Proxy URL BUZZ_LAUNCHDARKLY_RELAY_PROXY_URL

The value is parsed into one provider; there is no stacking, no precedence and no fallback from one provider to another, and the evaluator never changes for the life of the process. Every relay build carries the LaunchDarkly adapter (buzz-feature-flags/launchdarkly), which stays inert unless selected, so one published artifact serves every deployment and the deployment's configuration pins its provider.

The composition root lives in crates/buzz-relay/src/feature_flags.rs. It constructs the evaluator once and hands consumers the shared Arc<dyn FlagEvaluator>, keeping the concrete owner to drain environment diagnostics and to call LaunchDarkly's close() at shutdown.

Startup fails on a local configuration error: an unknown provider, launchdarkly without an SDK key, or a malformed Relay Proxy URL. It does not fail when LaunchDarkly is unreachable: the relay waits briefly for flag data, then serves declared defaults while the SDK keeps connecting, and buzz_feature_flags_ready{provider} reports whether the provider has initialized. Once initialized, LaunchDarkly serves its last-known values through a later outage, so the gauge stays 1 and flag values do not revert to defaults until the SDK reconnects and receives new data.

buzz-db Boundary (approved correction)

buzz-db may evaluate flags internally when choosing between equivalent query implementations. It should depend on buzz-feature-flags without enabling the LaunchDarkly feature, and it should never construct or import provider types.

Concise example (query path selection only):

use buzz_core::CommunityId;
use std::sync::Arc;

use buzz_feature_flags::{EvaluationContext, FlagEvaluator, IntegerFlag};

pub struct Db {
    feature_flags: Arc<dyn FlagEvaluator>,
}

impl Db {
    pub async fn list_events(
        &self,
        community: CommunityId,
    ) -> anyhow::Result<Vec<EventRow>> {
        let context = EvaluationContext::for_community(community);

        let query_version = self.feature_flags.evaluate_int(
            IntegerFlag::new("db.events.query-version", 1),
            &context,
        );

        if query_version >= 2 {
            self.list_events_v2_sql(community).await
        } else {
            self.list_events_v1_sql(community).await
        }
    }
}

SQL details are intentionally omitted in this example; the key point is that public Db methods accept only domain inputs, while feature-flag evaluator wiring remains internal to Db construction.

EvaluationContext::for_actor is appropriate only when the existing DB operation already receives an authoritative authenticated actor for domain behavior. Feature targeting must not add actor/pubkey parameters to otherwise actor-free DB APIs.

For LaunchDarkly, a community evaluation always includes kind community plus optional global kind pubkey; a process evaluation is a single kind relay context keyed relay. Percentage rollouts must explicitly choose a kind present in evaluation: rollouts must set contextKind to a kind present in evaluation (community or pubkey) when they are intended to bucket Buzz traffic. LaunchDarkly rollouts where configuration omits contextKind defaults to user; Buzz does not synthesize a user alias, so those rollouts bucket as zero and pick the first positive-weight variation.

Guardrails

  • Evaluation context inputs are authoritative server-resolved values (community, optional actor), not client-supplied targeting attributes.
  • Flags may choose between equivalent implementations only, with one exception: an operational limit (a concurrency cap) may be set by a flag when the flag can only tighten it. The consumer clamps the value to a ceiling fixed at startup, and a missing, non-positive or unavailable value means the ceiling. Shedding under such a limit must be retryable.
  • Flags must not weaken authorization, tenant isolation, ordering guarantees, transaction/consistency behavior, or schema invariants.
  • Declared defaults choose the established safe path.
  • Integer values must be validated at the consumer boundary before affecting behavior.

Fallback, Startup, and Lifecycle

  • StaticEvaluator always returns each flag's declared default.
  • EnvironmentEvaluator returns declared defaults when values are missing, empty, invalid, non-Unicode, or the normalized flag key is unusable. Present malformed values also produce a sanitized diagnostic; missing values do not.
  • LaunchDarkly adapter returns declared defaults when a flag is missing, wrong type, or evaluation fails.
  • LaunchDarkly numbers are SDK f64 values. Integer evaluation accepts only mathematically integral values that convert to i64 exactly from that f64 representation; otherwise it returns each flag's declared default.
  • Startup chooses one evaluator from configuration (StaticEvaluator, EnvironmentEvaluator, or LaunchDarkly); provider precedence/stacking is out of scope.
  • If LaunchDarkly is used, call evaluator close() during process shutdown. close() blocks the calling thread while it flushes analytics and has no deadline of its own, so async shutdown code should run it off the runtime and bound the wait (the relay gives it a detached thread and 2 s).
  • Safety invariants must never rely on remote-flag availability.

Build and Test Expectations

Build modes:

# Provider-neutral (default): no LaunchDarkly dependency activated
cargo build -p buzz-feature-flags

# Provider-neutral environment evaluator is included in the default build
cargo test -p buzz-feature-flags --test environment_evaluator

# LaunchDarkly adapter enabled
cargo build -p buzz-feature-flags --features launchdarkly

Dependency-graph expectation checks:

# Default graph should exclude launchdarkly-server-sdk
cargo tree -p buzz-feature-flags

# Feature graph should include launchdarkly-server-sdk
cargo tree -p buzz-feature-flags --features launchdarkly

Testing expectations:

  • Run provider-neutral tests and LaunchDarkly-feature tests for this crate.
  • When buzz-db adopts flag-gated query selection, add parity tests proving old/new query paths return equivalent rows, ordering, and transactional behavior for the same inputs.

References