mirror of
https://github.com/block/buzz.git
synced 2026-11-09 10:51:42 +00:00
## Goal Give operators a way to relieve database pressure in seconds, without restarting relay pods. Today every concurrency limit is an environment variable, so changing one means a rollout. During an incident, that rollout's restart sets off a reconnect storm of its own. ## What changes **Two runtime-tunable caps.** Each is read from an integer feature flag in a new process-wide evaluation context, and re-read every 5 s: | Flag | Caps | Ceiling | |---|---|---| | `buzz.relay.concurrency.history-reads` | subscription-history reads holding a reader connection (the #9132 gate) | the existing `pool − pool/5` share, which keeps readers free for authorization | | `buzz.relay.concurrency.auth` | verified NIP-42 AUTH frames running their database checks | none (unbounded today) | - **Flags can only tighten.** A positive value is clamped to the ceiling. Zero, a negative value, no value, or a provider that never initialized leaves the gate at its ceiling, which is today's behavior. Once LaunchDarkly has delivered values, they stay in force through a later outage, so a cap tightened during an incident survives a provider blip. - **Lowering never cancels admitted work.** A lowered gate admits nothing new until the work in flight falls under the new limit. - **Visibility:** `buzz_concurrency_limit{gate}` reports the applied limit (`0` means no limit), and each change logs a line. **One resizable gate.** `HistoryReadGate` becomes `buzz_db::ConcurrencyGate`, a FIFO semaphore whose limit can move anywhere in `1..=ceiling`: - Lowering it forgets free permits and records any shortfall as debt. - A permit released while there is debt is retired instead of returned. - Raising it repays debt first, then adds permits. All of this, and admission itself, settles against the debt under one lock, so a newer limit always supersedes an older one; there is no background resize task to race. Tokio hands a released permit to a queued waiter before it runs; if that waiter is then cancelled, tokio returns the permit to the semaphore directly, so admission retires any permit it gets while debt is outstanding. The AUTH cap uses the same type. **AUTH shedding.** The permit is taken after signature verification, so only real logins queue for the database checks. The wait is bounded at 2 s. A shed login gets a bare close with no `OK`, exactly like an auth timeout. I chose this from what each client does on main today: - **`OK false`:** the desktop app retries on an unjittered 60 s ladder. Mobile parks the session until the app resumes. The agent harness fails at startup. - **Bare close:** the desktop app, mobile, the agent harness and the CLI all reconnect on their jittered, capped backoff. On the relay, the shed is visible as access line `denied/concurrency_limit` and as `buzz_auth_outcomes_total{outcome="concurrency_limit"}`. **The feature-flag provider is chosen at startup.** `BUZZ_FEATURE_FLAGS_PROVIDER` takes `static` (the default), `environment` or `launchdarkly`, with `BUZZ_LAUNCHDARKLY_SDK_KEY` and an optional `BUZZ_LAUNCHDARKLY_RELAY_PROXY_URL`: - **Exactly one provider.** The value is parsed into a single provider, so there is no stacking and no fallback, and the evaluator never changes for the life of the process. The variable sits outside the `BUZZ_FEATURE_FLAG_` namespace, so it can never be read as a flag. - **Startup failures.** Local misconfiguration fails startup: an unknown provider, a missing key, or a malformed proxy URL. LaunchDarkly being unreachable, or rejecting the key, does not. The relay starts on declared defaults, the SDK keeps connecting, and `buzz_feature_flags_ready{provider}` shows whether the provider has initialized. A revoked key must not crash-loop every pod over a tuning knob. - **Every build carries the LaunchDarkly adapter.** It stays inert unless selected, so one published image serves every deployment and each deployment's configuration pins its provider. ## Design-doc amendments (`crates/buzz-feature-flags/README.md`) - **Provider selection.** The compile-time "exactly one provider" contract becomes the startup selection above. The only-one guarantee now holds per process instead of per build. - **Startup.** Failing closed now covers local configuration only. - **Guardrails.** "Flags may choose between equivalent implementations only" gains one exception: a flag may set an operational limit as long as it can only tighten it, and shedding under that limit stays retryable. - **Evaluation context.** `EvaluationContext::for_process()` adds a process context, which LaunchDarkly sees as a single `relay`-kind context keyed `relay`. ## What got simpler - **Removed the compile-time provider contract:** the nested fixture crate (with its own 2.8k-line `Cargo.lock`), its 229-line contract test, and the exact-one `compile_error!` guards it required consumers to copy. One startup parse plus a unit test replaces them. - **One gate type serves both caps.** The history-read gate became that generic type instead of gaining a sibling. - **Net change:** about −2.3k lines. ## Review-Proven Rules applied - **#1:** shedding is retryable (`PoolTimedOut` → CLOSED for history, a bare close for AUTH) and never an empty success. Provider failures are logged and surfaced through a gauge. - **#2:** the gate settles every limit change and permit release under one lock. The table test covers lowering while idle, lowering below the count held, lowering then raising before holders leave, lowering then raising past the count held, and restoring the ceiling. - **#3:** the tests drive the production call sites: `Db::get_channel_window_with_session` under a lowered and raised cap (Postgres), and `handle_auth` under a full AUTH gate. The controller test runs the real refresh loop. - **#4:** values are clamped, waits are bounded, and the refresh runs on a fixed interval with skipped missed ticks. - **#9:** AUTH shedding sends clients down their jittered reconnect backoff rather than an unjittered or terminal path (see above). ## Review rework (`92a975c8`, `3f09cb33`) - **Keys renamed** to `buzz.relay.concurrency.*`: LaunchDarkly keys cannot be renamed later, and the flag project may be shared with other services. - **Shutdown:** LaunchDarkly's final flush has no deadline of its own, so `close()` now runs it on a detached thread and waits at most 2 s, after the audit drain. (The blocking pool would not bound it: the runtime waits for blocking tasks when it drops.) - **Refresh loop:** a panicking refresh is caught, logged and counted (`buzz_concurrency_limit_refresh_panics_total`), and the next tick retries, so the caps cannot freeze silently. - **Runbook:** `docs/operator-concurrency-caps.md`, covering how to tighten a cap and the cases in which a tightened cap returns to its ceiling (mistyped or deleted flag, pod started during a provider outage). - **AUTH test:** asserts the second login passes the gate and returns its permit; the gate permit is renamed so it no longer shares a name with the NIP-FI effect permit. Gated on a Linux workstation at `92a975c8` (`3f09cb33` is rustfmt only; `cargo fmt --check` passes there): clippy `-D warnings`, `buzz-feature-flags` (50), `buzz-relay --lib --bins` (1740), the new modules under parallel `cargo test`, and the Postgres AUTH tests (27). Each new test fails under its mutant: the refresh without `catch_unwind`, `close()` with no deadline (hangs until killed), and the AUTH handler with the gate skipped. ## Testing Re-run in full at `3c289134` after review rework, on a Linux workstation: - `cargo fmt --check`; `cargo clippy -p buzz-feature-flags -p buzz-db -p buzz-relay --all-targets --all-features -D warnings`. - `buzz-feature-flags` with and without `launchdarkly` (50 and 24 tests), `buzz-db --lib` (209), and `buzz-relay --lib --bins` (1738), plus a plain parallel `cargo test` of the new relay modules. - Postgres: the `buzz-db` routed-read and history-gate tests (55), and the `buzz-relay` AUTH and connection tests (38). - `a_cancelled_waiter_cannot_reopen_a_lowered_gate` fails with the debt check in `admit` removed. - The new infra-free relay tests are added to the CI unit lane (`Justfile` and `scripts/run-tests.sh`). - E2E against a local relay at `06c39d64`: cap gauges, sequential logins and history under cap 1, a 20-login burst (19 shed with a bare close, all reconnected), and LaunchDarkly misconfiguration and unreachability. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
211 lines
8.4 KiB
Markdown
211 lines
8.4 KiB
Markdown
# buzz-feature-flags
|
|
|
|
Provider-neutral, typed feature-flag contracts for Buzz server crates.
|
|
|
|
This crate is infrastructure-only. It defines a small API for flag evaluation,
|
|
plus a static evaluator and an optional LaunchDarkly adapter behind a compile
|
|
feature.
|
|
|
|
## Purpose
|
|
|
|
`buzz-feature-flags` centralizes typed flag evaluation so server crates can use
|
|
flags without importing vendor SDK types.
|
|
|
|
Provider-neutral types:
|
|
|
|
- `BooleanFlag`
|
|
- `IntegerFlag` (`i64`, including negative values)
|
|
- `EvaluationContext` (a `CommunityId` with an optional actor `PublicKey`, or
|
|
`for_process()` for process-wide settings such as concurrency caps)
|
|
- `FlagEvaluator` (typed `evaluate_bool` and `evaluate_int`)
|
|
- `StaticEvaluator` (always returns declared defaults)
|
|
- `EnvironmentEvaluator` (process-wide immutable environment snapshot)
|
|
- `EnvironmentDiagnostic` (sanitized malformed-value report)
|
|
|
|
LaunchDarkly support is optional and compile-gated behind
|
|
`buzz-feature-flags/launchdarkly`.
|
|
|
|
`EnvironmentEvaluator` is provider-neutral and always available (no Cargo
|
|
feature and no external dependency).
|
|
|
|
## Environment Variable Naming
|
|
|
|
`EnvironmentEvaluator` reads only the fixed namespace
|
|
`BUZZ_FEATURE_FLAG_`.
|
|
|
|
For each flag key:
|
|
|
|
- uppercase ASCII letters,
|
|
- replace each run of non-ASCII-alphanumeric characters with `_`,
|
|
- trim leading/trailing `_`,
|
|
- prepend `BUZZ_FEATURE_FLAG_`.
|
|
|
|
Example: `relay.feature.query-v2` → `BUZZ_FEATURE_FLAG_RELAY_FEATURE_QUERY_V2`.
|
|
|
|
Normalized names must be unique across declared flags. Collisions (for example
|
|
`a-b` and `a_b`) intentionally map to the same environment variable. Enforcing
|
|
uniqueness belongs in a future flag registry; this crate does not add a
|
|
registry yet.
|
|
|
|
## Relay Provider Selection
|
|
|
|
`buzz-relay` builds exactly one evaluator at startup, chosen by
|
|
`BUZZ_FEATURE_FLAGS_PROVIDER` (outside the `BUZZ_FEATURE_FLAG_` namespace, so
|
|
it can never be read as a flag):
|
|
|
|
| Value | Evaluator |
|
|
|-------|-----------|
|
|
| unset, `static` | `StaticEvaluator` |
|
|
| `environment` | `EnvironmentEvaluator::from_process_environment()` |
|
|
| `launchdarkly` | `LaunchDarklyEvaluator`, from `BUZZ_LAUNCHDARKLY_SDK_KEY` and the optional Relay Proxy URL `BUZZ_LAUNCHDARKLY_RELAY_PROXY_URL` |
|
|
|
|
The value is parsed into one provider; there is no stacking, no precedence and
|
|
no fallback from one provider to another, and the evaluator never changes for
|
|
the life of the process. Every relay build carries the LaunchDarkly adapter
|
|
(`buzz-feature-flags/launchdarkly`), which stays inert unless selected, so one
|
|
published artifact serves every deployment and the deployment's configuration
|
|
pins its provider.
|
|
|
|
The composition root lives in `crates/buzz-relay/src/feature_flags.rs`. It
|
|
constructs the evaluator once and hands consumers the shared
|
|
`Arc<dyn FlagEvaluator>`, keeping the concrete owner to drain environment
|
|
diagnostics and to call LaunchDarkly's `close()` at shutdown.
|
|
|
|
Startup fails on a local configuration error: an unknown provider,
|
|
`launchdarkly` without an SDK key, or a malformed Relay Proxy URL. It does not
|
|
fail when LaunchDarkly is unreachable: the relay waits briefly for flag data,
|
|
then serves declared defaults while the SDK keeps connecting, and
|
|
`buzz_feature_flags_ready{provider}` reports whether the provider has
|
|
initialized. Once initialized, LaunchDarkly serves its last-known values
|
|
through a later outage, so the gauge stays 1 and flag values do not revert to
|
|
defaults until the SDK reconnects and receives new data.
|
|
|
|
## `buzz-db` Boundary (approved correction)
|
|
|
|
`buzz-db` may evaluate flags internally when choosing between equivalent query
|
|
implementations. It should depend on `buzz-feature-flags` without enabling the
|
|
LaunchDarkly feature, and it should never construct or import provider types.
|
|
|
|
Concise example (query path selection only):
|
|
|
|
```rust
|
|
use buzz_core::CommunityId;
|
|
use std::sync::Arc;
|
|
|
|
use buzz_feature_flags::{EvaluationContext, FlagEvaluator, IntegerFlag};
|
|
|
|
pub struct Db {
|
|
feature_flags: Arc<dyn FlagEvaluator>,
|
|
}
|
|
|
|
impl Db {
|
|
pub async fn list_events(
|
|
&self,
|
|
community: CommunityId,
|
|
) -> anyhow::Result<Vec<EventRow>> {
|
|
let context = EvaluationContext::for_community(community);
|
|
|
|
let query_version = self.feature_flags.evaluate_int(
|
|
IntegerFlag::new("db.events.query-version", 1),
|
|
&context,
|
|
);
|
|
|
|
if query_version >= 2 {
|
|
self.list_events_v2_sql(community).await
|
|
} else {
|
|
self.list_events_v1_sql(community).await
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
SQL details are intentionally omitted in this example; the key point is that
|
|
public `Db` methods accept only domain inputs, while feature-flag evaluator
|
|
wiring remains internal to `Db` construction.
|
|
|
|
`EvaluationContext::for_actor` is appropriate only when the existing DB
|
|
operation already receives an authoritative authenticated actor for domain
|
|
behavior. Feature targeting must not add actor/pubkey parameters to otherwise
|
|
actor-free DB APIs.
|
|
|
|
For LaunchDarkly, a community evaluation always includes kind `community` plus
|
|
optional global kind `pubkey`; a process evaluation is a single kind `relay`
|
|
context keyed `relay`. Percentage rollouts must explicitly choose a
|
|
kind present in evaluation: rollouts must set `contextKind` to a kind present
|
|
in evaluation (`community` or `pubkey`) when they are intended to bucket Buzz
|
|
traffic. LaunchDarkly rollouts where configuration omits `contextKind` defaults
|
|
to `user`; Buzz does not synthesize a `user` alias, so those rollouts bucket as
|
|
zero and pick the first positive-weight variation.
|
|
|
|
## Guardrails
|
|
|
|
- Evaluation context inputs are authoritative server-resolved values
|
|
(`community`, optional `actor`), not client-supplied targeting attributes.
|
|
- Flags may choose between **equivalent** implementations only, with one
|
|
exception: an operational limit (a concurrency cap) may be set by a flag
|
|
when the flag can only *tighten* it. The consumer clamps the value to a
|
|
ceiling fixed at startup, and a missing, non-positive or unavailable value
|
|
means the ceiling. Shedding under such a limit must be retryable.
|
|
- Flags must not weaken authorization, tenant isolation, ordering guarantees,
|
|
transaction/consistency behavior, or schema invariants.
|
|
- Declared defaults choose the established safe path.
|
|
- Integer values must be validated at the consumer boundary before affecting
|
|
behavior.
|
|
|
|
## Fallback, Startup, and Lifecycle
|
|
|
|
- `StaticEvaluator` always returns each flag's declared default.
|
|
- `EnvironmentEvaluator` returns declared defaults when values are missing,
|
|
empty, invalid, non-Unicode, or the normalized flag key is unusable. Present
|
|
malformed values also produce a sanitized diagnostic; missing values do not.
|
|
- LaunchDarkly adapter returns declared defaults when a flag is missing, wrong
|
|
type, or evaluation fails.
|
|
- LaunchDarkly numbers are SDK `f64` values. Integer evaluation accepts only
|
|
mathematically integral values that convert to `i64` exactly from that `f64`
|
|
representation; otherwise it returns each flag's declared default.
|
|
- Startup chooses one evaluator from configuration (`StaticEvaluator`,
|
|
`EnvironmentEvaluator`, or LaunchDarkly); provider precedence/stacking is out
|
|
of scope.
|
|
- If LaunchDarkly is used, call evaluator `close()` during process shutdown.
|
|
`close()` blocks the calling thread while it flushes analytics and has no
|
|
deadline of its own, so async shutdown code should run it off the runtime
|
|
and bound the wait (the relay gives it a detached thread and 2 s).
|
|
- Safety invariants must never rely on remote-flag availability.
|
|
|
|
## Build and Test Expectations
|
|
|
|
Build modes:
|
|
|
|
```bash
|
|
# Provider-neutral (default): no LaunchDarkly dependency activated
|
|
cargo build -p buzz-feature-flags
|
|
|
|
# Provider-neutral environment evaluator is included in the default build
|
|
cargo test -p buzz-feature-flags --test environment_evaluator
|
|
|
|
# LaunchDarkly adapter enabled
|
|
cargo build -p buzz-feature-flags --features launchdarkly
|
|
```
|
|
|
|
Dependency-graph expectation checks:
|
|
|
|
```bash
|
|
# Default graph should exclude launchdarkly-server-sdk
|
|
cargo tree -p buzz-feature-flags
|
|
|
|
# Feature graph should include launchdarkly-server-sdk
|
|
cargo tree -p buzz-feature-flags --features launchdarkly
|
|
```
|
|
|
|
Testing expectations:
|
|
|
|
- Run provider-neutral tests and LaunchDarkly-feature tests for this crate.
|
|
- When `buzz-db` adopts flag-gated query selection, add parity tests proving
|
|
old/new query paths return equivalent rows, ordering, and transactional
|
|
behavior for the same inputs.
|
|
|
|
## References
|
|
|
|
- [ARCHITECTURE.md](../../ARCHITECTURE.md)
|
|
- [docs/multi-tenant-relay.md](../../docs/multi-tenant-relay.md)
|