mirror of
https://github.com/block/buzz.git
synced 2026-10-27 04:30:52 +00:00
8519db1532
## What this PR does Implements NIP-FI stateless enforcement at the relay WebSocket upgrade point. Every connection attempt on a NIP-FI–enabled relay must carry a valid federated identity assertion signed for the connecting Nostr pubkey; the relay validates it before the first frame is read, pairs it with NIP-42 AUTH, and enforces a bounded session lifetime without persistent state. ### Core enforcement seam - **Assertion at upgrade** (`nip_fi_upgrade.rs`): identity header is parsed with the same `extract_bearer_token` and denied with the same `http_denial` as HTTP ingress (`nip_fi_http.rs`), JWT verified (RS256/ES256/EdDSA), expiry checked, and the Nostr pubkey embedded in the sub claim must match the NIP-42 AUTH pubkey that arrives in the socket within the AUTH timeout. Denial returns exact byte contracts — `401 + WWW-Authenticate: Nostr + "authentication required\n"` for missing evidence, `403 + "evidence rejected\n"` for malformed or invalid evidence. The active HTTP/1.1 gate runs before `bind_community`, so denied upgrades pay zero DB cost. - **NIP-42 pairing** (`nip_fi_gate.rs`, `audio/handler.rs`): after the assertion is validated at upgrade, a `SessionAdmissionGate` is created with the assertion's expiry deadline and a `CancellationToken`. The gate is passed to every effect-acquiring path so no new effect permit is granted after the session expires. AUTH is verified against the assertion pubkey; a mismatched AUTH key receives `NOTICE restricted: authorization denied` and a socket close. Every other post-upgrade AUTH denial on an FI connection uses the same terminal shape — the canonical spec NOTICE on the terminal queue, then close — so frame type and close behaviour reveal nothing about private policy: ban, allowlist, or relay-membership denial → `restricted: authorization denied`; ban/allowlist/membership lookup failure or an internal verifier failure → `restricted: authorization unavailable`; an invalid NIP-42 proof → `restricted: evidence rejected`. Connections without an assertion keep today's `OK false` replies. - **Session lifetime** (`nip_fi_session.rs`): three independent deadlines bound each session — token `exp`, max assertion age from config, and relay max connection lifetime. The first to fire cancels the gate and sends the terminal denied frame. - **Effect admission at expiry** (`audio/join.rs`, `handlers/req.rs`): no new effect permit is granted after expiry. Two paths carry calculated budgets. The audio **lease resolver** holds its permit for at most `worst_case::PERMIT_HOLD` (14.5 s), and cancellation **during that resolver** reaches socket close within `worst_case::EXIT` (25 s: resolver, release of the won lease, terminal flush). Redis checkout has one total deadline and directory scripts are single-round-trip `EVAL`, which is what makes those budgets hold. When post-write fence verification fails after a landed CAS, the acquire keeps the reply and attempts an exact owner+generation cleanup on the same connection without a serving-write lease; if that cleanup fails or its reply is uncertain, the lease expires by its 30 s TTL. Neither number bounds audio close in general: other admission and teardown waits (for example the channel lookup after the resolver) and the known limitations below have no client-side deadline. REQ liveness, search and history reads race gate cancellation: a lost race drops the read, releases the permit, retires the claim through `close_if_owner`, and sends no further EVENT or EOSE. A history statement timeout ends the read with a timed-out outcome, and its owner-checked retirement (`CLOSED`, map, index and topic release) runs after the race with the permit dropped, so cancellation cannot drop it halfway. Subscription registration stays one uncancellable unit, and the permit is still taken before the claim. - **JWKS recovery** (`main.rs`): builds on main's `warm_nip_fi_jwks_snapshots` / `nip_fi_jwks_refresh_loop`. Issuers warm concurrently and the first refresh is seeded from the warm result (one interval for a warm issuer, 5 s for a cold one). A refresh that finds no live snapshot retries at the fast cadence (5 s after a warm issuer dies, doubling to a 300 s cap) instead of waiting a full interval, and `run_jwks_refresh_supervisor` restarts the loop after a panic (1 s → 60 s backoff); On shutdown it is cancelled and awaited before the drain backstop is disarmed; cancel drops an in-flight fetch and starts no further issuer, so the join stays inside the drain bound. Logs carry `issuer_index` only. ### Known limitations - **Known limitations: bounded only by the dependency.** No new effect is admitted after expiry. Two awaits that are already admitted have no client-side bound, so if a dependency stalls at expiry, teardown waits for it: - **Audio join commit:** its effect permit spans `tx.commit()`, room publication and the `event_mentions` insert. During a DB or Redis stall, audio cleanup and socket close wait for the dependency. - **REQ subscription setup:** the first subscriber to a topic sends `Subscribe` to the pubsub task through a bounded channel (capacity 4096). The send blocks only if that queue is full, which happens when the pubsub task isn't keeping up (for example during reconnect backoff or a burst). Registration holds the connection's `conn.subscriptions` lifecycle lock across the send, so a blocked send also blocks other claims on that connection, including search, and any cleanup waiting on that lock. The send itself predates this PR; holding the REQ permit across it is new here. ### Archive/join serialization `commit_participant_join` now takes a row-level write lock (`SELECT ... FOR NO KEY UPDATE`) on the channel row at step 3. This serializes all join commits against concurrent `archive_channel` calls — the archive UPDATE (`archived_at` non-key column) blocks on the row lock, closing the READ COMMITTED race on both the `Existing` and `AutoAddRequired` paths. ### HTTP/2 extended-CONNECT gate (latent path) The root WebSocket handler uses a belt-and-suspenders approach. The HTTP/1.1 path (currently the only live WebSocket shape — workspace Axum does not enable `http2`) is gated pre-extractor via the `Upgrade: websocket` + `Connection: Upgrade` header predicate. A second gate point inside the `Ok(ws)` arm is structural future-proofing: if `http2` is ever enabled, any h2 extended-CONNECT shape the extractor accepts but the h1 predicate misses is caught there. The active HTTP/1.1 fire-point sits before `bind_community` (zero DB cost on denial); the latent h2 fire-point runs after it. ### Mixed-version huddle control (rolling deploy) Owners advertise the `huddle-commit-phase` mesh capability. The ingress selects the mode after the owner stream opens: it sends the appended `RegisterPeerCommitPhase` variant, and later `CommitConfirmed`, only when the owner's exact runtime gossip record carries that flag at that point, and latches the result for the session. A successful open implies the owner's record is present, so a record that lands during acquisition is not missed. Otherwise it sends plain `RegisterPeer`, which every owner decodes. A new owner publishes a plain `RegisterPeer` once at registration (pre-commit-phase behavior) and holds a `RegisterPeerCommitPhase` peer until its confirm; duplicate or legacy confirms publish nothing. Existing wire variant indices are unchanged. ### Off-mode compatibility When `BUZZ_NIP_FI_MODE=off` (or unset), the relay reads no identity header and installs no upgrade gate, NIP-42 pairing check, or session deadline. Existing NIP-42 AUTH flows are unchanged. Huddle commit-before-publish is not FI-gated and applies in every mode, so huddle join publication timing differs from main even in Off mode. External connection cancellation (socket close, archival, revoke) now stops an in-flight liveness, search or history read at its next await; an uncancelled connection's frames and payload bytes are unchanged. ## Test coverage Every test this PR adds, or whose assertions it changes, runs in exactly one CI lane: infra-free tests in `just test-unit`, Postgres-backed tests in `postgres_tests` modules under the PostgreSQL lane. The `buzz-relay` bin tests run whole in `just test-unit`. `scripts/run-tests.sh` mirrors the effective relay unit selection for machines without nextest, except main's pre-existing `api::admin::` module. Every `buzz-relay` test fixture reads `Config` through `Config::for_test()`, which holds `NIP_FI_ENV_LOCK`, so a single-process `cargo test` run cannot observe a half-written FI environment (`fixture_config_read_waits_for_fi_env_lock`). The stub pools in the router and AUTH fixtures use a 100ms acquire timeout, which shortens a database fallthrough rather than removing it. - **Upgrade/router gate** (`router::tests`): Enforce-mode `nip_fi_*` gate tests for h1 WS upgrade, NIP-11 content negotiation, and audio ingress. Off-mode witnesses `nip_fi_off_root_passes_without_header`, `nip_fi_off_root_ignores_malformed_header`, `nip_fi_off_audio_passes_without_header`, `nip_fi_off_audio_ignores_malformed_header` prove the gate is bypassed (status outside 401/403/503, and no-header status equals malformed-header status); they do not prove a successful upgrade. - **NIP-FI units**: `nip_fi_config`, `nip_fi_upgrade`, `nip_fi_gate`, `nip_fi_session`, covering assertion contracts, NIP-42 pairing, session lifetime, and gate behavior. - **Effect barrier at cancellation** (`postgres_tests`, PostgreSQL lane): `w3_b2_req_barrier_expiry_mid_flight_blocks_subscription_registration`, `w4_b2_count_barrier_expiry_mid_flight_blocks_count_query`. - **AUTH denial uniformity** (`handlers::auth::tests`): each production-handler witness asserts the exact terminal frame, no denial on the data or ctrl channel, and cancellation, plus the unchanged Off-mode `OK false`. Unit: `fi_invalid_nip42_proof_emits_terminal_evidence_rejected`, `fi_ban_check_error_emits_terminal_authorization_unavailable`, `handle_auth_pairing_mismatch_runs_full_root_denial_path`, and `nip42_denial_class_separates_internal_failure_from_bad_evidence` (internal verifier failure → `authorization unavailable`; each client-evidence NIP-42 failure → `evidence rejected`). PostgreSQL lane: `fi_ban_denial_emits_terminal_authorization_denied` (real `community_bans` row; Off mode keeps `OK false` on ctrl, then close), `fi_relay_membership_denial_emits_terminal_authorization_denied`, `fi_allowlist_check_error_emits_terminal_authorization_unavailable`, `fi_relay_membership_check_error_emits_terminal_authorization_unavailable`, `fix_4a_allowlist_denial_with_fi_assertion_emits_canonical_restricted_frame`. - **Bounded audio exits** (`audio::handler::tests`, `just test-unit`): `check_cancel_exit_with_never_ready_sink_returns_within_flush_budget` drives the production `check_cancel!` fence with a queued FI denial into a never-ready sink and requires it to return within `WS_TERMINAL_FLUSH_TIMEOUT`; `challenge_send_to_never_ready_sink_exits_within_flush_budget_on_cancel` requires the NIP-42 challenge send to yield to cancellation and exit within the same budget; every pre-`send_loop` audio exit write goes through the same `connection::send_exit_frames_bounded` helper. - **Audio NIP-42 classification** (`audio::handler::tests`, `just test-unit`): `audio_bad_nip42_proof_with_fi_emits_one_evidence_rejected_then_cancels` requires exactly one canonical `restricted: evidence rejected` frame and cancellation; `audio_bad_nip42_proof_off_mode_keeps_auth_failed_frame` pins the Off-mode frame. - **Effect permit before pre-auth writes** (PostgreSQL lane): `handlers::auth::tests::postgres_tests::expiry_before_nip_oa_materialize_writes_no_rows_and_commits_no_auth` expires the session just before NIP-OA materialize and asserts no `users` or agent-owner row and no committed auth. - **Community admission** (`just test-unit`): `state::tests::on_not_run_runs_once_on_each_deny_arm_and_never_on_admit`. - **Auth metrics contract**: `connection::tests::auth_lifecycle_reconciles_every_terminal_and_never_leaks_gauge` records every terminal including `PairingMismatch`; `metrics::contract_tests::*` derive the series count from `AuthOutcome::ALL`. - **JWKS** (`main.rs`): `hard_dead_warm_issuer_resets_to_fast_cadence` and `jwks_refresh_success_waits_interval_failure_uses_fast_backoff` drive the real loop with paused Tokio time; `cold_issuer_*` cover the backoff helper; `f1_supervisor_loop_drives_recovery_and_restores_admission` runs `run_jwks_refresh_supervisor` over a real `ProductionJwksSource` through injected panic → restart → hard-dead → 5 s retry → recovery, then checks `key_set` admission and that the first post-restart fetch follows the cold cadence (every issuer restarts cold); `jwks_refresh_cancel_drops_in_flight_fetch_and_skips_remaining_issuers` cancels during a pending fetch with a second issuer due. - **Bounded lease acquire** (`audio::handler::tests::postgres_tests`, PostgreSQL lane): `failed_post_cas_verification_is_fenced_and_discards_the_won_lease` requires a won-then-unverified lease to return fenced and be deleted by exact owner+generation; `stalled_lease_sql_and_stalled_exit_release_close_within_exit_bound` stalls lease SQL and the exit release and requires socket close within `worst_case::EXIT`. `audio::join::tests` pins `PERMIT_HOLD` at 14.5 s and `EXIT` at 25 s. - **REQ/search cancellation** (`handlers::req::tests`, `just test-unit`): `expiry_during_stalled_history_read_releases_permit_and_claim` and `expiry_during_stalled_search_read_releases_permit_and_claim` stall the read inside production `handle_req`, expire the gate, and require `expire()` to return within 2 s, an empty subscription map, and no EVENT or EOSE. - **History timeout retirement** (`handlers::req::tests::postgres_tests`, PostgreSQL lane): `history_timeout_retirement_survives_expiry` drives a two-channel history REQ into a real statement timeout, pauses retirement after the index removal and before the topic release, expires the gate, then resumes. It requires the map and index entries gone and both channel topic refcounts at zero. - **Archive/join serialization** (PostgreSQL lane): `f2a_archived_channel_rejects_existing_join`, `f2b_join_for_update_blocks_concurrent_archive_existing_path`, `f2c_join_for_update_blocks_concurrent_archive_auto_add_path`. - **Huddle room and handler**: `audio::room::tests` pending-peer tests, including `roster_revisions_are_ordered_and_snapshot_is_authoritative`; `fix_b_w1_w3_handler_pre_commit_cancel_emits_no_delta`, `fix_b_w2_success_emits_exactly_one_delta_with_monotone_revision`. PostgreSQL lane: `b1_cross_pod_bootstrap_precedes_buffered_owner_delta`, `commit_confirm_timeout_at_seam_triggers_teardown_on_stalled_owner_stream`, `confirm_failure_combined_outcome_no_orphan_join_exactly_one_48102`, `confirm_send_pending_expiry_terminates_client_with_denial_within_1s`, `c1_handler_ingress_sends_only_base_frames_to_legacy_owner`, `f7a_cross_pod_joined_payload_includes_owner_pod_peers`. - **Mixed-version control** (`audio::join::tests`): `shared_control_variants_are_byte_identical_to_base_wire` (against frozen `audio::join::base_wire`), `c1_new_ingress_speaks_base_wire_to_legacy_owner`, `c2_base_ingress_register_publishes_once_on_new_owner`, `c3_capable_owner_record_selects_commit_phase_hold_back`, `capability_record_installed_during_open_selects_commit_phase`, `duplicate_or_legacy_commit_confirmed_is_a_no_op`, `fix_b_commit_confirmed_arm_commits_peer_and_broadcasts_joined`, `pending_remote_stream_close_emits_no_joined_no_left`. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>