jerry/buzz - buzz - Gitea: Git with a cup of tea

jerry/buzz

Fork 0

mirror of https://github.com/block/buzz.git synced 2026-10-27 04:30:52 +00:00

Commit Graph

Author	SHA1	Message	Date
Luke Tornquist	7ce422417b	fix(mesh): expire dead peer records and instrument mesh boot (#8503 ) ## Summary Relay pods can stall in the `mesh_boot` startup step, and the mesh peer table grows without limit. One prod pod held 7,793 peer records. Most of those records belong to pods that no longer exist. Cause. Every boot gets a fresh runtime id, so a restarted pod leaves its old record behind. Nothing ever removed records, and scuttlebutt gossip kept spreading the dead ones around the fleet. Each reconcile pass, including the first one inside `boot_mesh`, then dials every record one at a time. This PR makes dead records expire. It also adds the measurements we need to see the next stall directly. ### Peer-table lifetime (`crates/buzz-relay-mesh/src/membership.rs`) - A peer record expires 5 minutes after its last heartbeat (`DEFAULT_PEER_RECORD_TTL`). For comparison, live runtimes refresh every 2s, and the Redis ready registry expires a silent runtime after 45s. - What happens to expired records: - They are rejected on arrival, so peers that haven't been upgraded can't re-add them. - They are left out of digests, deltas, and dial candidates. - They are pruned at the start of each reconcile pass. - Between prunes, they are not admitted (`has_peer`) and not routed (`peers`, `peer_has_capability`). - An expired entry that hasn't been pruned yet counts as absent, so any fresh record replaces it whatever its version. That lets a live peer's v1 Redis hint re-admit it before the next prune. - The TTL never prunes or hides a peer with a live connection, even if its heartbeat is stale. Its gossip can stop arriving while the transport stays up: a dead control stream, or a clock behind by more than the TTL. The connection proves the runtime is alive. Its stale record is still kept out of digests and dials. (The existing phi filter in `peers()` still drops a connected peer once its gossip stalls; this PR doesn't change that.) - `Connected` tracks the transport map. Because the TTL exemption trusts that flag, `install_peer` and `remove_peer` write it while holding the peers lock, so a connection that closes during install always ends `Disconnected`. A dial's `Connecting`/`Disconnected` writes are skipped when a connection is already installed, so a failed outbound dial can't overwrite a live inbound connection. Each reconcile pass also re-syncs the state from the transport map. - Gossip no longer sets connection state. Before, any gossiped record was marked `Connected`, even if this pod had never dialed it. That inflated the `/_mesh` counts. Now a record learned from gossip or the ready registry enters as `Disconnected`. Only the real connect and disconnect paths change the state. - Before building its candidate list, reconcile now skips peers that are already connected. A pass dials serially, and a dial to a dead runtime can take about a minute, so `dial_peer` re-reads the record before every address attempt. It stops once the record has expired or started draining, or once a connection has landed. ### Instrumentation - `mesh_boot` sub-steps use the existing startup-step gauges: `mesh_bind`, `mesh_publish`, and `mesh_initial_reconcile`. A pod stuck in `mesh_boot` now also shows which await it is stuck on. - New metrics (`crates/buzz-relay-mesh/src/telemetry.rs`): \| Metric \| Shows \| \|---\|---\| \| `buzz_mesh_dial_total{outcome}` / `buzz_mesh_dial_duration_seconds` \| Cost of each dial \| \| `buzz_mesh_reconcile_pass_seconds` \| Full pass time \| \| `buzz_mesh_reconcile_candidates` \| Dials planned for this pass (set after the registry scan) \| \| `buzz_mesh_peer_records{state,draining}` \| Table size \| \| `buzz_mesh_peer_record_max_age_seconds` \| Oldest heartbeat \| \| `buzz_mesh_peer_records_expired_total{reason=pruned\\|rejected}` \| Expiry activity \| - The dial and pass histograms get buckets up to 600s. The default `_seconds` buckets stop at 5s, which would hide exactly the stall these metrics are meant to show. ### CI The `buzz-relay-mesh` tests and the `buzz-relay` `mesh_boot::tests` did not run in any CI lane. This PR adds both to `just test-unit`, and adds the matching stanzas to `scripts/run-tests.sh`. ### What got simpler - The peer table now has a defined size, about one record per live pod, where before it grew forever. - One wrong state write was deleted (gossip marking records `Connected`). - Two test suites that silently never ran now run. ### Rollout notes - Rejecting expired records works one pod at a time. During a rollout, pods that haven't been upgraded keep gossiping dead records. Upgraded pods reject them, which shows up as `reason="rejected"` on the expired counter. - A pod whose clock runs more than 5 minutes behind has all of its gossip rejected. It stays reachable through its Redis ready record and any live connection. A `reason="rejected"` count that keeps climbing outside a rollout points to that skew. - Upgraded pods leave expired records out of their digests. Pods that haven't been upgraded therefore resend the whole ghost set (about 7.8k records, roughly 1 MB) on every 2s gossip tick, on each old↔new connection. So during the rollout, `reason="rejected"` will climb by about the ghost count per tick per connection. This is expected and stops when the rollout finishes. ### Not in this PR The boot dial itself is still unbounded: it runs serially, has no per-dial timeout, and dials gossip-learned records. Bounding it changes how a pod joins the mesh, so it goes in a separate PR, and these metrics will measure it. ### Related issue None found. ### Testing - New tests: - `membership::tests`: expired gossip is rejected; prune drops only expired records; prune keeps a connected peer with a stale heartbeat; expired unconnected records are neither admitted nor routed before a prune; a fresh record replaces an expired, unpruned entry whatever its version; expired records are neither dialed nor gossiped before a prune; gossip never sets connection state; table stats. - `runtime::tests::reconcile_pass_publishes_dial_and_peer_table_metrics` - `runtime::tests::reconcile_pass_prunes_expired_peer_records` (binds the prune to the reconcile pass) - `runtime::tests::dial_skips_candidate_that_expired_after_the_pass_snapshot` - `runtime::tests::finishing_dial_does_not_overwrite_a_connection_that_landed` - `runtime::tests::connection_closed_during_install_ends_disconnected_and_expires` - `runtime::tests::reconcile_pass_resyncs_connection_state_from_transport` - `mesh_boot::tests::mesh_boot_sub_steps_name_the_step_that_failed` - Blox at `e06f55a4` (tree `866add1c`): `just fmt-check`, `just clippy`, `buzz-relay-mesh` nextest 3x, and the `buzz-relay` `mesh_boot`/`startup_steps`/`metrics` tests all pass. Three of the new tests fail against a mutant that removes the fix they pin (dial ignores expiry; dial state writes ignore the transport map; no reconcile resync). --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-26 09:47:30 -04:00
Will Pfleger	8519db1532	feat(buzz-relay): NIP-FI stateless enforcement (S3) — upgrade gate, NIP-42 pairing, session lifetime, JWKS warm (#7224 ) ## What this PR does Implements NIP-FI stateless enforcement at the relay WebSocket upgrade point. Every connection attempt on a NIP-FI–enabled relay must carry a valid federated identity assertion signed for the connecting Nostr pubkey; the relay validates it before the first frame is read, pairs it with NIP-42 AUTH, and enforces a bounded session lifetime without persistent state. ### Core enforcement seam - Assertion at upgrade (`nip_fi_upgrade.rs`): identity header is parsed with the same `extract_bearer_token` and denied with the same `http_denial` as HTTP ingress (`nip_fi_http.rs`), JWT verified (RS256/ES256/EdDSA), expiry checked, and the Nostr pubkey embedded in the sub claim must match the NIP-42 AUTH pubkey that arrives in the socket within the AUTH timeout. Denial returns exact byte contracts — `401 + WWW-Authenticate: Nostr + "authentication required\n"` for missing evidence, `403 + "evidence rejected\n"` for malformed or invalid evidence. The active HTTP/1.1 gate runs before `bind_community`, so denied upgrades pay zero DB cost. - NIP-42 pairing (`nip_fi_gate.rs`, `audio/handler.rs`): after the assertion is validated at upgrade, a `SessionAdmissionGate` is created with the assertion's expiry deadline and a `CancellationToken`. The gate is passed to every effect-acquiring path so no new effect permit is granted after the session expires. AUTH is verified against the assertion pubkey; a mismatched AUTH key receives `NOTICE restricted: authorization denied` and a socket close. Every other post-upgrade AUTH denial on an FI connection uses the same terminal shape — the canonical spec NOTICE on the terminal queue, then close — so frame type and close behaviour reveal nothing about private policy: ban, allowlist, or relay-membership denial → `restricted: authorization denied`; ban/allowlist/membership lookup failure or an internal verifier failure → `restricted: authorization unavailable`; an invalid NIP-42 proof → `restricted: evidence rejected`. Connections without an assertion keep today's `OK false` replies. - Session lifetime (`nip_fi_session.rs`): three independent deadlines bound each session — token `exp`, max assertion age from config, and relay max connection lifetime. The first to fire cancels the gate and sends the terminal denied frame. - Effect admission at expiry (`audio/join.rs`, `handlers/req.rs`): no new effect permit is granted after expiry. Two paths carry calculated budgets. The audio lease resolver holds its permit for at most `worst_case::PERMIT_HOLD` (14.5 s), and cancellation during that resolver reaches socket close within `worst_case::EXIT` (25 s: resolver, release of the won lease, terminal flush). Redis checkout has one total deadline and directory scripts are single-round-trip `EVAL`, which is what makes those budgets hold. When post-write fence verification fails after a landed CAS, the acquire keeps the reply and attempts an exact owner+generation cleanup on the same connection without a serving-write lease; if that cleanup fails or its reply is uncertain, the lease expires by its 30 s TTL. Neither number bounds audio close in general: other admission and teardown waits (for example the channel lookup after the resolver) and the known limitations below have no client-side deadline. REQ liveness, search and history reads race gate cancellation: a lost race drops the read, releases the permit, retires the claim through `close_if_owner`, and sends no further EVENT or EOSE. A history statement timeout ends the read with a timed-out outcome, and its owner-checked retirement (`CLOSED`, map, index and topic release) runs after the race with the permit dropped, so cancellation cannot drop it halfway. Subscription registration stays one uncancellable unit, and the permit is still taken before the claim. - JWKS recovery (`main.rs`): builds on main's `warm_nip_fi_jwks_snapshots` / `nip_fi_jwks_refresh_loop`. Issuers warm concurrently and the first refresh is seeded from the warm result (one interval for a warm issuer, 5 s for a cold one). A refresh that finds no live snapshot retries at the fast cadence (5 s after a warm issuer dies, doubling to a 300 s cap) instead of waiting a full interval, and `run_jwks_refresh_supervisor` restarts the loop after a panic (1 s → 60 s backoff); On shutdown it is cancelled and awaited before the drain backstop is disarmed; cancel drops an in-flight fetch and starts no further issuer, so the join stays inside the drain bound. Logs carry `issuer_index` only. ### Known limitations - Known limitations: bounded only by the dependency. No new effect is admitted after expiry. Two awaits that are already admitted have no client-side bound, so if a dependency stalls at expiry, teardown waits for it: - Audio join commit: its effect permit spans `tx.commit()`, room publication and the `event_mentions` insert. During a DB or Redis stall, audio cleanup and socket close wait for the dependency. - REQ subscription setup: the first subscriber to a topic sends `Subscribe` to the pubsub task through a bounded channel (capacity 4096). The send blocks only if that queue is full, which happens when the pubsub task isn't keeping up (for example during reconnect backoff or a burst). Registration holds the connection's `conn.subscriptions` lifecycle lock across the send, so a blocked send also blocks other claims on that connection, including search, and any cleanup waiting on that lock. The send itself predates this PR; holding the REQ permit across it is new here. ### Archive/join serialization `commit_participant_join` now takes a row-level write lock (`SELECT ... FOR NO KEY UPDATE`) on the channel row at step 3. This serializes all join commits against concurrent `archive_channel` calls — the archive UPDATE (`archived_at` non-key column) blocks on the row lock, closing the READ COMMITTED race on both the `Existing` and `AutoAddRequired` paths. ### HTTP/2 extended-CONNECT gate (latent path) The root WebSocket handler uses a belt-and-suspenders approach. The HTTP/1.1 path (currently the only live WebSocket shape — workspace Axum does not enable `http2`) is gated pre-extractor via the `Upgrade: websocket` + `Connection: Upgrade` header predicate. A second gate point inside the `Ok(ws)` arm is structural future-proofing: if `http2` is ever enabled, any h2 extended-CONNECT shape the extractor accepts but the h1 predicate misses is caught there. The active HTTP/1.1 fire-point sits before `bind_community` (zero DB cost on denial); the latent h2 fire-point runs after it. ### Mixed-version huddle control (rolling deploy) Owners advertise the `huddle-commit-phase` mesh capability. The ingress selects the mode after the owner stream opens: it sends the appended `RegisterPeerCommitPhase` variant, and later `CommitConfirmed`, only when the owner's exact runtime gossip record carries that flag at that point, and latches the result for the session. A successful open implies the owner's record is present, so a record that lands during acquisition is not missed. Otherwise it sends plain `RegisterPeer`, which every owner decodes. A new owner publishes a plain `RegisterPeer` once at registration (pre-commit-phase behavior) and holds a `RegisterPeerCommitPhase` peer until its confirm; duplicate or legacy confirms publish nothing. Existing wire variant indices are unchanged. ### Off-mode compatibility When `BUZZ_NIP_FI_MODE=off` (or unset), the relay reads no identity header and installs no upgrade gate, NIP-42 pairing check, or session deadline. Existing NIP-42 AUTH flows are unchanged. Huddle commit-before-publish is not FI-gated and applies in every mode, so huddle join publication timing differs from main even in Off mode. External connection cancellation (socket close, archival, revoke) now stops an in-flight liveness, search or history read at its next await; an uncancelled connection's frames and payload bytes are unchanged. ## Test coverage Every test this PR adds, or whose assertions it changes, runs in exactly one CI lane: infra-free tests in `just test-unit`, Postgres-backed tests in `postgres_tests` modules under the PostgreSQL lane. The `buzz-relay` bin tests run whole in `just test-unit`. `scripts/run-tests.sh` mirrors the effective relay unit selection for machines without nextest, except main's pre-existing `api::admin::` module. Every `buzz-relay` test fixture reads `Config` through `Config::for_test()`, which holds `NIP_FI_ENV_LOCK`, so a single-process `cargo test` run cannot observe a half-written FI environment (`fixture_config_read_waits_for_fi_env_lock`). The stub pools in the router and AUTH fixtures use a 100ms acquire timeout, which shortens a database fallthrough rather than removing it. - Upgrade/router gate (`router::tests`): Enforce-mode `nip_fi_` gate tests for h1 WS upgrade, NIP-11 content negotiation, and audio ingress. Off-mode witnesses `nip_fi_off_root_passes_without_header`, `nip_fi_off_root_ignores_malformed_header`, `nip_fi_off_audio_passes_without_header`, `nip_fi_off_audio_ignores_malformed_header` prove the gate is bypassed (status outside 401/403/503, and no-header status equals malformed-header status); they do not prove a successful upgrade. - NIP-FI units: `nip_fi_config`, `nip_fi_upgrade`, `nip_fi_gate`, `nip_fi_session`, covering assertion contracts, NIP-42 pairing, session lifetime, and gate behavior. - Effect barrier at cancellation* (`postgres_tests`, PostgreSQL lane): `w3_b2_req_barrier_expiry_mid_flight_blocks_subscription_registration`, `w4_b2_count_barrier_expiry_mid_flight_blocks_count_query`. - AUTH denial uniformity (`handlers::auth::tests`): each production-handler witness asserts the exact terminal frame, no denial on the data or ctrl channel, and cancellation, plus the unchanged Off-mode `OK false`. Unit: `fi_invalid_nip42_proof_emits_terminal_evidence_rejected`, `fi_ban_check_error_emits_terminal_authorization_unavailable`, `handle_auth_pairing_mismatch_runs_full_root_denial_path`, and `nip42_denial_class_separates_internal_failure_from_bad_evidence` (internal verifier failure → `authorization unavailable`; each client-evidence NIP-42 failure → `evidence rejected`). PostgreSQL lane: `fi_ban_denial_emits_terminal_authorization_denied` (real `community_bans` row; Off mode keeps `OK false` on ctrl, then close), `fi_relay_membership_denial_emits_terminal_authorization_denied`, `fi_allowlist_check_error_emits_terminal_authorization_unavailable`, `fi_relay_membership_check_error_emits_terminal_authorization_unavailable`, `fix_4a_allowlist_denial_with_fi_assertion_emits_canonical_restricted_frame`. - Bounded audio exits (`audio::handler::tests`, `just test-unit`): `check_cancel_exit_with_never_ready_sink_returns_within_flush_budget` drives the production `check_cancel!` fence with a queued FI denial into a never-ready sink and requires it to return within `WS_TERMINAL_FLUSH_TIMEOUT`; `challenge_send_to_never_ready_sink_exits_within_flush_budget_on_cancel` requires the NIP-42 challenge send to yield to cancellation and exit within the same budget; every pre-`send_loop` audio exit write goes through the same `connection::send_exit_frames_bounded` helper. - Audio NIP-42 classification (`audio::handler::tests`, `just test-unit`): `audio_bad_nip42_proof_with_fi_emits_one_evidence_rejected_then_cancels` requires exactly one canonical `restricted: evidence rejected` frame and cancellation; `audio_bad_nip42_proof_off_mode_keeps_auth_failed_frame` pins the Off-mode frame. - Effect permit before pre-auth writes (PostgreSQL lane): `handlers::auth::tests::postgres_tests::expiry_before_nip_oa_materialize_writes_no_rows_and_commits_no_auth` expires the session just before NIP-OA materialize and asserts no `users` or agent-owner row and no committed auth. - Community admission (`just test-unit`): `state::tests::on_not_run_runs_once_on_each_deny_arm_and_never_on_admit`. - Auth metrics contract: `connection::tests::auth_lifecycle_reconciles_every_terminal_and_never_leaks_gauge` records every terminal including `PairingMismatch`; `metrics::contract_tests::` derive the series count from `AuthOutcome::ALL`. - JWKS* (`main.rs`): `hard_dead_warm_issuer_resets_to_fast_cadence` and `jwks_refresh_success_waits_interval_failure_uses_fast_backoff` drive the real loop with paused Tokio time; `cold_issuer_` cover the backoff helper; `f1_supervisor_loop_drives_recovery_and_restores_admission` runs `run_jwks_refresh_supervisor` over a real `ProductionJwksSource` through injected panic → restart → hard-dead → 5 s retry → recovery, then checks `key_set` admission and that the first post-restart fetch follows the cold cadence (every issuer restarts cold); `jwks_refresh_cancel_drops_in_flight_fetch_and_skips_remaining_issuers` cancels during a pending fetch with a second issuer due. - Bounded lease acquire* (`audio::handler::tests::postgres_tests`, PostgreSQL lane): `failed_post_cas_verification_is_fenced_and_discards_the_won_lease` requires a won-then-unverified lease to return fenced and be deleted by exact owner+generation; `stalled_lease_sql_and_stalled_exit_release_close_within_exit_bound` stalls lease SQL and the exit release and requires socket close within `worst_case::EXIT`. `audio::join::tests` pins `PERMIT_HOLD` at 14.5 s and `EXIT` at 25 s. - REQ/search cancellation (`handlers::req::tests`, `just test-unit`): `expiry_during_stalled_history_read_releases_permit_and_claim` and `expiry_during_stalled_search_read_releases_permit_and_claim` stall the read inside production `handle_req`, expire the gate, and require `expire()` to return within 2 s, an empty subscription map, and no EVENT or EOSE. - History timeout retirement (`handlers::req::tests::postgres_tests`, PostgreSQL lane): `history_timeout_retirement_survives_expiry` drives a two-channel history REQ into a real statement timeout, pauses retirement after the index removal and before the topic release, expires the gate, then resumes. It requires the map and index entries gone and both channel topic refcounts at zero. - Archive/join serialization (PostgreSQL lane): `f2a_archived_channel_rejects_existing_join`, `f2b_join_for_update_blocks_concurrent_archive_existing_path`, `f2c_join_for_update_blocks_concurrent_archive_auto_add_path`. - Huddle room and handler: `audio::room::tests` pending-peer tests, including `roster_revisions_are_ordered_and_snapshot_is_authoritative`; `fix_b_w1_w3_handler_pre_commit_cancel_emits_no_delta`, `fix_b_w2_success_emits_exactly_one_delta_with_monotone_revision`. PostgreSQL lane: `b1_cross_pod_bootstrap_precedes_buffered_owner_delta`, `commit_confirm_timeout_at_seam_triggers_teardown_on_stalled_owner_stream`, `confirm_failure_combined_outcome_no_orphan_join_exactly_one_48102`, `confirm_send_pending_expiry_terminates_client_with_denial_within_1s`, `c1_handler_ingress_sends_only_base_frames_to_legacy_owner`, `f7a_cross_pod_joined_payload_includes_owner_pod_peers`. - Mixed-version control (`audio::join::tests`): `shared_control_variants_are_byte_identical_to_base_wire` (against frozen `audio::join::base_wire`), `c1_new_ingress_speaks_base_wire_to_legacy_owner`, `c2_base_ingress_register_publishes_once_on_new_owner`, `c3_capable_owner_record_selects_commit_phase_hold_back`, `capability_record_installed_during_open_selects_commit_phase`, `duplicate_or_legacy_commit_confirmed_is_a_no_op`, `fix_b_commit_confirmed_arm_commits_peer_and_broadcasts_joined`, `pending_remote_stream_close_emits_no_joined_no_left`. --------- Signed-off-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz> Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>	2026-09-29 14:27:40 -04:00
Tyler	ccb021d713	Relay mesh: cross-pod tunnel + huddle transport (buzz-relay-mesh) (#1670 ) Signed-off-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co> Signed-off-by: Tyler Longwell <tlongwell@squareup.com> Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com> Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1cc3ha7z055mu0rwwu7806t2wt8mj3pvu0uv5mfp2c50dahaqhczshdalg6 <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>	2026-07-14 16:56:48 -04:00

Author

SHA1

Message

Date

Luke Tornquist

7ce422417b

fix(mesh): expire dead peer records and instrument mesh boot (#8503 )

## Summary

Relay pods can stall in the `mesh_boot` startup step, and the mesh peer
table grows without limit. One prod pod held 7,793 peer records. Most of
those records belong to pods that no longer exist.

**Cause.** Every boot gets a fresh runtime id, so a restarted pod leaves
its old record behind. Nothing ever removed records, and scuttlebutt
gossip kept spreading the dead ones around the fleet. Each reconcile
pass, including the first one inside `boot_mesh`, then dials every
record one at a time.

This PR makes dead records expire. It also adds the measurements we need
to see the next stall directly.

### Peer-table lifetime (`crates/buzz-relay-mesh/src/membership.rs`)
- A peer record expires 5 minutes after its last heartbeat
(`DEFAULT_PEER_RECORD_TTL`). For comparison, live runtimes refresh every
2s, and the Redis ready registry expires a silent runtime after 45s.
- What happens to expired records:
- They are **rejected on arrival**, so peers that haven't been upgraded
can't re-add them.
- They are **left out** of digests, deltas, and dial candidates.
- They are **pruned** at the start of each reconcile pass.
- Between prunes, they are not admitted (`has_peer`) and not routed
(`peers`, `peer_has_capability`).
- An expired entry that hasn't been pruned yet counts as absent, so any
fresh record replaces it whatever its version. That lets a live peer's
v1 Redis hint re-admit it before the next prune.
- **The TTL never prunes or hides a peer with a live connection**, even
if its heartbeat is stale. Its gossip can stop arriving while the
transport stays up: a dead control stream, or a clock behind by more
than the TTL. The connection proves the runtime is alive. Its stale
record is still kept out of digests and dials. (The existing phi filter
in `peers()` still drops a connected peer once its gossip stalls; this
PR doesn't change that.)
- **`Connected` tracks the transport map.** Because the TTL exemption
trusts that flag, `install_peer` and `remove_peer` write it while
holding the peers lock, so a connection that closes during install
always ends `Disconnected`. A dial's `Connecting`/`Disconnected` writes
are skipped when a connection is already installed, so a failed outbound
dial can't overwrite a live inbound connection. Each reconcile pass also
re-syncs the state from the transport map.
- **Gossip no longer sets connection state.** Before, any gossiped
record was marked `Connected`, even if this pod had never dialed it.
That inflated the `/_mesh` counts. Now a record learned from gossip or
the ready registry enters as `Disconnected`. Only the real connect and
disconnect paths change the state.
- Before building its candidate list, reconcile now skips peers that are
already connected. A pass dials serially, and a dial to a dead runtime
can take about a minute, so `dial_peer` re-reads the record before every
address attempt. It stops once the record has expired or started
draining, or once a connection has landed.

### Instrumentation
- **`mesh_boot` sub-steps** use the existing startup-step gauges:
`mesh_bind`, `mesh_publish`, and `mesh_initial_reconcile`. A pod stuck
in `mesh_boot` now also shows which await it is stuck on.
- **New metrics** (`crates/buzz-relay-mesh/src/telemetry.rs`):

| Metric | Shows |
|---|---|
| `buzz_mesh_dial_total{outcome}` / `buzz_mesh_dial_duration_seconds` |
Cost of each dial |
| `buzz_mesh_reconcile_pass_seconds` | Full pass time |
| `buzz_mesh_reconcile_candidates` | Dials planned for this pass (set
after the registry scan) |
| `buzz_mesh_peer_records{state,draining}` | Table size |
| `buzz_mesh_peer_record_max_age_seconds` | Oldest heartbeat |
| `buzz_mesh_peer_records_expired_total{reason=pruned\|rejected}` |
Expiry activity |

- The dial and pass histograms get buckets up to 600s. The default
`_seconds` buckets stop at 5s, which would hide exactly the stall these
metrics are meant to show.

### CI
The `buzz-relay-mesh` tests and the `buzz-relay` `mesh_boot::tests` did
not run in any CI lane. This PR adds both to `just test-unit`, and adds
the matching stanzas to `scripts/run-tests.sh`.

### What got simpler
- The peer table now has a defined size, about one record per live pod,
where before it grew forever.
- One wrong state write was deleted (gossip marking records
`Connected`).
- Two test suites that silently never ran now run.

### Rollout notes
- Rejecting expired records works one pod at a time. During a rollout,
pods that haven't been upgraded keep gossiping dead records. Upgraded
pods reject them, which shows up as `reason="rejected"` on the expired
counter.
- A pod whose clock runs more than 5 minutes behind has all of its
gossip rejected. It stays reachable through its Redis ready record and
any live connection. A `reason="rejected"` count that keeps climbing
outside a rollout points to that skew.
- Upgraded pods leave expired records out of their digests. Pods that
haven't been upgraded therefore resend the whole ghost set (about 7.8k
records, roughly 1 MB) on every 2s gossip tick, on each old↔new
connection. So during the rollout, `reason="rejected"` will climb by
about the ghost count per tick per connection. This is expected and
stops when the rollout finishes.

### Not in this PR
The boot dial itself is still unbounded: it runs serially, has no
per-dial timeout, and dials gossip-learned records. Bounding it changes
how a pod joins the mesh, so it goes in a separate PR, and these metrics
will measure it.

### Related issue
None found.

### Testing
- New tests:
- `membership::tests`: expired gossip is rejected; prune drops only
expired records; prune keeps a connected peer with a stale heartbeat;
expired unconnected records are neither admitted nor routed before a
prune; a fresh record replaces an expired, unpruned entry whatever its
version; expired records are neither dialed nor gossiped before a prune;
gossip never sets connection state; table stats.
- `runtime::tests::reconcile_pass_publishes_dial_and_peer_table_metrics`
- `runtime::tests::reconcile_pass_prunes_expired_peer_records` (binds
the prune to the reconcile pass)
-
`runtime::tests::dial_skips_candidate_that_expired_after_the_pass_snapshot`
-
`runtime::tests::finishing_dial_does_not_overwrite_a_connection_that_landed`
-
`runtime::tests::connection_closed_during_install_ends_disconnected_and_expires`
-
`runtime::tests::reconcile_pass_resyncs_connection_state_from_transport`
- `mesh_boot::tests::mesh_boot_sub_steps_name_the_step_that_failed`
- Blox at `e06f55a4` (tree `866add1c`): `just fmt-check`, `just clippy`,
`buzz-relay-mesh` nextest 3x, and the `buzz-relay`
`mesh_boot`/`startup_steps`/`metrics` tests all pass. Three of the new
tests fail against a mutant that removes the fix they pin (dial ignores
expiry; dial state writes ignore the transport map; no reconcile
resync).

---------

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>

2026-10-26 09:47:30 -04:00

Will Pfleger

8519db1532

feat(buzz-relay): NIP-FI stateless enforcement (S3) — upgrade gate, NIP-42 pairing, session lifetime, JWKS warm (#7224 )

## What this PR does

Implements NIP-FI stateless enforcement at the relay WebSocket upgrade
point. Every connection attempt on a NIP-FI–enabled relay must carry a
valid federated identity assertion signed for the connecting Nostr
pubkey; the relay validates it before the first frame is read, pairs it
with NIP-42 AUTH, and enforces a bounded session lifetime without
persistent state.

### Core enforcement seam

- **Assertion at upgrade** (`nip_fi_upgrade.rs`): identity header is
parsed with the same `extract_bearer_token` and denied with the same
`http_denial` as HTTP ingress (`nip_fi_http.rs`), JWT verified
(RS256/ES256/EdDSA), expiry checked, and the Nostr pubkey embedded in
the sub claim must match the NIP-42 AUTH pubkey that arrives in the
socket within the AUTH timeout. Denial returns exact byte contracts —
`401 + WWW-Authenticate: Nostr + "authentication required\n"` for
missing evidence, `403 + "evidence rejected\n"` for malformed or invalid
evidence. The active HTTP/1.1 gate runs before `bind_community`, so
denied upgrades pay zero DB cost.
- **NIP-42 pairing** (`nip_fi_gate.rs`, `audio/handler.rs`): after the
assertion is validated at upgrade, a `SessionAdmissionGate` is created
with the assertion's expiry deadline and a `CancellationToken`. The gate
is passed to every effect-acquiring path so no new effect permit is
granted after the session expires. AUTH is verified against the
assertion pubkey; a mismatched AUTH key receives `NOTICE restricted:
authorization denied` and a socket close. Every other post-upgrade AUTH
denial on an FI connection uses the same terminal shape — the canonical
spec NOTICE on the terminal queue, then close — so frame type and close
behaviour reveal nothing about private policy: ban, allowlist, or
relay-membership denial → `restricted: authorization denied`;
ban/allowlist/membership lookup failure or an internal verifier failure
→ `restricted: authorization unavailable`; an invalid NIP-42 proof →
`restricted: evidence rejected`. Connections without an assertion keep
today's `OK false` replies.
- **Session lifetime** (`nip_fi_session.rs`): three independent
deadlines bound each session — token `exp`, max assertion age from
config, and relay max connection lifetime. The first to fire cancels the
gate and sends the terminal denied frame.
- **Effect admission at expiry** (`audio/join.rs`, `handlers/req.rs`):
no new effect permit is granted after expiry. Two paths carry calculated
budgets. The audio **lease resolver** holds its permit for at most
`worst_case::PERMIT_HOLD` (14.5 s), and cancellation **during that
resolver** reaches socket close within `worst_case::EXIT` (25 s:
resolver, release of the won lease, terminal flush). Redis checkout has
one total deadline and directory scripts are single-round-trip `EVAL`,
which is what makes those budgets hold. When post-write fence
verification fails after a landed CAS, the acquire keeps the reply and
attempts an exact owner+generation cleanup on the same connection
without a serving-write lease; if that cleanup fails or its reply is
uncertain, the lease expires by its 30 s TTL. Neither number bounds
audio close in general: other admission and teardown waits (for example
the channel lookup after the resolver) and the known limitations below
have no client-side deadline. REQ liveness, search and history reads
race gate cancellation: a lost race drops the read, releases the permit,
retires the claim through `close_if_owner`, and sends no further EVENT
or EOSE. A history statement timeout ends the read with a timed-out
outcome, and its owner-checked retirement (`CLOSED`, map, index and
topic release) runs after the race with the permit dropped, so
cancellation cannot drop it halfway. Subscription registration stays one
uncancellable unit, and the permit is still taken before the claim.
- **JWKS recovery** (`main.rs`): builds on main's
`warm_nip_fi_jwks_snapshots` / `nip_fi_jwks_refresh_loop`. Issuers warm
concurrently and the first refresh is seeded from the warm result (one
interval for a warm issuer, 5 s for a cold one). A refresh that finds no
live snapshot retries at the fast cadence (5 s after a warm issuer dies,
doubling to a 300 s cap) instead of waiting a full interval, and
`run_jwks_refresh_supervisor` restarts the loop after a panic (1 s → 60
s backoff); On shutdown it is cancelled and awaited before the drain
backstop is disarmed; cancel drops an in-flight fetch and starts no
further issuer, so the join stays inside the drain bound. Logs carry
`issuer_index` only.

### Known limitations

- **Known limitations: bounded only by the dependency.** No new effect
is admitted after expiry. Two awaits that are already admitted have no
client-side bound, so if a dependency stalls at expiry, teardown waits
for it:
- **Audio join commit:** its effect permit spans `tx.commit()`, room
publication and the `event_mentions` insert. During a DB or Redis stall,
audio cleanup and socket close wait for the dependency.
- **REQ subscription setup:** the first subscriber to a topic sends
`Subscribe` to the pubsub task through a bounded channel (capacity
4096). The send blocks only if that queue is full, which happens when
the pubsub task isn't keeping up (for example during reconnect backoff
or a burst). Registration holds the connection's `conn.subscriptions`
lifecycle lock across the send, so a blocked send also blocks other
claims on that connection, including search, and any cleanup waiting on
that lock. The send itself predates this PR; holding the REQ permit
across it is new here.

### Archive/join serialization

`commit_participant_join` now takes a row-level write lock (`SELECT ...
FOR NO KEY UPDATE`) on the channel row at step 3. This serializes all
join commits against concurrent `archive_channel` calls — the archive
UPDATE (`archived_at` non-key column) blocks on the row lock, closing
the READ COMMITTED race on both the `Existing` and `AutoAddRequired`
paths.

### HTTP/2 extended-CONNECT gate (latent path)

The root WebSocket handler uses a belt-and-suspenders approach. The
HTTP/1.1 path (currently the only live WebSocket shape — workspace Axum
does not enable `http2`) is gated pre-extractor via the `Upgrade:
websocket` + `Connection: Upgrade` header predicate. A second gate point
inside the `Ok(ws)` arm is structural future-proofing: if `http2` is
ever enabled, any h2 extended-CONNECT shape the extractor accepts but
the h1 predicate misses is caught there. The active HTTP/1.1 fire-point
sits before `bind_community` (zero DB cost on denial); the latent h2
fire-point runs after it.

### Mixed-version huddle control (rolling deploy)

Owners advertise the `huddle-commit-phase` mesh capability. The ingress
selects the mode after the owner stream opens: it sends the appended
`RegisterPeerCommitPhase` variant, and later `CommitConfirmed`, only
when the owner's exact runtime gossip record carries that flag at that
point, and latches the result for the session. A successful open implies
the owner's record is present, so a record that lands during acquisition
is not missed. Otherwise it sends plain `RegisterPeer`, which every
owner decodes. A new owner publishes a plain `RegisterPeer` once at
registration (pre-commit-phase behavior) and holds a
`RegisterPeerCommitPhase` peer until its confirm; duplicate or legacy
confirms publish nothing. Existing wire variant indices are unchanged.

### Off-mode compatibility

When `BUZZ_NIP_FI_MODE=off` (or unset), the relay reads no identity
header and installs no upgrade gate, NIP-42 pairing check, or session
deadline. Existing NIP-42 AUTH flows are unchanged. Huddle
commit-before-publish is not FI-gated and applies in every mode, so
huddle join publication timing differs from main even in Off mode.
External connection cancellation (socket close, archival, revoke) now
stops an in-flight liveness, search or history read at its next await;
an uncancelled connection's frames and payload bytes are unchanged.

## Test coverage

Every test this PR adds, or whose assertions it changes, runs in exactly
one CI lane: infra-free tests in `just test-unit`, Postgres-backed tests
in `postgres_tests` modules under the PostgreSQL lane. The `buzz-relay`
bin tests run whole in `just test-unit`. `scripts/run-tests.sh` mirrors
the effective relay unit selection for machines without nextest, except
main's pre-existing `api::admin::` module. Every `buzz-relay` test
fixture reads `Config` through `Config::for_test()`, which holds
`NIP_FI_ENV_LOCK`, so a single-process `cargo test` run cannot observe a
half-written FI environment
(`fixture_config_read_waits_for_fi_env_lock`). The stub pools in the
router and AUTH fixtures use a 100ms acquire timeout, which shortens a
database fallthrough rather than removing it.

- **Upgrade/router gate** (`router::tests`): Enforce-mode `nip_fi_*`
gate tests for h1 WS upgrade, NIP-11 content negotiation, and audio
ingress. Off-mode witnesses `nip_fi_off_root_passes_without_header`,
`nip_fi_off_root_ignores_malformed_header`,
`nip_fi_off_audio_passes_without_header`,
`nip_fi_off_audio_ignores_malformed_header` prove the gate is bypassed
(status outside 401/403/503, and no-header status equals
malformed-header status); they do not prove a successful upgrade.
- **NIP-FI units**: `nip_fi_config`, `nip_fi_upgrade`, `nip_fi_gate`,
`nip_fi_session`, covering assertion contracts, NIP-42 pairing, session
lifetime, and gate behavior.
- **Effect barrier at cancellation** (`postgres_tests`, PostgreSQL
lane):
`w3_b2_req_barrier_expiry_mid_flight_blocks_subscription_registration`,
`w4_b2_count_barrier_expiry_mid_flight_blocks_count_query`.
- **AUTH denial uniformity** (`handlers::auth::tests`): each
production-handler witness asserts the exact terminal frame, no denial
on the data or ctrl channel, and cancellation, plus the unchanged
Off-mode `OK false`. Unit:
`fi_invalid_nip42_proof_emits_terminal_evidence_rejected`,
`fi_ban_check_error_emits_terminal_authorization_unavailable`,
`handle_auth_pairing_mismatch_runs_full_root_denial_path`, and
`nip42_denial_class_separates_internal_failure_from_bad_evidence`
(internal verifier failure → `authorization unavailable`; each
client-evidence NIP-42 failure → `evidence rejected`). PostgreSQL lane:
`fi_ban_denial_emits_terminal_authorization_denied` (real
`community_bans` row; Off mode keeps `OK false` on ctrl, then close),
`fi_relay_membership_denial_emits_terminal_authorization_denied`,
`fi_allowlist_check_error_emits_terminal_authorization_unavailable`,
`fi_relay_membership_check_error_emits_terminal_authorization_unavailable`,
`fix_4a_allowlist_denial_with_fi_assertion_emits_canonical_restricted_frame`.
- **Bounded audio exits** (`audio::handler::tests`, `just test-unit`):
`check_cancel_exit_with_never_ready_sink_returns_within_flush_budget`
drives the production `check_cancel!` fence with a queued FI denial into
a never-ready sink and requires it to return within
`WS_TERMINAL_FLUSH_TIMEOUT`;
`challenge_send_to_never_ready_sink_exits_within_flush_budget_on_cancel`
requires the NIP-42 challenge send to yield to cancellation and exit
within the same budget; every pre-`send_loop` audio exit write goes
through the same `connection::send_exit_frames_bounded` helper.
- **Audio NIP-42 classification** (`audio::handler::tests`, `just
test-unit`):
`audio_bad_nip42_proof_with_fi_emits_one_evidence_rejected_then_cancels`
requires exactly one canonical `restricted: evidence rejected` frame and
cancellation; `audio_bad_nip42_proof_off_mode_keeps_auth_failed_frame`
pins the Off-mode frame.
- **Effect permit before pre-auth writes** (PostgreSQL lane):
`handlers::auth::tests::postgres_tests::expiry_before_nip_oa_materialize_writes_no_rows_and_commits_no_auth`
expires the session just before NIP-OA materialize and asserts no
`users` or agent-owner row and no committed auth.
- **Community admission** (`just test-unit`):
`state::tests::on_not_run_runs_once_on_each_deny_arm_and_never_on_admit`.
- **Auth metrics contract**:
`connection::tests::auth_lifecycle_reconciles_every_terminal_and_never_leaks_gauge`
records every terminal including `PairingMismatch`;
`metrics::contract_tests::*` derive the series count from
`AuthOutcome::ALL`.
- **JWKS** (`main.rs`): `hard_dead_warm_issuer_resets_to_fast_cadence`
and `jwks_refresh_success_waits_interval_failure_uses_fast_backoff`
drive the real loop with paused Tokio time; `cold_issuer_*` cover the
backoff helper;
`f1_supervisor_loop_drives_recovery_and_restores_admission` runs
`run_jwks_refresh_supervisor` over a real `ProductionJwksSource` through
injected panic → restart → hard-dead → 5 s retry → recovery, then checks
`key_set` admission and that the first post-restart fetch follows the
cold cadence (every issuer restarts cold);
`jwks_refresh_cancel_drops_in_flight_fetch_and_skips_remaining_issuers`
cancels during a pending fetch with a second issuer due.
- **Bounded lease acquire** (`audio::handler::tests::postgres_tests`,
PostgreSQL lane):
`failed_post_cas_verification_is_fenced_and_discards_the_won_lease`
requires a won-then-unverified lease to return fenced and be deleted by
exact owner+generation;
`stalled_lease_sql_and_stalled_exit_release_close_within_exit_bound`
stalls lease SQL and the exit release and requires socket close within
`worst_case::EXIT`. `audio::join::tests` pins `PERMIT_HOLD` at 14.5 s
and `EXIT` at 25 s.
- **REQ/search cancellation** (`handlers::req::tests`, `just
test-unit`):
`expiry_during_stalled_history_read_releases_permit_and_claim` and
`expiry_during_stalled_search_read_releases_permit_and_claim` stall the
read inside production `handle_req`, expire the gate, and require
`expire()` to return within 2 s, an empty subscription map, and no EVENT
or EOSE.
- **History timeout retirement**
(`handlers::req::tests::postgres_tests`, PostgreSQL lane):
`history_timeout_retirement_survives_expiry` drives a two-channel
history REQ into a real statement timeout, pauses retirement after the
index removal and before the topic release, expires the gate, then
resumes. It requires the map and index entries gone and both channel
topic refcounts at zero.
- **Archive/join serialization** (PostgreSQL lane):
`f2a_archived_channel_rejects_existing_join`,
`f2b_join_for_update_blocks_concurrent_archive_existing_path`,
`f2c_join_for_update_blocks_concurrent_archive_auto_add_path`.
- **Huddle room and handler**: `audio::room::tests` pending-peer tests,
including `roster_revisions_are_ordered_and_snapshot_is_authoritative`;
`fix_b_w1_w3_handler_pre_commit_cancel_emits_no_delta`,
`fix_b_w2_success_emits_exactly_one_delta_with_monotone_revision`.
PostgreSQL lane: `b1_cross_pod_bootstrap_precedes_buffered_owner_delta`,
`commit_confirm_timeout_at_seam_triggers_teardown_on_stalled_owner_stream`,
`confirm_failure_combined_outcome_no_orphan_join_exactly_one_48102`,
`confirm_send_pending_expiry_terminates_client_with_denial_within_1s`,
`c1_handler_ingress_sends_only_base_frames_to_legacy_owner`,
`f7a_cross_pod_joined_payload_includes_owner_pod_peers`.
- **Mixed-version control** (`audio::join::tests`):
`shared_control_variants_are_byte_identical_to_base_wire` (against
frozen `audio::join::base_wire`),
`c1_new_ingress_speaks_base_wire_to_legacy_owner`,
`c2_base_ingress_register_publishes_once_on_new_owner`,
`c3_capable_owner_record_selects_commit_phase_hold_back`,
`capability_record_installed_during_open_selects_commit_phase`,
`duplicate_or_legacy_commit_confirmed_is_a_no_op`,
`fix_b_commit_confirmed_arm_commits_peer_and_broadcasts_joined`,
`pending_remote_stream_close_emits_no_joined_no_left`.

---------

Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>

2026-09-29 14:27:40 -04:00

Tyler

ccb021d713

Relay mesh: cross-pod tunnel + huddle transport (buzz-relay-mesh) (#1670 )

Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: Tyler Longwell <tlongwell@squareup.com>
Signed-off-by: tlongwell-block <109685178+tlongwell-block@users.noreply.github.com>
Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub1mprnacetjua2xx3p5eddmhxyk6wv929ymm5py8kd2xfxurxahspqqlgyta <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1cc3ha7z055mu0rwwu7806t2wt8mj3pvu0uv5mfp2c50dahaqhczshdalg6 <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub17jjz49l9jjmhhk7cac63j8yt9z555n9cw8vk7v5jz4vzw4ppld5qgj57cc <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1jh9wn95s0472h86ahapupaf7m6kx4v9sx2n0atj2hltcfer8k06s5n3pyf <95cae996907d7cab9f5dbf43c0f53edeac6ab0b032a6feae4abfd784e467b3f5@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub1t2tgm7d8f995uqvmnm8h88sg3wnpp9a5xysjf6dg3tjmgt3ltulqdp8ehr <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>

2026-07-14 16:56:48 -04:00

3 Commits