## Problem
A rolling deploy can stall relay startup for 30s per dead mesh peer.
- A pod publishes its mesh ready record during `mesh_boot` and clears it
only on graceful shutdown. If a later fatal startup check kills the pod,
the record lingers until it expires.
- Dialing that address never gets an answer, so `MeshEndpoint::connect`
waits out QUIC's default 30s idle timeout. `reconcile_once` dials
candidates one at a time, so every dead peer adds another 30s, both in
the boot-time `mesh_initial_reconcile` step and in every later reconcile
pass.
- The fatal check that killed those pods was the git conformance probe.
A transient S3 `500 InternalError` on one racer was treated as a
conformance failure.
## Changes
1. **Bounded, concurrent mesh dials**
(`buzz-relay-mesh/src/runtime.rs`). Each connect attempt is wrapped in
`DIAL_TIMEOUT` (2s; in-cluster handshakes take milliseconds), and a
reconcile pass dials its candidates concurrently. A pass with dead peers
now costs about one dial timeout instead of 30s per peer. The candidate
set is bounded by the membership table.
2. **Conformance probe before mesh boot** (`buzz-relay/src/main.rs`).
This only moves an existing block, so a pod that fails the probe exits
before it publishes a ready record peers would dial.
3. **5xx racers are unknown outcomes**
(`buzz-relay/src/api/git/store.rs`). The race phases already drop
socket/send failures from the observer set as "outcome unknown". A 5xx
is the backend declining to classify the write, so it now joins them
through one shared `racer_outcome_unknown` helper. 4xx answers,
including 412, still count as evidence. A backend that 5xxs every racer
still fails closed on the two-observer floor. Because a dropped racer's
write may still have landed, every `if_match_race` round now reads the
pointer back and requires the winner's exact body and ETag. Otherwise a
backend that commits a stale write and then answers 5xx would pass in
the last round. §Conformance steps 2–3 in
`docs/git-on-object-storage.md` now state this drop-and-floor rule.
## What got simpler
The two duplicated drop-classification match arms in the probe now share
one helper. The probe/mesh fix is a reorder of existing startup steps,
not a new mechanism or rollback path.
## Tests
- `runtime::tests::reconcile_pass_bounds_dials_to_unresponsive_peers`:
three black-hole peers (bound UDP sockets that never read) plus one live
peer. The pass must finish within `2 × DIAL_TIMEOUT` (a serial pass
would take at least `3 × DIAL_TIMEOUT`, and without the timeout about
30s), the live peer must connect, and each dead peer must end
`Disconnected`.
- `store::tests::probe_drops_5xx_racers_but_keeps_4xx_as_evidence`: pins
the classification boundary (5xx and I/O are unknown; 400/403/412/499
are evidence).
- `store::fault_probe::*`: runs `run_conformance_probe` against an
in-process S3 fake that injects racer faults at the HTTP boundary, so
responses pass through rust-s3's `fail-on-err` mapping and retry. One
5xx racer in either race phase is dropped and admitted (`transport_drops
== 1`). Too few classified observers fail closed: one observer in either
phase, and all-5xx in `if_none_match_race`. An unexpected 4xx racer
fails closed in both phases. A racer whose stale write lands and is then
answered 5xx fails closed on the post-round readback
(`dropped_racer_whose_stale_write_lands_fails_closed`). A clean fake
passes with zero drops.
- CI lane: no `api::store` test ran in any CI lane before,
including the existing ones. The `just test-unit` nextest filter and the
`scripts/run-tests.sh` fallback now select `api::store::tests` and
`::fault_probe`. The live-MinIO `::probe` module stays opt-in.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Authored by Brain and opened on behalf of Wes (`wesbillman`).
## Summary
Add `buzz repos default-branch get/set` and the relay operation it
needs. Git push changes branch tips but cannot select the server's
symbolic HEAD. This selects an existing branch without deleting
branches, moving refs, or changing packs.
- Reuse immutable Git manifests and captured-ETag CAS; concurrent
pushes/default changes conflict, including stale no-ops.
- Require request-specific NIP-98, body binding, fail-closed replay
protection, host tenancy, current channel membership and
repository-management authority. Push/project roles alone are
insufficient.
- Send the CLI mutation once, without redirects; ambiguous delivery
retains the observed digest and is non-retryable.
- Document the narrow HTTP exception to Nostr-first, delegation scope
and admission-time ACL revocation. Kind:30618 remains derived; the
manifest CAS is authoritative.
- Register the Postgres/Redis/MinIO route and fresh-clone tests in the
existing Backend Integration lane.
## Review fixes — `7457bda50afa6b52e36315572fa96928b2e21db6`
- Reject signed valueless `payload` tags when verifying a body; preserve
the shared verifier's intentional absent-payload compatibility. The real
settings route now proves 401, unchanged manifest pointer, and unchanged
kind:30618 event IDs for malformed and other invalid credentials.
- Require a nonempty returned branch and `head == refs/heads/{branch}`;
SET additionally requires the returned branch to equal the attempted
branch. Malformed/mismatched mutation responses remain non-retryable
`DeliveryUnknown`, retaining the attempted branch and original digest.
Invalid reads fail before any POST.
- Strengthen the later-push/fresh-clone regression to select
`release/v1` while `main` exists.
## Verification
At `7457bda50afa6b52e36315572fa96928b2e21db6`:
- Full pre-push passed (all 14 Rust test groups, serialized Tauri
tests/clippy, file-size and branch-skew gates; 522 seconds). No hooks
disabled.
- All four isolated Postgres/Redis/MinIO settings tests pass, including
real Smart HTTP `ls-remote --symref` and fresh clone on `release/v1`.
- All 15 focused NIP-98 verifier tests and three default-branch CLI
tests pass; malformed credentials and missing-branch responses were
reproduced failing before the fixes.
- Five targeted mutants are killed: restoring the valueless-payload
bypass, weakening canonical HEAD validation, accepting an empty branch,
dropping requested-branch confirmation, and omitting hydrated HEAD for
the selected non-main branch. Mutations ran in a separate worktree and
target directory; original patch bytes were verified restored.
- `cargo clippy -p buzz-auth -p buzz-cli -p buzz-relay --all-targets --
-D warnings`, formatting, diff and file-size checks pass on the fixed
tree.
Earlier validation at `bb195b140`: seven Git authorization database
regressions and 28 existing transport tests passed (three infrastructure
cases in that module not selected); four mutations killed
management/replay/CAS/redirect regressions. Full pre-push passed with
`RUST_TEST_THREADS=1` (all 14 Rust groups, Tauri clippy and 3,173 tests,
size/branch-skew gates). Serial execution avoids an unrelated
process-global Desktop discovery counter flake; no hooks disabled or
Desktop changes.
Carl and Jude's review findings are addressed in the new head; **fresh
reviewer and exact-head security confirmation remain required**. The
earlier source review is not approval of this revision. No claim of
green new-head CI.
## Rollout
Merge and **deploy the relay support first**, then build/use the updated
CLI and explicitly set/verify the desired default. No schema migration.
This PR does not deploy anything or change a live repository.
Originating conversation (channel
`8e2fda80-9142-4df8-8e14-da288bea6471`):
buzz://message?channel=8e2fda80-9142-4df8-8e14-da288bea6471&id=8a7a9cf12737abc1e16605f241300be940a1691ef0992b35197afe26c563108d
---------
Signed-off-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
Co-authored-by: Brain <1a02c72794dcd0f07058a353bc3a81f4028b8c77c92c87fce6d5c8b85970a20b@buzz.block.builderlab.xyz>
