mirror of
https://github.com/block/buzz.git
synced 2026-11-06 17:31:19 +00:00
9b4b73eb08
## Problem A rolling deploy can stall relay startup for 30s per dead mesh peer. - A pod publishes its mesh ready record during `mesh_boot` and clears it only on graceful shutdown. If a later fatal startup check kills the pod, the record lingers until it expires. - Dialing that address never gets an answer, so `MeshEndpoint::connect` waits out QUIC's default 30s idle timeout. `reconcile_once` dials candidates one at a time, so every dead peer adds another 30s, both in the boot-time `mesh_initial_reconcile` step and in every later reconcile pass. - The fatal check that killed those pods was the git conformance probe. A transient S3 `500 InternalError` on one racer was treated as a conformance failure. ## Changes 1. **Bounded, concurrent mesh dials** (`buzz-relay-mesh/src/runtime.rs`). Each connect attempt is wrapped in `DIAL_TIMEOUT` (2s; in-cluster handshakes take milliseconds), and a reconcile pass dials its candidates concurrently. A pass with dead peers now costs about one dial timeout instead of 30s per peer. The candidate set is bounded by the membership table. 2. **Conformance probe before mesh boot** (`buzz-relay/src/main.rs`). This only moves an existing block, so a pod that fails the probe exits before it publishes a ready record peers would dial. 3. **5xx racers are unknown outcomes** (`buzz-relay/src/api/git/store.rs`). The race phases already drop socket/send failures from the observer set as "outcome unknown". A 5xx is the backend declining to classify the write, so it now joins them through one shared `racer_outcome_unknown` helper. 4xx answers, including 412, still count as evidence. A backend that 5xxs every racer still fails closed on the two-observer floor. Because a dropped racer's write may still have landed, every `if_match_race` round now reads the pointer back and requires the winner's exact body and ETag. Otherwise a backend that commits a stale write and then answers 5xx would pass in the last round. §Conformance steps 2–3 in `docs/git-on-object-storage.md` now state this drop-and-floor rule. ## What got simpler The two duplicated drop-classification match arms in the probe now share one helper. The probe/mesh fix is a reorder of existing startup steps, not a new mechanism or rollback path. ## Tests - `runtime::tests::reconcile_pass_bounds_dials_to_unresponsive_peers`: three black-hole peers (bound UDP sockets that never read) plus one live peer. The pass must finish within `2 × DIAL_TIMEOUT` (a serial pass would take at least `3 × DIAL_TIMEOUT`, and without the timeout about 30s), the live peer must connect, and each dead peer must end `Disconnected`. - `store::tests::probe_drops_5xx_racers_but_keeps_4xx_as_evidence`: pins the classification boundary (5xx and I/O are unknown; 400/403/412/499 are evidence). - `store::fault_probe::*`: runs `run_conformance_probe` against an in-process S3 fake that injects racer faults at the HTTP boundary, so responses pass through rust-s3's `fail-on-err` mapping and retry. One 5xx racer in either race phase is dropped and admitted (`transport_drops == 1`). Too few classified observers fail closed: one observer in either phase, and all-5xx in `if_none_match_race`. An unexpected 4xx racer fails closed in both phases. A racer whose stale write lands and is then answered 5xx fails closed on the post-round readback (`dropped_racer_whose_stale_write_lands_fails_closed`). A clean fake passes with zero drops. - CI lane: no `api::store` test ran in any CI lane before, including the existing ones. The `just test-unit` nextest filter and the `scripts/run-tests.sh` fallback now select `api: