fix(mesh): expire dead peer records and instrument mesh boot (#8503)

## Summary

Relay pods can stall in the `mesh_boot` startup step, and the mesh peer
table grows without limit. One prod pod held 7,793 peer records. Most of
those records belong to pods that no longer exist.

**Cause.** Every boot gets a fresh runtime id, so a restarted pod leaves
its old record behind. Nothing ever removed records, and scuttlebutt
gossip kept spreading the dead ones around the fleet. Each reconcile
pass, including the first one inside `boot_mesh`, then dials every
record one at a time.

This PR makes dead records expire. It also adds the measurements we need
to see the next stall directly.

### Peer-table lifetime (`crates/buzz-relay-mesh/src/membership.rs`)
- A peer record expires 5 minutes after its last heartbeat
(`DEFAULT_PEER_RECORD_TTL`). For comparison, live runtimes refresh every
2s, and the Redis ready registry expires a silent runtime after 45s.
- What happens to expired records:
- They are **rejected on arrival**, so peers that haven't been upgraded
can't re-add them.
  - They are **left out** of digests, deltas, and dial candidates.
  - They are **pruned** at the start of each reconcile pass.
- Between prunes, they are not admitted (`has_peer`) and not routed
(`peers`, `peer_has_capability`).
- An expired entry that hasn't been pruned yet counts as absent, so any
fresh record replaces it whatever its version. That lets a live peer's
v1 Redis hint re-admit it before the next prune.
- **The TTL never prunes or hides a peer with a live connection**, even
if its heartbeat is stale. Its gossip can stop arriving while the
transport stays up: a dead control stream, or a clock behind by more
than the TTL. The connection proves the runtime is alive. Its stale
record is still kept out of digests and dials. (The existing phi filter
in `peers()` still drops a connected peer once its gossip stalls; this
PR doesn't change that.)
- **`Connected` tracks the transport map.** Because the TTL exemption
trusts that flag, `install_peer` and `remove_peer` write it while
holding the peers lock, so a connection that closes during install
always ends `Disconnected`. A dial's `Connecting`/`Disconnected` writes
are skipped when a connection is already installed, so a failed outbound
dial can't overwrite a live inbound connection. Each reconcile pass also
re-syncs the state from the transport map.
- **Gossip no longer sets connection state.** Before, any gossiped
record was marked `Connected`, even if this pod had never dialed it.
That inflated the `/_mesh` counts. Now a record learned from gossip or
the ready registry enters as `Disconnected`. Only the real connect and
disconnect paths change the state.
- Before building its candidate list, reconcile now skips peers that are
already connected. A pass dials serially, and a dial to a dead runtime
can take about a minute, so `dial_peer` re-reads the record before every
address attempt. It stops once the record has expired or started
draining, or once a connection has landed.

### Instrumentation
- **`mesh_boot` sub-steps** use the existing startup-step gauges:
`mesh_bind`, `mesh_publish`, and `mesh_initial_reconcile`. A pod stuck
in `mesh_boot` now also shows which await it is stuck on.
- **New metrics** (`crates/buzz-relay-mesh/src/telemetry.rs`):

  | Metric | Shows |
  |---|---|
| `buzz_mesh_dial_total{outcome}` / `buzz_mesh_dial_duration_seconds` |
Cost of each dial |
  | `buzz_mesh_reconcile_pass_seconds` | Full pass time |
| `buzz_mesh_reconcile_candidates` | Dials planned for this pass (set
after the registry scan) |
  | `buzz_mesh_peer_records{state,draining}` | Table size |
  | `buzz_mesh_peer_record_max_age_seconds` | Oldest heartbeat |
| `buzz_mesh_peer_records_expired_total{reason=pruned\|rejected}` |
Expiry activity |

- The dial and pass histograms get buckets up to 600s. The default
`_seconds` buckets stop at 5s, which would hide exactly the stall these
metrics are meant to show.

### CI
The `buzz-relay-mesh` tests and the `buzz-relay` `mesh_boot::tests` did
not run in any CI lane. This PR adds both to `just test-unit`, and adds
the matching stanzas to `scripts/run-tests.sh`.

### What got simpler
- The peer table now has a defined size, about one record per live pod,
where before it grew forever.
- One wrong state write was deleted (gossip marking records
`Connected`).
- Two test suites that silently never ran now run.

### Rollout notes
- Rejecting expired records works one pod at a time. During a rollout,
pods that haven't been upgraded keep gossiping dead records. Upgraded
pods reject them, which shows up as `reason="rejected"` on the expired
counter.
- A pod whose clock runs more than 5 minutes behind has all of its
gossip rejected. It stays reachable through its Redis ready record and
any live connection. A `reason="rejected"` count that keeps climbing
outside a rollout points to that skew.
- Upgraded pods leave expired records out of their digests. Pods that
haven't been upgraded therefore resend the whole ghost set (about 7.8k
records, roughly 1 MB) on every 2s gossip tick, on each old↔new
connection. So during the rollout, `reason="rejected"` will climb by
about the ghost count per tick per connection. This is expected and
stops when the rollout finishes.

### Not in this PR
The boot dial itself is still unbounded: it runs serially, has no
per-dial timeout, and dials gossip-learned records. Bounding it changes
how a pod joins the mesh, so it goes in a separate PR, and these metrics
will measure it.

### Related issue
None found.

### Testing
- New tests:
- `membership::tests`: expired gossip is rejected; prune drops only
expired records; prune keeps a connected peer with a stale heartbeat;
expired unconnected records are neither admitted nor routed before a
prune; a fresh record replaces an expired, unpruned entry whatever its
version; expired records are neither dialed nor gossiped before a prune;
gossip never sets connection state; table stats.
- `runtime::tests::reconcile_pass_publishes_dial_and_peer_table_metrics`
- `runtime::tests::reconcile_pass_prunes_expired_peer_records` (binds
the prune to the reconcile pass)
-
`runtime::tests::dial_skips_candidate_that_expired_after_the_pass_snapshot`
-
`runtime::tests::finishing_dial_does_not_overwrite_a_connection_that_landed`
-
`runtime::tests::connection_closed_during_install_ends_disconnected_and_expires`
-
`runtime::tests::reconcile_pass_resyncs_connection_state_from_transport`
  - `mesh_boot::tests::mesh_boot_sub_steps_name_the_step_that_failed`
- Blox at `e06f55a4` (tree `866add1c`): `just fmt-check`, `just clippy`,
`buzz-relay-mesh` nextest 3x, and the `buzz-relay`
`mesh_boot`/`startup_steps`/`metrics` tests all pass. Three of the new
tests fail against a mutant that removes the fix they pin (dial ignores
expiry; dial state writes ignore the transport map; no reconcile
resync).

---------

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>

Justfile

@@ -417,6 +417,10 @@ test-unit:
         # already runs it.
         cargo test -p buzz-auth --doc
         cargo nextest run -p buzz-voice --lib
+        # Inter-relay mesh: membership, gossip, registry, and loopback-UDP
+        # runtime tests. Infra-free (Redis-backed paths are unit-faked or
+        # never dialed). Keep the Cargo fallback in step.
+        cargo nextest run -p buzz-relay-mesh
         cargo nextest run -p buzz-cli
         # buzz-sdk builder/validation unit tests: pure event-builder and input
         # validation (e.g. the canvas writer-discipline/skew guard and the
@@ -532,12 +536,14 @@ test-unit:
         # (JWKS source + refresh-loop composition), which `--lib` cannot reach
         # because they live in the binary target; the nested
         # `tests::postgres_tests::` stays in the PostgreSQL lane.
-        # The fourth clause adds the startup step timers (startup_steps) and the
-        # env- and log-capture tests in telemetry and config. All three modules
-        # are infra-free; config parses env into a struct and never connects.
+        # The fourth clause adds the startup step timers (startup_steps), the
+        # mesh boot wiring (mesh_boot; Redis and Postgres point at refused
+        # ports), and the env- and log-capture tests in telemetry and config.
+        # All four modules are infra-free; config parses env into a struct and
+        # never connects.
         # `^config::` is anchored so it does not also select nip_fi_config.
         cargo nextest run -p buzz-relay --lib --bin buzz-relay \
-            -E 'test(/^api::admin::/) + test(/^push_runtime::/) + test(/^operator_listener::tests::/) + test(/^handlers::channel_authz::/) + test(/^handlers::moderation_authz::/) + test(/^handlers::side_effects::tests::/) + test(/^handlers::system_channels::tests::/) + test(/^nip11::tests::/) + test(/^storage_sweep::tests::/) + test(/^nip_fi_core::tests::/) + test(/^nip_fi_http::tests::/) + test(/^nip_fi_config::tests::/) + test(/^nip_fi_resolver::tests::/) + test(/^readiness::tests::/) + test(/^router::tests::/) + test(/^api::parse_query_tests::/) + test(/^api::profiles::/) + test(/^api::git::transport::off_mode_precedence_tests::/) + test(/^audio::join::tests::/) + test(/^audio::handler::tests::/) + test(/^nip_fi_gate::tests::/) + test(/^nip_fi_session::tests::/) + test(/^nip_fi_shadow::tests::/) + test(/^nip_fi_shadow_session::tests::/) + test(/^startup_steps::tests::/) + test(/^telemetry::tests::/) + test(/^config::tests::/) + test(=state::tests::neither_a_confirmed_inactive_community_nor_a_failed_lookup_admits_the_socket) + test(=state::tests::periodic_revalidation_disconnects_inside_the_fenced_callback) + test(=state::tests::bare_community_disconnect_fails_closed_when_the_fence_is_unavailable) + test(=connection::tests::send_loop_sends_policy_close_when_community_is_archived) + test(=state::tests::community_disconnect_then_nip_fi_keeps_community_deleted_reason) + test(=state::tests::disconnect_community_wins_reason_losing_nip_fi_does_not_enqueue_frame) + test(=handlers::req::tests::timed_out_historical_read_deregisters_before_closed) + test(=handlers::req::tests::superseded_timeout_leaves_replacement_intact) + test(=handlers::req::tests::search_claim_retires_live_and_yields_to_replacement) + test(=handlers::req::tests::concurrent_claims_and_stale_teardowns_keep_the_last_owner) + test(=handlers::req::tests::timeout_closed_is_emitted_before_a_replacement_can_claim) + test(=handlers::req::tests::revoke_then_replacement_keeps_replacement_whole) + test(=handlers::req::tests::claims_after_connection_cleanup_are_refused) + test(=handlers::req::tests::dropped_terminal_frame_cancels_connection) + test(=handlers::req::tests::revoke_dropped_terminal_frame_cancels_connection) + test(=handlers::req::tests::count_fallback_covers_filters_that_can_match_private_deletions) + test(=handlers::req::tests::group_state_t_and_creator_tags_reach_sql_only_for_group_state_kinds) + test(=api::bridge::postgres_tests::bridge_search_mode_extension_accepts_prefix_snake_or_camel_case) + test(=api::bridge::postgres_tests::bridge_search_sort_extension_selects_newest_only_when_named) + (kind(bin) & (test(/^tests::/) + test(/^composition_tests::/) + test(/^env_filter_tests::/)) - test(/^tests::postgres_tests::/))'
+            -E 'test(/^api::admin::/) + test(/^push_runtime::/) + test(/^operator_listener::tests::/) + test(/^handlers::channel_authz::/) + test(/^handlers::moderation_authz::/) + test(/^handlers::side_effects::tests::/) + test(/^handlers::system_channels::tests::/) + test(/^nip11::tests::/) + test(/^storage_sweep::tests::/) + test(/^nip_fi_core::tests::/) + test(/^nip_fi_http::tests::/) + test(/^nip_fi_config::tests::/) + test(/^nip_fi_resolver::tests::/) + test(/^readiness::tests::/) + test(/^router::tests::/) + test(/^api::parse_query_tests::/) + test(/^api::profiles::/) + test(/^api::git::transport::off_mode_precedence_tests::/) + test(/^audio::join::tests::/) + test(/^audio::handler::tests::/) + test(/^nip_fi_gate::tests::/) + test(/^nip_fi_session::tests::/) + test(/^nip_fi_shadow::tests::/) + test(/^nip_fi_shadow_session::tests::/) + test(/^startup_steps::tests::/) + test(/^mesh_boot::tests::/) + test(/^telemetry::tests::/) + test(/^config::tests::/) + test(=state::tests::neither_a_confirmed_inactive_community_nor_a_failed_lookup_admits_the_socket) + test(=state::tests::periodic_revalidation_disconnects_inside_the_fenced_callback) + test(=state::tests::bare_community_disconnect_fails_closed_when_the_fence_is_unavailable) + test(=connection::tests::send_loop_sends_policy_close_when_community_is_archived) + test(=state::tests::community_disconnect_then_nip_fi_keeps_community_deleted_reason) + test(=state::tests::disconnect_community_wins_reason_losing_nip_fi_does_not_enqueue_frame) + test(=handlers::req::tests::timed_out_historical_read_deregisters_before_closed) + test(=handlers::req::tests::superseded_timeout_leaves_replacement_intact) + test(=handlers::req::tests::search_claim_retires_live_and_yields_to_replacement) + test(=handlers::req::tests::concurrent_claims_and_stale_teardowns_keep_the_last_owner) + test(=handlers::req::tests::timeout_closed_is_emitted_before_a_replacement_can_claim) + test(=handlers::req::tests::revoke_then_replacement_keeps_replacement_whole) + test(=handlers::req::tests::claims_after_connection_cleanup_are_refused) + test(=handlers::req::tests::dropped_terminal_frame_cancels_connection) + test(=handlers::req::tests::revoke_dropped_terminal_frame_cancels_connection) + test(=handlers::req::tests::count_fallback_covers_filters_that_can_match_private_deletions) + test(=handlers::req::tests::group_state_t_and_creator_tags_reach_sql_only_for_group_state_kinds) + test(=api::bridge::postgres_tests::bridge_search_mode_extension_accepts_prefix_snake_or_camel_case) + test(=api::bridge::postgres_tests::bridge_search_sort_extension_selects_newest_only_when_named) + (kind(bin) & (test(/^tests::/) + test(/^composition_tests::/) + test(/^env_filter_tests::/)) - test(/^tests::postgres_tests::/))'
         # Note on audio::join::tests scope: the full suite is infra-free (no DB,
         # no Redis). The infra-free audio/FI regression witnesses — bootstrap
         # ordering barrier, CommitConfirmed arm, pending-close invisibility,