Files
buzz/crates/buzz-test-client/src/lib.rs
Matt Toohey 1893c3ab5a test: dedupe collect_until_eose by event id and uncap two shell tests (#8677)
## Summary

Fixes two CI flakes from the 21–28 Oct 2026 audit, both in test code
with no product change:

- The relay E2E helper `collect_until_eose` now drops repeat copies of
an event id.
- Two Windows shell tests drop a 5 s cap that a cold Git Bash start can
exceed.

This is one of seven PRs split out of the `flakey-tests` branch
(siblings below).

### Related issue

None found for these two. The buzz-terminal `passwd_shell()` `getpwuid`
race
(`env_fence_tests::resolve_shell_falls_through_to_passwd_not_the_default`,
run 38590095289) is left to open PR #7687, which already replaces
`getpwuid`. #8636 and #7976 edit `buzz-test-client/src/lib.rs`, and
#7607 and #7915 edit `buzz-dev-mcp/src/shell.rs`, but none of them
changes `collect_until_eose` or these timeouts.

### Flaky tests fixed

| Test | Failing runs | Cause | Change |
|---|---|---|---|
| Relay E2E
`e2e_project::test_project_tombstone_deletes_coordinate_and_spares_members`
(e2e_project.rs:338, "left: 2 right: 1") | 40408678412#1, job
121879630394 (PR #8620, schema-only) | The EVENT handler sends OK, then
fans out in a spawned task. `handle_req` registers the live subscription
before its history query, and `req.rs`'s `seen_ids` dedups history rows
only. If fan-out runs after the REQ is registered, the event arrives
once live and once from history, both before EOSE, and
`collect_until_eose` kept every frame. | `collect_until_eose` keeps the
first copy of each event id, as production clients do. Its doc comment
says a test of the relay's own dedup must read frames with `recv_event`.
A new unit test, `collect_until_eose_keeps_one_copy_of_each_event_id`,
uses a fake relay that sends the duplicate. |
| Windows `buzz-dev-mcp` `shell::tests::basic_echo` (shell.rs:1026, exit
124) | 39675698081#1, job 119485363399 (green on re-run) | 124 is the
shell tool's own timeout exit code. `basic_echo` capped the shell at 5
s, and it is one of the first tests in the process to start Git Bash,
which can take longer than 5 s on a cold runner. | `timeout_ms: None`
(the production 120 s default) in `basic_echo` and in
`workdir_is_honored`, which has the same exposure. `timeout_fires` still
covers timeouts. |

Pre-check: no e2e test relies on the relay's own history dedup. Every
exact-count assertion uses a single filter. The NIP-33 "only newest
remains" assertions would see a failed replacement as two different ids,
and this dedup keeps both.

Local evidence is from the original `flakey-tests` commit (fc4518d39):

- The new unit test: FAIL before, PASS after.
- 13 of the 14 e2e files that call the helper, plus `e2e_thread_roots`,
against an isolated local relay. All pass, except `e2e_relay`
`test_unarchive_emits_member_added_notification`. That test fails by
itself 3/5 with the original helper too, and CI's Relay E2E lane doesn't
select it.
- The Windows failure can't be reproduced on macOS. The Windows Rust
lane is the check.

Follow-up (out of scope): make the relay deliver each event id at most
once per subscription, keeping the live/history overlap.

### Verification

This branch was pushed without local re-verification, by the user's
choice: local test runs were overloading the machine and failing for
that reason. **Hosted CI is the check.** It was pushed with `git push
--no-verify`, because the pre-push hook runs clippy and unit tests.

- Both files are byte-identical to their final versions on
`flakey-tests` (67185df6a).
- Checked for this PR: `git diff --check`, `rustfmt --check`, and a
conflict-free merge with all six siblings onto `main`.

### Readiness

Test-only change. AGENTS.md's readiness steps (agent review, then human
confirmation) are still outstanding, so `buzz-review-completed` is not
set.

### Merge order

Independent of the other six. It merges cleanly onto `main`, and with
every sibling, in any order.

### Sibling PRs

Split from `flakey-tests` (one PR per area): #8676 (PostgreSQL lease +
lane), #8678 (mobile), #8679 (Tauri apt cache + Windows Rust cap), #8680
(desktop product races), #8681 (desktop E2E state gates), #8682 (desktop
E2E composer + row actions).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Matt Toohey <contact@matttoohey.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 10:00:21 +11:00

420 lines
15 KiB
Rust

#![deny(unsafe_code)]
#![warn(missing_docs)]
//! Minimal NIP-01 WebSocket test client for the Buzz relay.
use std::collections::HashSet;
use std::time::Duration;
use nostr::{Event, EventBuilder, Filter, Keys, Kind, Tag};
use serde_json::{json, Value};
use thiserror::Error;
use tracing::debug;
use buzz_ws_client::NostrWsConnection;
pub use buzz_ws_client::{parse_relay_message, OkResponse, RelayMessage, WsClientError};
/// Scenario-driven load generator behind the `buzz-loadgen` binary.
/// The scenario schema is documented in `perf/LOADGEN.md`.
#[allow(missing_docs)]
pub mod loadgen;
/// Errors returned by [`BuzzTestClient`] operations.
#[derive(Debug, Error)]
pub enum TestClientError {
/// A WebSocket transport error occurred.
#[error("WebSocket error: {0}")]
WebSocket(tokio_tungstenite::tungstenite::Error),
/// A JSON serialization or deserialization error occurred.
#[error("JSON error: {0}")]
Json(serde_json::Error),
/// Failed to build a Nostr event.
#[error("Nostr event builder error: {0}")]
EventBuilder(String),
/// Failed to parse a URL.
#[error("URL parse error: {0}")]
Url(String),
/// The relay did not respond within the expected time.
#[error("Timeout waiting for relay message")]
Timeout,
/// The WebSocket connection was closed before the operation completed.
#[error("Connection closed unexpectedly")]
ConnectionClosed,
/// The relay sent a message that was not expected at this point.
#[error("Unexpected relay message: {0}")]
UnexpectedMessage(String),
/// NIP-42 authentication was rejected by the relay.
#[error("Authentication failed: {0}")]
AuthFailed(String),
/// The relay rejected the submitted event.
#[error("Event rejected by relay: {0}")]
EventRejected(String),
/// No NIP-42 AUTH challenge was received from the relay.
#[error("No AUTH challenge received from relay")]
NoAuthChallenge,
}
impl From<WsClientError> for TestClientError {
fn from(e: WsClientError) -> Self {
match e {
WsClientError::WebSocket(e) => TestClientError::WebSocket(e),
WsClientError::Json(e) => TestClientError::Json(e),
WsClientError::EventBuilder(s) => TestClientError::EventBuilder(s),
WsClientError::Url(s) => TestClientError::Url(s),
WsClientError::Timeout => TestClientError::Timeout,
WsClientError::ConnectionClosed => TestClientError::ConnectionClosed,
WsClientError::UnexpectedMessage(s) => TestClientError::UnexpectedMessage(s),
WsClientError::AuthFailed(s) => TestClientError::AuthFailed(s),
WsClientError::EventRejected(s) => TestClientError::EventRejected(s),
WsClientError::NoAuthChallenge => TestClientError::NoAuthChallenge,
}
}
}
impl From<nostr::event::builder::Error> for TestClientError {
fn from(e: nostr::event::builder::Error) -> Self {
TestClientError::EventBuilder(e.to_string())
}
}
/// WebSocket test client for integration testing against a running Buzz relay.
pub struct BuzzTestClient {
inner: NostrWsConnection,
}
impl BuzzTestClient {
/// Connects to the relay at `url` and performs NIP-42 authentication with `keys`.
pub async fn connect(url: &str, keys: &Keys) -> Result<Self, TestClientError> {
let mut client = Self::connect_unauthenticated(url).await?;
client.authenticate(keys).await?;
Ok(client)
}
/// Connects to the relay at `url` without performing authentication.
pub async fn connect_unauthenticated(url: &str) -> Result<Self, TestClientError> {
let inner = NostrWsConnection::connect(url).await?;
debug!("connected to relay at {url}");
Ok(Self { inner })
}
/// Performs NIP-42 authentication using `keys` against the connected relay.
pub async fn authenticate(&mut self, keys: &Keys) -> Result<(), TestClientError> {
self.inner.authenticate(keys, None).await?;
Ok(())
}
/// Performs NIP-42 authentication with a NIP-OA `auth` tag, establishing
/// the agent→owner relationship in the relay's DB. Call this as the
/// *agent* connection; the tag must be signed by `owner_keys`.
pub async fn authenticate_with_nip_oa(
&mut self,
agent_keys: &Keys,
auth_tag: &Tag,
) -> Result<(), TestClientError> {
self.inner.authenticate(agent_keys, Some(auth_tag)).await?;
Ok(())
}
/// Sends a signed event to the relay and waits for the OK response.
pub async fn send_event(&mut self, event: Event) -> Result<OkResponse, TestClientError> {
Ok(self.inner.send_event(event).await?)
}
/// Builds and sends a text message event to `channel_id` using the given `kind`.
pub async fn send_text_message(
&mut self,
keys: &Keys,
channel_id: &str,
content: &str,
kind: u16,
) -> Result<OkResponse, TestClientError> {
let h_tag = Tag::parse(["h", channel_id])
.map_err(|e| TestClientError::EventBuilder(e.to_string()))?;
let event = EventBuilder::new(Kind::Custom(kind), content)
.tags([h_tag])
.sign_with_keys(keys)?;
self.send_event(event).await
}
/// Sends a REQ message to open a subscription with the given `sub_id` and `filters`.
pub async fn subscribe(
&mut self,
sub_id: &str,
filters: Vec<Filter>,
) -> Result<(), TestClientError> {
let mut msg: Vec<Value> = Vec::with_capacity(2 + filters.len());
msg.push(json!("REQ"));
msg.push(json!(sub_id));
for f in filters {
msg.push(serde_json::to_value(&f).map_err(WsClientError::Json)?);
}
self.inner.send_raw(&Value::Array(msg)).await?;
Ok(())
}
/// Sends a CLOSE message to cancel the subscription identified by `sub_id`.
pub async fn close_subscription(&mut self, sub_id: &str) -> Result<(), TestClientError> {
self.inner.send_raw(&json!(["CLOSE", sub_id])).await?;
Ok(())
}
/// Sends a raw JSON value as a WebSocket text frame.
pub async fn send_raw(&mut self, value: &serde_json::Value) -> Result<(), TestClientError> {
self.inner.send_raw(value).await?;
Ok(())
}
/// Receives the next relay message, waiting up to `timeout_dur`.
pub async fn recv_event(
&mut self,
timeout_dur: Duration,
) -> Result<RelayMessage, TestClientError> {
Ok(self.inner.next_event(timeout_dur).await?)
}
/// Collects the events for `sub_id` until EOSE is received, waiting up to
/// `timeout_dur`, keeping the first copy of each event id.
///
/// The relay registers a REQ's live subscription before it queries
/// history, so an event whose live fan-out runs late can arrive twice
/// before EOSE. Production clients drop the repeat, and so does this
/// helper. A test of the relay's own per-subscription dedup must read
/// frames with [`Self::recv_event`] instead.
pub async fn collect_until_eose(
&mut self,
sub_id: &str,
timeout_dur: Duration,
) -> Result<Vec<Event>, TestClientError> {
let deadline = tokio::time::Instant::now() + timeout_dur;
let mut events = Vec::new();
let mut seen = HashSet::new();
loop {
let remaining = deadline
.checked_duration_since(tokio::time::Instant::now())
.unwrap_or(Duration::ZERO);
if remaining.is_zero() {
return Err(TestClientError::Timeout);
}
match self.inner.next_event(remaining).await? {
RelayMessage::Event {
subscription_id,
event,
} if subscription_id == sub_id => {
if seen.insert(event.id) {
events.push(*event);
}
}
RelayMessage::Eose { subscription_id } if subscription_id == sub_id => {
return Ok(events);
}
_ => {}
}
}
}
/// Closes the WebSocket connection gracefully.
pub async fn disconnect(self) -> Result<(), TestClientError> {
self.inner.disconnect().await?;
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use nostr::{EventBuilder, Keys, Kind, RelayUrl, Tag};
#[test]
fn parse_relay_messages() {
struct Case {
json: &'static str,
check: fn(RelayMessage),
}
let cases = vec![
Case {
json: r#"["OK","abc123",true,""]"#,
check: |msg| match msg {
RelayMessage::Ok(ok) => {
assert_eq!(ok.event_id, "abc123");
assert!(ok.accepted);
assert_eq!(ok.message, "");
}
_ => panic!("expected Ok"),
},
},
Case {
json: r#"["OK","def456",false,"blocked: not authorized"]"#,
check: |msg| match msg {
RelayMessage::Ok(ok) => {
assert_eq!(ok.event_id, "def456");
assert!(!ok.accepted);
assert_eq!(ok.message, "blocked: not authorized");
}
_ => panic!("expected Ok"),
},
},
Case {
json: r#"["EOSE","sub1"]"#,
check: |msg| match msg {
RelayMessage::Eose { subscription_id } => assert_eq!(subscription_id, "sub1"),
_ => panic!("expected Eose"),
},
},
Case {
json: r#"["NOTICE","hello from relay"]"#,
check: |msg| match msg {
RelayMessage::Notice { message } => assert_eq!(message, "hello from relay"),
_ => panic!("expected Notice"),
},
},
Case {
json: r#"["AUTH","deadbeef1234"]"#,
check: |msg| match msg {
RelayMessage::Auth { challenge } => assert_eq!(challenge, "deadbeef1234"),
_ => panic!("expected Auth"),
},
},
Case {
json: r#"["CLOSED","sub2","auth-required: must authenticate"]"#,
check: |msg| match msg {
RelayMessage::Closed {
subscription_id,
message,
} => {
assert_eq!(subscription_id, "sub2");
assert_eq!(message, "auth-required: must authenticate");
}
_ => panic!("expected Closed"),
},
},
];
for case in cases {
let msg = parse_relay_message(case.json).expect(case.json);
(case.check)(msg);
}
}
#[test]
fn parse_unknown_message_type_errors() {
let result = parse_relay_message(r#"["UNKNOWN","data"]"#);
assert!(result.is_err());
}
#[test]
fn auth_event_has_relay_and_challenge_tags() {
let keys = Keys::generate();
let relay_url: RelayUrl = "ws://localhost:3000".parse().unwrap();
let event = EventBuilder::auth("test-challenge", relay_url)
.sign_with_keys(&keys)
.unwrap();
assert_eq!(event.kind, Kind::Authentication);
let tags: Vec<Vec<String>> = event
.tags
.iter()
.map(|t| t.as_slice().iter().map(|s| s.to_string()).collect())
.collect();
assert!(
tags.iter().any(|t| t.len() >= 2 && t[0] == "relay"),
"missing relay tag"
);
assert!(
tags.iter()
.any(|t| t.len() >= 2 && t[0] == "challenge" && t[1] == "test-challenge"),
"missing challenge tag"
);
}
#[test]
fn text_event_carries_h_tag() {
let keys = Keys::generate();
let channel_id = "my-channel-123";
let h_tag = Tag::parse(["h", channel_id]).unwrap();
let event = EventBuilder::new(Kind::Custom(9), "hello")
.tags([h_tag])
.sign_with_keys(&keys)
.unwrap();
assert_eq!(event.kind, Kind::Custom(9));
let tags: Vec<Vec<String>> = event
.tags
.iter()
.map(|t| t.as_slice().iter().map(|s| s.to_string()).collect())
.collect();
assert!(
tags.iter()
.any(|t| t.len() >= 2 && t[0] == "h" && t[1] == channel_id),
"missing h tag"
);
}
/// The relay registers a REQ's live subscription before its history
/// query, so an event whose fan-out task runs late arrives once live and
/// once from history, both before EOSE.
#[tokio::test]
async fn collect_until_eose_keeps_one_copy_of_each_event_id() {
use futures_util::SinkExt;
use tokio_tungstenite::tungstenite::Message;
let keys = Keys::generate();
let raced = EventBuilder::new(Kind::TextNote, "raced")
.sign_with_keys(&keys)
.unwrap();
let other = EventBuilder::new(Kind::TextNote, "other")
.sign_with_keys(&keys)
.unwrap();
let frames = [
json!(["EVENT", "sub", raced]),
json!(["EVENT", "elsewhere", other]),
json!(["EVENT", "sub", other]),
json!(["EVENT", "sub", raced]),
json!(["EOSE", "sub"]),
];
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("bind fake relay");
let address = listener.local_addr().expect("fake relay address");
let relay = tokio::spawn(async move {
let (stream, _) = listener.accept().await.expect("accept client");
let mut ws = tokio_tungstenite::accept_async(stream)
.await
.expect("fake relay handshake");
for frame in frames {
ws.send(Message::Text(frame.to_string().into()))
.await
.expect("send frame");
}
ws
});
let mut client = BuzzTestClient::connect_unauthenticated(&format!("ws://{address}"))
.await
.expect("connect to fake relay");
let events = client
.collect_until_eose("sub", Duration::from_secs(5))
.await
.expect("collect until EOSE");
let ids: Vec<_> = events.iter().map(|event| event.id).collect();
assert_eq!(ids, [raced.id, other.id]);
drop(relay.await.expect("join fake relay"));
}
}