mirror of
https://github.com/block/buzz.git
synced 2026-11-04 08:21:48 +00:00
Long threads (over ~100 replies) time out in desktop. The aux hop that
loads reactions and other e-tag references builds its filter as one
`tags @> …` term per id, joined with `OR`. Past about 100 ids the
planner switches to walking each partition's primary key backward,
expecting to stop early. When few rows actually match, it scans the
whole community. With a dirty GIN pending list, a 188-reply thread never
finishes.
### What changes
- **Set-based e-tag reads (`buzz-db`).** The aux e-tag query now puts
the id set behind a `MATERIALIZED` CTE and matches with `ANY`. Each
e-tag read also runs under a transaction-local `statement_timeout` of 20
s.
- **Terminal timeout contract (`buzz-relay`).** When a read hits the
deadline, the client gets a terminal timeout (`CLOSED` for REQ, the
timeout error for COUNT and search), not a retryable failure.
- **Subscription lifecycle (`buzz-relay`).** The per-connection
subscription map lock is now the lifecycle lock. Claims, replacement,
client `CLOSE`, owner-checked timeout and search cleanup, revoke, and
connection teardown all change the map, the registry, topic refcounts
and terminal frames while holding it. As a result, a timed-out
subscription's `CLOSED` goes out before any replacement with the same id
can claim it, and claims made after teardown are refused.
- **Dropped terminal frame disconnects (`buzz-relay`).** If the `CLOSED`
for a retired subscription (owner-checked timeout or channel eviction)
can't be queued, the relay cancels the connection instead of leaving the
client holding a subscription that will never receive fan-out. A
client-initiated `CLOSE` ack is exempt.
- **COUNT under the same deadline (`buzz-db`).** The fast COUNT path
runs e-tag counts under the same transaction-local 20 s cap as SELECT,
so WS COUNT and HTTP `/count` produce the terminal timeout too.
- **Replica reads surface the deadline (`buzz-db`).** When a
replica-routed read is cancelled by `statement_timeout` (SQLSTATE
`57014`), the error is returned instead of re-running the query on the
writer, which would run the same slow statement. Other replica errors
still fall back to the writer.
`PLANS/REPLICA_FULL_READ_ROUTING_DESIGN.md` records the exception.
- **Deadline handling (desktop).** Only the relay's `query timed out`
counts as a query deadline. A client-side `request timed out` stays
retryable. For one-shot reads a deadline is terminal: the global query
retry (budget 1), thread-replies retry and reaction hydration don't
re-send the same slow query. Reaction ids held back by a deadline are
released when the channel is opened again, so reopening retries them
while same-channel re-renders don't. Live subscriptions, in both the
renderer and the native archive-sync socket, keep treating a deadline
`CLOSED` as transient: they back off and resubscribe with the same
filter instead of being dropped.
- **Deadline handling (mobile).** Mobile recognizes the deadline on both
transports: HTTP `503 {"error":"query timed out"}` and the WS `CLOSED`
reason.
- The root `ProviderScope` retry policy returns no retry for deadline
errors and keeps Riverpod's default backoff for everything else.
- Background thread-summary recounts don't retry a deadline.
- Forum posts polling pauses while the last load settled on a deadline;
reopening the forum retries.
- A channel's newest-window load surfaces a deadline instead of falling
back to legacy WS history, and the channel list's batched queries return
"unavailable" on a deadline instead of fanning out to per-filter
fallbacks. An unavailable batch is not treated as empty. Known
last-message timestamps and unread state are kept, and unread catch-up
moves `lastMessageAt` forward (never backward) the same way live events
do, so a cold start with no known timestamp still gets one.
- A live subscription that gets a deadline `CLOSED` backs off and
resubscribes with the same filter. History requests still fail on it.
- Settled load failures show a Retry control, matching desktop's
thread-replies error card: a shared `LoadErrorView` on the channel
timeline, forum thread and forum posts list, and an icon button beside
the thread replies summary. While the thread query has no value, the
thread page shows the route snapshot plus live, cached and optimistic
replies, with the loading or error status still visible.
- Other re-sends (reconnect, foreground resume, live events, the unread
backstop, scroll) are not suppressed; each is bounded by the relay's 20
s statement deadline.
- **Long-thread seeder.** `just seed-long-thread [replies]` posts one
thread (default 187 replies) with some reactions to the local dev relay,
for reproducing long threads in the apps. It uses a throwaway key,
refuses non-loopback relays and ignores ambient `BUZZ_*` credentials.
Writes are paced to fit the stock relay rate limit.
**Lifecycle concurrency.** The lock serializes one connection's
subscription changes. Other connections can still be delayed by pressure
on the shared pubsub command queue, and by serial eviction loops waiting
behind a blocked connection. This PR does not claim strict isolation
between connections.
**Recovery vs. load.** Clients stop automatic retries of the same
one-shot read, but they don't remember deadlines across other triggers:
reconnect, resume, live events and the unread backstop can each send the
read again later, bounded by the 20 s statement deadline. A live
subscription that hits the deadline resubscribes with its original
filter under the existing 1–30 s backoff, and against a filter that
keeps timing out it retries at the 30 s cap indefinitely, so the worst
case is one bounded query per subscription every 30 s. `since` is
deliberately not advanced to now on resubscribe, because that would skip
events missed while the subscription was closed. Unchanged from `main`:
when mobile's channel history sync fails with cached messages available,
the cached messages are shown without the error or Retry control.
### Benchmark
Local reproduction on an isolated bench stack (not committed):
- PG16, migrations applied, one 188-reply thread plus 5M background
events in the same community.
- Each request replays the desktop thread-replies filter through the
relay, with `include_aux` on. Times are total ms for 3 runs, with a 60 s
client cap.
GIN pending-list state is controlled with `VACUUM` (clean) or by
refilling the pending list (dirty), and recorded per partition with
`pgstatginindex`. Dirty runs held 402 pending pages on every partition's
tags index at the start and end of each matrix. Clean runs held 0–1
pages (relay startup writes).
| build | GIN pending state | limit 10 | 50 | 100 | 200 (188 ids) |
|---|---|---|---|---|---|
| `main` | clean | 26 / 16 / 13 | 56 / 51 / 52 | 92 / 93 / 95 | 662 /
639 / 641 |
| `main` | dirty | 315 / 298 / 327 | 22141 / 15888 / 16769 | 28992 /
27819 / 28080 | timeout ×3 (>60 s) |
| this PR | clean | 35 / 21 / 14 | 57 / 49 / 48 | 82 / 80 / 75 | 153 /
150 / 135 |
| this PR | dirty | 290 / 254 / 251 | 1257 / 1204 / 1206 | 2432 / 2378 /
2382 | 4333 / 4706 / 5104 |
Baseline `origin/main` `b37e47721`; this PR at `5f2650f5f`, whose e-tag
read builders are identical to the current head. At head `1ad81d868`
against `origin/main` `15d44dcce`, limit 200 measured 145–179 ms clean
(`main` 683–715) and 4.41–4.51 s dirty (`main` timeout ×3).
Every completed cell returns the same signed event set: 209 events, 187
replies at limit 200. With `include_aux` off, every cell is ≤66 ms. The
thread-window loader (`select_aux`) is unchanged from `main`, and its
timings and captured plans match `main`'s.
Actual aux-hop statements captured from each relay binary with
`auto_explain` (analyze + buffers), execution ms:
| build | clean, 100 ids | clean, 188 ids | dirty, 100 ids | dirty, 188
ids |
|---|---|---|---|---|
| `main` (N-way `OR`) | 40 | 298 | 28785 | 62892 |
| this PR (`MATERIALIZED` fence + `ANY`) | 23 | 45 | 1992 | 3539 |
**What the fence changes**
- On `main`, the planner combines `ORDER BY created_at … LIMIT` with an
overestimated match count and walks each partition's primary key
backward, expecting to stop early. With few real matches it scans the
whole community: the dirty 188-id statement touches ~4.9M shared buffers
on `main` vs ~0.6M with the fence.
- The fence prevents that early-stop plan. Across the measured cases
(1–188 ids, both pending states), the planner used the GIN tags index.
The fence doesn't force GIN in general; a broad enough filter could
still get a different plan.
**Tradeoff:** the fence materializes every match before sorting. In this
bench small limits stayed within run-to-run noise of `main`. A single
hot id (5,000 reactions, limit 100) measured 30–34 ms unfenced vs 35 ms
fenced.
**Deadline:** each e-tag read (SELECT and COUNT) runs under a
transaction-local `statement_timeout` of 20 s. It only tightens: a
shorter operator timeout (`BUZZ_DB_STATEMENT_TIMEOUT_MS`) is kept. It's
a per-statement limit, not a per-request one, so a request that runs
several pages or hops can take longer than 20 s in total. The benchmark
does not force the deadline.
---------
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
Co-authored-by: Hayt <211b96e6a2b7f45fd4047988976c7bbbeeda0c15f3ae7b32eec20834b5a55118@buzz.block.builderlab.xyz>
461 lines
17 KiB
Rust
461 lines
17 KiB
Rust
//! Finite CLOSED recovery through the real native session and WebSocket loop.
|
|
use super::*;
|
|
|
|
fn fetch(
|
|
session: &Arc<RelaySession>,
|
|
timeout: Duration,
|
|
) -> tokio::task::JoinHandle<Result<Vec<Event>, String>> {
|
|
let session = Arc::clone(session);
|
|
tokio::spawn(async move {
|
|
session
|
|
.fetch_events(
|
|
serde_json::json!({"kinds": [9], "#h": ["unread"], "since": 42, "limit": 1000}),
|
|
timeout,
|
|
)
|
|
.await
|
|
})
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn quota_retry_preserves_events_without_duplicates_and_keeps_archive_live() {
|
|
let (url, mut frames, commands) = stub_relay().await;
|
|
let (session, mut archive) = start(url, Keys::generate(), None).await;
|
|
session.set_subscriptions(vec![probe_subscription()]).await;
|
|
assert_eq!(next_req(&mut frames, "archive").await, PROBE_ID);
|
|
let pending = fetch(&session, Duration::from_secs(10));
|
|
let id = next_req(&mut frames, "finite").await;
|
|
let event = EventBuilder::text_note("retained")
|
|
.sign_with_keys(&Keys::generate())
|
|
.unwrap();
|
|
let encoded = serde_json::to_value(&event).unwrap();
|
|
commands
|
|
.send(StubCommand::Event(id.clone(), encoded.clone()))
|
|
.await
|
|
.unwrap();
|
|
let refused = Instant::now();
|
|
commands
|
|
.send(StubCommand::Closed(
|
|
id.clone(),
|
|
"rate-limited: quota exceeded; retry in 1s".into(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
// Non-finite delivery is a receive-loop barrier, not a sleep or test-side retry setup.
|
|
commands
|
|
.send(StubCommand::Event(PROBE_ID.into(), encoded.clone()))
|
|
.await
|
|
.unwrap();
|
|
assert_eq!(
|
|
*tokio::time::timeout(Duration::from_secs(3), archive.recv())
|
|
.await
|
|
.unwrap()
|
|
.unwrap()
|
|
.event,
|
|
event
|
|
);
|
|
assert!(
|
|
!pending.is_finished(),
|
|
"quota refusal must retain the finite operation"
|
|
);
|
|
assert_eq!(next_req(&mut frames, "quota retry").await, id);
|
|
assert!(refused.elapsed() >= Duration::from_secs(1));
|
|
commands
|
|
.send(StubCommand::Event(id.clone(), encoded))
|
|
.await
|
|
.unwrap();
|
|
commands.send(StubCommand::Eose(id.clone())).await.unwrap();
|
|
assert_eq!(pending.await.unwrap().unwrap(), vec![event]);
|
|
assert_eq!(
|
|
next_frame(&mut frames, "finite CLOSE").await,
|
|
Frame::Close(id)
|
|
);
|
|
session.shutdown();
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn zero_hint_exhausts_three_retries_as_an_error() {
|
|
let (url, mut frames, commands) = stub_relay().await;
|
|
let (session, _archive) = start(url, Keys::generate(), None).await;
|
|
let pending = fetch(&session, Duration::from_secs(12));
|
|
let id = next_req(&mut frames, "finite").await;
|
|
for attempt in 0..4 {
|
|
commands
|
|
.send(StubCommand::Closed(
|
|
id.clone(),
|
|
"rate-limited: quota exceeded; retry in 0s".into(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
if attempt < 3 {
|
|
assert_eq!(next_req(&mut frames, "bounded retry").await, id);
|
|
}
|
|
}
|
|
let error = pending.await.unwrap().unwrap_err();
|
|
assert!(error.contains("rate-limited:"), "{error}");
|
|
assert!(session.requests.lock().await.is_empty());
|
|
session.shutdown();
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn terminal_refusal_and_deadline_do_not_become_empty_success() {
|
|
for message in [
|
|
"restricted: denied",
|
|
"error: transient",
|
|
"error: query timed out",
|
|
"rate-limited: quota exceeded; retry in 10s",
|
|
"rate-limited: quota exceeded",
|
|
"rate-limited: quota exceeded; retry in 18446744073709551615s",
|
|
] {
|
|
let (url, mut frames, commands) = stub_relay().await;
|
|
let (session, _archive) = start(url, Keys::generate(), None).await;
|
|
let pending = fetch(&session, Duration::from_millis(300));
|
|
let id = next_req(&mut frames, "finite").await;
|
|
commands
|
|
.send(StubCommand::Closed(id, message.into()))
|
|
.await
|
|
.unwrap();
|
|
let error = pending.await.unwrap().unwrap_err();
|
|
if message.starts_with("rate-limited:") {
|
|
assert!(error.contains("timed out"), "{error}");
|
|
} else {
|
|
assert!(error.contains(message), "{error}");
|
|
}
|
|
assert!(session.requests.lock().await.is_empty());
|
|
assert!(session.state.lock().await.transient.is_empty());
|
|
assert!(
|
|
frames.try_recv().is_err(),
|
|
"must not dispatch through denial/deadline"
|
|
);
|
|
session.shutdown();
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn dropping_finite_caller_closes_request_without_stopping_archive() {
|
|
let (url, mut frames, commands) = stub_relay().await;
|
|
let (session, mut archive) = start(url, Keys::generate(), None).await;
|
|
session.set_subscriptions(vec![probe_subscription()]).await;
|
|
assert_eq!(next_req(&mut frames, "archive").await, PROBE_ID);
|
|
let pending = fetch(&session, Duration::from_secs(10));
|
|
let id = next_req(&mut frames, "finite").await;
|
|
pending.abort();
|
|
assert!(pending.await.unwrap_err().is_cancelled());
|
|
assert_eq!(
|
|
next_frame(&mut frames, "cancelled finite CLOSE").await,
|
|
Frame::Close(id)
|
|
);
|
|
assert!(session.requests.lock().await.is_empty());
|
|
assert!(session.state.lock().await.transient.is_empty());
|
|
let event = EventBuilder::text_note("archive remains live")
|
|
.sign_with_keys(&Keys::generate())
|
|
.unwrap();
|
|
commands
|
|
.send(StubCommand::Event(
|
|
PROBE_ID.into(),
|
|
serde_json::to_value(&event).unwrap(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
assert_eq!(
|
|
*tokio::time::timeout(Duration::from_secs(3), archive.recv())
|
|
.await
|
|
.unwrap()
|
|
.unwrap()
|
|
.event,
|
|
event
|
|
);
|
|
session.shutdown();
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn reconnect_preserves_quota_hold_and_partial_events_do_not_reset_budget() {
|
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
|
let url = format!("ws://{}", listener.local_addr().unwrap());
|
|
let (seen_tx, mut seen_rx) = mpsc::channel(8);
|
|
let event = EventBuilder::text_note("partial replay")
|
|
.sign_with_keys(&Keys::generate())
|
|
.unwrap();
|
|
let server = tokio::spawn(async move {
|
|
let mut original: Option<serde_json::Value> = None;
|
|
let mut refused = std::time::Instant::now();
|
|
for connection in 0..2 {
|
|
let (tcp, _) = listener.accept().await.unwrap();
|
|
let mut ws = tokio_tungstenite::accept_async(tcp).await.unwrap();
|
|
ws.send(Message::Text(
|
|
serde_json::json!(["AUTH", "reconnect-test"])
|
|
.to_string()
|
|
.into(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
let mut requests = 0;
|
|
while let Some(Ok(Message::Text(text))) = ws.next().await {
|
|
let frame: serde_json::Value = serde_json::from_str(&text).unwrap();
|
|
match frame[0].as_str() {
|
|
Some("AUTH") => ws
|
|
.send(Message::Text(
|
|
serde_json::json!(["OK", frame[1]["id"], true, ""])
|
|
.to_string()
|
|
.into(),
|
|
))
|
|
.await
|
|
.unwrap(),
|
|
Some("REQ") => {
|
|
if let Some(expected) = &original {
|
|
assert_eq!(&frame, expected);
|
|
assert!(
|
|
refused.elapsed()
|
|
>= Duration::from_secs(if connection == 1 && requests == 0 {
|
|
2
|
|
} else {
|
|
1
|
|
}),
|
|
"reconnect must not erase due time"
|
|
);
|
|
} else {
|
|
original = Some(frame.clone());
|
|
}
|
|
seen_tx.send(frame.clone()).await.unwrap();
|
|
ws.send(Message::Text(
|
|
serde_json::json!(["EVENT", frame[1], event])
|
|
.to_string()
|
|
.into(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
let hint = if connection == 0 { 2 } else { 0 };
|
|
refused = std::time::Instant::now();
|
|
ws.send(Message::Text(
|
|
serde_json::json!([
|
|
"CLOSED",
|
|
frame[1],
|
|
format!("rate-limited: quota exceeded; retry in {hint}s")
|
|
])
|
|
.to_string()
|
|
.into(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
requests += 1;
|
|
if connection == 0 {
|
|
ws.close(None).await.unwrap();
|
|
break;
|
|
}
|
|
if requests == 3 {
|
|
return;
|
|
}
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
}
|
|
});
|
|
let (session, _archive) = start(url, Keys::generate(), None).await;
|
|
let pending = fetch(&session, Duration::from_secs(15));
|
|
let error = tokio::time::timeout(Duration::from_secs(12), pending)
|
|
.await
|
|
.unwrap()
|
|
.unwrap()
|
|
.unwrap_err();
|
|
assert!(error.contains("rate-limited:"), "{error}");
|
|
let mut count = 0;
|
|
while seen_rx.try_recv().is_ok() {
|
|
count += 1;
|
|
}
|
|
assert_eq!(
|
|
count, 4,
|
|
"initial request plus three retries across both sockets"
|
|
);
|
|
assert!(session.requests.lock().await.is_empty());
|
|
server.await.unwrap();
|
|
session.shutdown();
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn caller_drop_during_authentication_is_reclaimed_without_restarting_connect() {
|
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
|
let url = format!("ws://{}", listener.local_addr().unwrap());
|
|
let (connected_tx, connected) = oneshot::channel();
|
|
let (release, release_rx) = oneshot::channel();
|
|
let server = tokio::spawn(async move {
|
|
let (tcp, _) = listener.accept().await.unwrap();
|
|
let mut ws = tokio_tungstenite::accept_async(tcp).await.unwrap();
|
|
connected_tx.send(()).unwrap();
|
|
release_rx.await.unwrap();
|
|
ws.send(Message::Text(
|
|
serde_json::json!(["AUTH", "held-auth"]).to_string().into(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
let Some(Ok(Message::Text(text))) = ws.next().await else {
|
|
panic!("same authentication attempt must survive wake")
|
|
};
|
|
let auth: serde_json::Value = serde_json::from_str(&text).unwrap();
|
|
assert_eq!(auth[0], "AUTH");
|
|
ws.send(Message::Text(
|
|
serde_json::json!(["OK", auth[1]["id"], true, ""])
|
|
.to_string()
|
|
.into(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
let next = tokio::time::timeout(Duration::from_millis(200), ws.next()).await;
|
|
assert!(next.is_err(), "cancelled request must not be replayed");
|
|
});
|
|
let (session, _archive) = start(url, Keys::generate(), None).await;
|
|
connected.await.unwrap();
|
|
let pending = fetch(&session, Duration::from_secs(10));
|
|
tokio::time::timeout(Duration::from_secs(2), async {
|
|
loop {
|
|
if !session.state.lock().await.transient.is_empty() {
|
|
break;
|
|
}
|
|
tokio::task::yield_now().await;
|
|
}
|
|
})
|
|
.await
|
|
.unwrap();
|
|
pending.abort();
|
|
assert!(pending.await.unwrap_err().is_cancelled());
|
|
tokio::time::timeout(Duration::from_secs(2), async {
|
|
loop {
|
|
if session.requests.lock().await.is_empty()
|
|
&& session.state.lock().await.transient.is_empty()
|
|
{
|
|
break;
|
|
}
|
|
tokio::task::yield_now().await;
|
|
}
|
|
})
|
|
.await
|
|
.expect("cleanup must not require successful auth");
|
|
release.send(()).unwrap();
|
|
server.await.unwrap();
|
|
session.shutdown();
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn cancellation_and_stale_eose_during_hold_cannot_complete_or_retry() {
|
|
for shutdown in [false, true] {
|
|
let (url, mut frames, commands) = stub_relay().await;
|
|
let (session, mut archive) = start(url, Keys::generate(), None).await;
|
|
session.set_subscriptions(vec![probe_subscription()]).await;
|
|
assert_eq!(next_req(&mut frames, "archive").await, PROBE_ID);
|
|
let pending = fetch(&session, Duration::from_secs(10));
|
|
let id = next_req(&mut frames, "finite").await;
|
|
commands
|
|
.send(StubCommand::Closed(
|
|
id.clone(),
|
|
"rate-limited: quota exceeded; retry in 1s".into(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
commands.send(StubCommand::Eose(id)).await.unwrap();
|
|
let event = EventBuilder::text_note("barrier")
|
|
.sign_with_keys(&Keys::generate())
|
|
.unwrap();
|
|
commands
|
|
.send(StubCommand::Event(
|
|
PROBE_ID.into(),
|
|
serde_json::to_value(event).unwrap(),
|
|
))
|
|
.await
|
|
.unwrap();
|
|
tokio::time::timeout(Duration::from_secs(2), archive.recv())
|
|
.await
|
|
.unwrap()
|
|
.unwrap();
|
|
assert!(
|
|
!pending.is_finished(),
|
|
"stale EOSE must not finish refused history"
|
|
);
|
|
if shutdown {
|
|
session.shutdown();
|
|
assert!(pending.await.unwrap().unwrap_err().contains("cancelled"));
|
|
} else {
|
|
pending.abort();
|
|
assert!(pending.await.unwrap_err().is_cancelled());
|
|
}
|
|
tokio::time::timeout(Duration::from_secs(2), async {
|
|
loop {
|
|
if session.requests.lock().await.is_empty()
|
|
&& session.state.lock().await.transient.is_empty()
|
|
{
|
|
break;
|
|
}
|
|
tokio::task::yield_now().await;
|
|
}
|
|
})
|
|
.await
|
|
.unwrap();
|
|
assert!(
|
|
tokio::time::timeout(Duration::from_millis(1100), frames.recv())
|
|
.await
|
|
.ok()
|
|
.flatten()
|
|
.is_none(),
|
|
"cancelled retry must never reopen"
|
|
);
|
|
session.shutdown();
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn timeout_cleanup_remains_owned_when_caller_drops_under_state_contention() {
|
|
let (url, mut frames, _commands) = stub_relay().await;
|
|
let (session, _archive) = start(url, Keys::generate(), None).await;
|
|
let pending = fetch(&session, Duration::from_millis(150));
|
|
let id = next_req(&mut frames, "finite").await;
|
|
let state = session.state.lock().await;
|
|
// Hold the real cleanup lock beyond the caller's absolute deadline. Abort
|
|
// only after cleanup has been runnable, then let the existing loop reclaim.
|
|
tokio::time::sleep(Duration::from_millis(250)).await;
|
|
pending.abort();
|
|
assert!(pending.await.unwrap_err().is_cancelled());
|
|
drop(state);
|
|
assert_eq!(
|
|
next_frame(&mut frames, "cancelled cleanup CLOSE").await,
|
|
Frame::Close(id)
|
|
);
|
|
assert!(session.requests.lock().await.is_empty());
|
|
assert!(session.state.lock().await.transient.is_empty());
|
|
session.shutdown();
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn cancellation_during_registration_does_not_leave_transient_state() {
|
|
let (url, mut frames, _commands) = stub_relay().await;
|
|
let (session, _archive) = start(url, Keys::generate(), None).await;
|
|
session.set_subscriptions(vec![probe_subscription()]).await;
|
|
assert_eq!(next_req(&mut frames, "archive").await, PROBE_ID);
|
|
let state = session.state.lock().await;
|
|
let pending = fetch(&session, Duration::from_secs(10));
|
|
tokio::time::timeout(Duration::from_secs(2), async {
|
|
loop {
|
|
if !session.requests.lock().await.is_empty() {
|
|
break;
|
|
}
|
|
tokio::task::yield_now().await;
|
|
}
|
|
})
|
|
.await
|
|
.unwrap();
|
|
pending.abort();
|
|
assert!(pending.await.unwrap_err().is_cancelled());
|
|
drop(state);
|
|
tokio::time::timeout(Duration::from_secs(2), async {
|
|
loop {
|
|
if session.requests.lock().await.is_empty()
|
|
&& session.state.lock().await.transient.is_empty()
|
|
{
|
|
break;
|
|
}
|
|
tokio::task::yield_now().await;
|
|
}
|
|
})
|
|
.await
|
|
.unwrap();
|
|
assert!(frames.try_recv().is_err());
|
|
session.shutdown();
|
|
}
|