Files
buzz/.github/workflows/_ci-relay.yml
Bradley Axen a3a47e6294 feat(relay): add an unlisted channel setting (#8579)
🤖

## Summary

Sharing a Solo chat with a large channel today means adding every person
as a member, one at a time, and the app blocks the share past 100
people. The plan is to share it the way Google Docs does with "anyone at
the company with the link". Buzz's `open` visibility already gives that
kind of access: anyone in the community can read and join an open
channel. The problem is that an open channel also shows up in everyone's
channel list and search, and a private chat shouldn't appear there just
because its link was posted somewhere.

This PR adds a channel setting, `unlisted`. An **unlisted open channel**
can be read and joined by anyone who names it by id (from a shared
link). For non-members, it is left out of every read that does not name
it. Members see it as before. This is the relay side only; the buzz-app
change comes later. #8575 (batch adds) remains the path for small
audiences.

Unlisted is not an access control. Anyone who has the id can read the
channel, and that is intended.

## Solo flows this supports

Each user action is one event. The buzz-app side comes later; the relay
supports all of these now.

| Flow | Event | Result |
| --- | --- | --- |
| New Solo chat | kind:9007 `visibility private` + `["unlisted",
"true"]` | The chat starts unlisted, so it stays unlisted when any
client later opens it. |
| Share | One kind:9002: `visibility open` + `unlisted true` +
`join_role guest\|member`. Then post the link. | Anyone with the link
can read the chat by id, both history and live. Nobody is added as a
member, so the chat isn't in anyone's channel list. A non-member can't
post; `join_role` decides whether someone who joins can. |
| View | A REQ that names the chat by `#h`, or by `#d` for its metadata
| A non-member can read it. It is never listed for them. |
| Reply | kind:9021 join (the app sends it before a viewer's first
reply) | Joining makes them a member with the role from `join_role`, so
with `member` they can post. A kind:9022 leave makes them a viewer
again; the chat stays readable by id. |
| Unshare | kind:9002 `visibility private` | The existing open→private
eviction closes non-members' live subscriptions. New reads by
non-members are refused. |
| Convert to a channel | One kind:9002: `["t", ""]` + `unlisted false`,
plus `visibility` | Clears the Solo label. Non-members can now find it
in discovery. |

**Sidebar pins are not relay state.** Pin and unpin live in the app's
existing kind:30078 sidebar preferences, which sync across devices and
don't touch membership. Don't map them to join/leave: someone a chat was
shared with by member add is already a member, so a leave would remove
their access.

Your own Solo chats keep showing up, because you're their owner member,
and members see unlisted channels everywhere. Existing Solo chats aren't
backfilled. They get the flag when they're shared, because the share
event sets it.

**Open question, not built here:** someone who only views a shared chat
isn't a member. A mention of them doesn't reach their inbox (`#p:[me]`
without `#h`) and sends them no push.

## Behavior

| Reader | Reads that don't name the channel (kind:39000 without `#d`,
`#P` creator queries, messages, COUNT, NIP-50 search, artifact queries,
channel list) | Reads that name it (`#h`, or `#d` on 39000–39003) |
kind:9021 join |
| --- | --- | --- | --- |
| Member | Listed | Readable | n/a |
| Non-member, unlisted open channel | **Not listed** | Readable | Works;
`join_role` still decides the role |
| Non-member, private channel (unlisted or not) | Not listed | Not
readable | Refused |

- **Setting it:** kind:9002 `["unlisted", "true"]` / `["unlisted",
"false"]`. Owner or admin only, like `visibility`. A missing tag means
no change. It can be sent in the same event as `["visibility", "open"]`,
so a private chat becomes open and unlisted in one step. Any other value
is refused before the event is stored. The relay posts an
`unlisted_changed` system message and clears the accessible-channels
cache across nodes. Nobody loses access when a channel is unlisted, so
no subscriptions are evicted.
- **At creation:** kind:9007 also takes `["unlisted", "true"|"false"]`.
A missing tag means listed. Any other value is refused before the event
is stored, as on kind:9002. It is stored with the channel row in the
create's transaction, on both the client-UUID path and the
relay-chosen-UUID path.
- **kind:39000** carries a bare `["unlisted"]` flag while the setting is
on. It does not add NIP-29 `hidden`, so the metadata stays readable by
id.
- **NIP-11** advertises `buzz-unlisted-channels` in
`supported_extensions`. A relay without this PR would apply
`["visibility", "open"]` and ignore `unlisted`, which would list the
chat publicly. So buzz-app must check for the extension before it sends
that pair.
- **CLI:** `buzz channels update --channel <id> --unlisted true|false`
and `buzz channels create ... --unlisted`.
`buzz_sdk::build_update_channel` takes a new `unlisted: Option<bool>`
argument, and `buzz_sdk::build_create_channel` takes `unlisted: bool`,
which emits the tag only when true.

### Open question: does `unlisted` only mean something when the channel
is open?

**Decision: store it on its own; it only has an effect while the channel
is open.** Private channels are already invisible to non-members, so on
a private channel the flag does nothing. It is still stored and
published, for three reasons:
- a client can mark a Solo chat unlisted before opening it, or in the
same event that opens it, so the chat is never listed even for a moment;
- switching private → open → private → open keeps the setting;
- there is no cross-field rule in kind:9002 to validate or keep in sync.

kind:39000 publishes the flag as stored, whatever the visibility, so
clients can show the setting.

## How

The relay already has a "readable only when named" path for system
channels (NIP-CL). Unlisted open channels reuse it:

- **`get_accessible_channel_ids`**: the open half (the implicit scope of
a REQ that doesn't name a channel) adds `AND NOT unlisted`. The member
half is unchanged, so members still have the channel.
- **`get_readable_named_only_channels`**: also returns unlisted open
channels of the legacy types. Every reader of that set (REQ, COUNT,
NIP-50 search, HTTP `/query` and `/count`, thread windows, workflow run
reads) adds a named-only channel only to a filter that names it, through
`filter_channel_scope`. An unlisted channel's type is `stream`/`forum`,
so the `#t:["system"]`/`#P` system opt-in never adds it.
- **`get_accessible_channels`** (the channel list): open channels are
listed to non-members only when not unlisted.
- **`query_artifacts`**: an artifact in an unlisted open channel reaches
a non-member only when the query's `#h` names that channel.
- **Live fan-out:** no change was needed. A global subscription never
receives channel-scoped events, kind:39000 included (`fan_out_scoped`).
A channel-scoped subscription has to name the channel, and that goes
through the same named-only authorization.

**Left alone on purpose:** access checks that read a named channel or
event keep treating an unlisted channel as open. These are
`handle_join_request` (kind:9021), the ingest write gate,
operator-listener mention delivery (`load_deliverable_event`), personal
read-state writes, and push matching. Push matching still requires
membership, so non-members get no pushes from an unlisted channel.

**Named reads checked:** these callers use the implicit channel set
(`get_accessible_channel_ids`, its cache, `get_accessible_channels`).
Every one that reads a channel the request names also adds the
named-only set, so a non-member can still read an unlisted open chat by
id. That covers WS REQ (history, live registration, NIP-50 search), WS
COUNT, HTTP `/query` and `/count`, thread windows and workflow run
reads. Other named reads don't use the implicit set: they check
visibility or membership, so an unlisted open chat counts as open. Those
are live fan-out, the ingest write gate, typing, kind:9021 join, huddle
audio join, operator-listener delivery, the workflow sink and read-state
writes. Feed queries (mentions, needs-action, activity) and the
`/buzz/v1/me/sidebar` accessory API are lists, so leaving out chats you
only view is intended. Media reads don't check channels. `buzz-auth`'s
`ChannelAccessChecker` has no production implementation.

## Database (docs/database.md checklist)

- **Phase:** expand only. The migration adds a column with a constant
default, and the same release ships the code that reads and writes it.
No backfill or contract step follows.
- No trigger, function, procedure, rule or foreign key is added.
- No migration changes existing rows. `ADD COLUMN ... BOOLEAN NOT NULL
DEFAULT FALSE` stores the constant default in the catalog, so the table
is not rewritten.
- Every location in Rule 4 is updated:
`migrations/0069_channel_unlisted.sql`,
`schema/tables/public/channels.sql`, and the migration count. No seeds,
types or reconcile changes are needed.
`admin_schema_parity_between_desired_state_and_migrations` now compares
`channels` columns between the pgschema desired state and the migrated
database. I failed it once by removing the column from `channels.sql`.
Only columns are compared for `channels`: on current `main`, pgschema
already re-emits `idx_channels_id_live` without its `INCLUDE
(community_id)`, an index drift that predates this PR and is noted in
the test.
- **Currently deployed relay against the new schema:** it never selects
or writes `unlisted`, and every insert gets the default. All
channel-record queries name their columns. Nobody can set the flag until
this relay ships, because the old relay refuses a kind:9002 whose only
tag is `unlisted`.
- The migration starts with `SET LOCAL lock_timeout = '5s'`. No index is
added: the new predicates ride the existing `(community_id, visibility)`
index.

## Rollout

1. Deploy. Every channel stays listed, so nothing changes.
2. **Don't unlist any channel while an old relay is still running.** An
old relay ignores the column and lists the channel to everyone. The
NIP-11 extension lets clients gate on this. Unlisted isn't an access
boundary, so a mixed fleet only exposes the channel in listings for a
while.
3. **Rollback:** an older relay ignores the column, so unlisted channels
show up as ordinary open channels again. Nothing breaks.

## Tests

Postgres (`buzz-db`, discovered by the Postgres lane):
- `unlisted_open_channel_is_named_only_for_non_members`: sets the flag
through `update_channel`. Then checks that a non-member doesn't get it
from `get_accessible_channel_ids` or `get_accessible_channels`, and does
get it from `get_readable_named_only_channels`; that a member still
lists it; that `put_member` (the kind:9021 path) lets the non-member
join, after which they list it; and that listing it again shows it to a
new reader.
- `unlisted_private_channel_stays_private_and_opens_unlisted`: a private
unlisted channel is not readable by name, and opening it keeps the flag.
- `group_metadata_publishes_the_stored_unlisted_flag`, plus the unit
test `publishes_the_unlisted_flag_as_stored`.
- `unlisted_channel_artifacts_are_read_by_name_for_non_members`.
- `channel_created_unlisted_is_never_listed`: creating through
`create_channel_with_event` (the kind:9007 path) and
`create_labeled_channel` stores the flag, and the channel is never
listed to a non-member.
- Mutation-checked: each of `NOT unlisted` in the implicit scope, `NOT
c.unlisted` in the channel list, the unlisted branch of the named-only
query, its `visibility = 'open'` guard, and the artifact condition was
removed one at a time, and each removal failed at least one of these
tests.

Relay E2E (`test_nip29_unlisted_channel_wire`, added to the CI relay E2E
step), against a local relay built from this branch:
- A non-member or an unknown value is refused.
- kind:39000 by `#d` carries `unlisted` and `public`, and not `hidden`.
- kind:39000 discovery without `#d`: non-members don't see the channel,
members do.
- History and NIP-50 search: unnamed, a non-member doesn't see the
channel's messages; with `#h`, they do; a member sees them unnamed.
- A kind:9021 join works, and the joiner then lists the channel.
- Listing it again shows it to everyone.
- A reader whose channel set was cached before each change sees the
change at once. I ran this against a relay with the cache invalidation
removed, and it failed.

Relay E2E (`test_nip29_solo_share_and_convert_wire`, added to the CI
relay E2E step), each flow as a single event:
- kind:9007 with an unknown `unlisted` value is refused. A chat created
with `unlisted true` publishes the flag, and stays unlisted after a
plain `visibility open`.
- Share (open + unlisted + `join_role guest` in one kind:9002): a
non-member doesn't list the chat. They read it by `#h`, both history and
a live message, and can't post. A kind:9021 join lists it for them as a
read-only guest. A kind:9022 leave unlists it again, and it stays
readable by `#h`.
- Unshare (`visibility private`): the viewer's live subscription gets
CLOSED, and a new `#h` read is refused.
- Convert (`["t", ""]` + `unlisted false`): the Solo label is cleared,
and a non-member finds the chat in discovery.
- I ran this against a relay that dropped the kind:9007 flag, and it
failed.

Also run: `cargo test` for buzz-db, buzz-sdk, buzz-cli, buzz-core,
buzz-relay and buzz-test-client; the Postgres lane for buzz-db and
buzz-relay (1045 tests); and the existing unlisted, announce, departure,
guest, role-change and NIP-43 snapshot wire tests. In the lane, two
tests failed on pool timeouts
(`resend_validated_after_the_first_commit_is_a_duplicate`,
`edit_replaces_and_clears_labels`) and passed when rerun alone. An
earlier run had the same kind of flake:
`routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold` (a
timing budget) and `resent_edit_is_a_duplicate` (a pool timeout). In a
full local `e2e_relay` run, three tests outside CI's list also failed
(`test_invite_*` twice,
`test_unarchive_emits_member_added_notification`). They fail the same
way on a relay built from `main`.

## Not in this PR

- The buzz-app follow-up: support open Solo chats, the join-from-card
view, the large-audience prompt, and gating on `buzz-unlisted-channels`.
- A NIP-CL update. The behavior is documented in NOSTR.md ("Unlisted
channels").





buzz-review-completed

---------

Signed-off-by: Bradley Axen <baxen@squareup.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: ART <ea39c1c8acf57180161ef4ba4ed68abf8589f5291956b909a5aa991fc66664ca@buzz.block.builderlab.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mbot <3754f8729004d95654c46dbab3129e4ab9ef05cc2534e2a3fbfc155983bd637b@buzz.block.builderlab.xyz>
Co-authored-by: Luke Tornquist <tornquist@squareup.com>
Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
2026-10-26 14:03:07 -04:00

677 lines
31 KiB
YAML

name: CI / Relay and PostgreSQL
on:
workflow_call:
inputs:
rust:
required: true
type: boolean
desktop:
required: true
type: boolean
desktop_rust:
required: true
type: boolean
lane:
required: true
type: string
outputs:
desktop_e2e_relay_result:
value: ${{ jobs.results.outputs.desktop_e2e_relay_result }}
desktop_e2e_integration_result:
value: ${{ jobs.results.outputs.desktop_e2e_integration_result }}
backend_integration_result:
value: ${{ jobs.results.outputs.backend_integration_result }}
relay_e2e_result:
value: ${{ jobs.results.outputs.relay_e2e_result }}
postgres_tests_result:
value: ${{ jobs.results.outputs.postgres_tests_result }}
env:
CARGO_TERM_COLOR: always
BUZZ_TEST_POSTGRES_PASSWORD: buzz_dev
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/.cache/ms-playwright
jobs:
# The relay binaries and the two nextest archives compile independently, so
# they build in parallel; consumers download every desktop-e2e-relay-*
# artifact into target/ci as before.
desktop-e2e-relay:
name: Desktop E2E Relay (${{ matrix.artifact }})
runs-on: ubuntu-latest
timeout-minutes: 45
if: inputs.lane == 'artifacts' && (github.event_name == 'push' || inputs.desktop || inputs.desktop_rust || inputs.rust)
strategy:
fail-fast: false
matrix:
include:
- artifact: relay-binaries
paths: |
target/ci/buzz-relay
target/ci/git-credential-nostr
- artifact: backend-integration-tests
paths: target/ci/backend-integration-tests.tar.zst
- artifact: postgres-tests
paths: target/ci/postgres-tests.tar.zst
permissions:
contents: read
env:
SCCACHE_GHA_ENABLED: "true"
SCCACHE_GHA_RW_MODE: ${{ (github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.pull_request.number == 5224)) && 'READ_WRITE' || 'READ_ONLY' }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
# Reuse each artifact when none of its inputs changed (desktop-only PRs
# hit this every time). The key covers everything they embed, including
# migrations via sqlx migrate!.
- name: Restore relay artifacts cache
id: relay-artifacts-cache
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ matrix.paths }}
key: relay-artifacts-${{ matrix.artifact }}-${{ runner.os }}-${{ hashFiles('crates/**', 'migrations/**', 'Dockerfile', 'Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', '.cargo/config.toml', '.config/nextest.toml', 'scripts/postgres-test-*.sh', 'scripts/check-postgres-test-discovery.py', '.github/workflows/ci.yml', '.github/workflows/_ci-*.yml') }}
- uses: rui314/setup-mold@7e4f20ad28a2e8ca6fd0892ccf72e2abb706b9c3 # v1
if: steps.relay-artifacts-cache.outputs.cache-hit != 'true'
# All lanes restore the shared Rust cache; only the postgres-tests lane,
# which compiles the widest test graph, writes it so a faster lane
# cannot save a narrower target dir under the same key.
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32
if: steps.relay-artifacts-cache.outputs.cache-hit != 'true'
with:
workspaces: |
.
desktop/src-tauri
save-if: ${{ github.event_name != 'pull_request' && matrix.artifact == 'postgres-tests' }}
# Cache rustc outputs for unchanged workspace crates. Trusted pushes write;
# the bounded PR 5224 trial writes only to its isolated merge-ref scope.
- name: Set up sccache
if: steps.relay-artifacts-cache.outputs.cache-hit != 'true'
uses: Mozilla-Actions/sccache-action@fc920bf0ec8de6ee65d409111f7ec508035751ba # v0.0.11 # zizmor: ignore[cache-poisoning] Bounded trial: only PR 5224 writes to its isolated merge-ref scope; trusted pushes retain production writes.
with:
version: v0.16.0
- name: Install cargo-nextest
if: steps.relay-artifacts-cache.outputs.cache-hit != 'true' && matrix.artifact != 'relay-binaries'
uses: taiki-e/install-action@0fd46367812ee04360509b4169d9f659d6892bb2 # v2.79.15
with:
tool: cargo-nextest@0.9.136
- name: Build relay binaries
if: steps.relay-artifacts-cache.outputs.cache-hit != 'true' && matrix.artifact == 'relay-binaries'
env:
RUSTC_WRAPPER: sccache
run: cargo build --profile ci -p buzz-relay -p git-credential-nostr
- name: Build backend integration test archive
if: steps.relay-artifacts-cache.outputs.cache-hit != 'true' && matrix.artifact == 'backend-integration-tests'
env:
RUSTC_WRAPPER: sccache
run: |
cargo nextest archive \
--cargo-profile ci \
-p buzz-db \
-p buzz-relay \
-p buzz-test-client \
--lib \
--test e2e_event_reminder \
--test e2e_author_only_deletion \
--test e2e_existence_oracle \
--archive-file target/ci/backend-integration-tests.tar.zst
- name: Build PostgreSQL test archive
if: steps.relay-artifacts-cache.outputs.cache-hit != 'true' && matrix.artifact == 'postgres-tests'
env:
RUSTC_WRAPPER: sccache
run: |
postgres_package_args=()
while IFS= read -r package; do
postgres_package_args+=(-p "$package")
done < <(scripts/postgres-test-packages.sh)
if [[ "${#postgres_package_args[@]}" -eq 0 ]]; then
echo "no PostgreSQL test packages were discovered" >&2
exit 1
fi
cargo nextest archive \
--cargo-profile ci \
"${postgres_package_args[@]}" \
--lib \
--tests \
--archive-file target/ci/postgres-tests.tar.zst
- name: Save relay artifacts cache
# PR-scoped exact-source entries cannot warm main or other PRs and churn
# the shared cache pool. sccache provides read-only PR reuse instead.
if: steps.relay-artifacts-cache.outputs.cache-hit != 'true' && github.event_name == 'push'
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ matrix.paths }}
key: relay-artifacts-${{ matrix.artifact }}-${{ runner.os }}-${{ hashFiles('crates/**', 'migrations/**', 'Dockerfile', 'Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', '.cargo/config.toml', '.config/nextest.toml', 'scripts/postgres-test-*.sh', 'scripts/check-postgres-test-discovery.py', '.github/workflows/ci.yml', '.github/workflows/_ci-*.yml') }}
- name: Upload relay artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: desktop-e2e-relay-${{ matrix.artifact }}
path: ${{ matrix.paths }}
if-no-files-found: error
retention-days: 1
postgres-tests:
name: PostgreSQL Tests
runs-on: ubuntu-latest
timeout-minutes: 10
if: inputs.lane == 'postgres' && (github.event_name == 'push' || inputs.rust)
permissions:
contents: read
services:
postgres:
image: postgres:16
env:
POSTGRES_USER: buzz
POSTGRES_PASSWORD: ${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}
POSTGRES_DB: postgres
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U buzz -d postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 12
redis:
image: redis:7
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 5s
--health-timeout 5s
--health-retries 12
env:
PGHOST: localhost
PGPORT: "5432"
PGUSER: buzz
PG_BIN_DIR: /usr/bin
REDIS_URL: redis://localhost:6379
PGSCHEMA_PLAN_HOST: localhost
PGSCHEMA_PLAN_PORT: "5432"
PGSCHEMA_PLAN_USER: buzz
PGSCHEMA_PLAN_PASSWORD: buzz_dev
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- name: Tune disposable PostgreSQL
env:
PGPASSWORD: ${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}
run: |
# Database-per-test cloning forces checkpoints. Durability is redundant
# for this disposable service and makes runtime depend on runner disk I/O.
psql --dbname postgres --set ON_ERROR_STOP=1 <<'SQL'
ALTER SYSTEM SET fsync = off;
ALTER SYSTEM SET full_page_writes = off;
ALTER SYSTEM SET synchronous_commit = off;
SELECT pg_reload_conf();
SQL
psql --dbname postgres --tuples-only --no-align --command \
"SELECT bool_and(setting = 'off') FROM pg_settings WHERE name IN ('fsync', 'full_page_writes', 'synchronous_commit')" \
| grep --fixed-strings --line-regexp t
- name: Install cargo-nextest
uses: taiki-e/install-action@0fd46367812ee04360509b4169d9f659d6892bb2 # v2.79.15
with:
tool: cargo-nextest@0.9.136
- name: Download backend test archive
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: desktop-e2e-relay-*
merge-multiple: true
path: target/ci
- name: Restore relay executable permission
run: chmod +x ./target/ci/buzz-relay
- name: PostgreSQL-backed tests
env:
BUZZ_POSTGRES_ADMIN_URL: postgres://buzz:${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}@localhost:5432/postgres
PGPASSWORD: ${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}
run: |
scripts/postgres-test-run.sh \
--archive-file target/ci/postgres-tests.tar.zst
desktop-e2e-integration-shard:
name: Desktop E2E Integration (${{ matrix.shard }}/${{ strategy.job-total }})
runs-on: ubuntu-latest
# A shard needs ~10m; Playwright install-deps is bounded by
# ci-apt-retry.sh (~25m worst case), so a degraded mirror fails with its
# own error instead of the job timeout.
timeout-minutes: 40
if: inputs.lane == 'required' && (github.event_name == 'push' || inputs.desktop || inputs.desktop_rust || inputs.rust)
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3]
permissions:
contents: read
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- name: Start integration services
run: |
for attempt in 1 2 3; do
if docker compose up -d postgres redis minio minio-init; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "docker compose up failed after 3 attempts" >&2
exit 1
fi
echo "docker compose up failed (attempt $attempt), retrying in $((attempt * 5))s..." >&2
sleep $((attempt * 5))
done
- name: Get pnpm store directory
id: pnpm-cache
run: echo "STORE_PATH=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"
- name: Restore pnpm store cache
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: pnpm-${{ runner.os }}-
- name: Install desktop dependencies
run: just desktop-install-ci
- name: Get Playwright version
id: pw-version
run: echo "version=$(cd desktop && node -e "console.log(require('@playwright/test/package.json').version)")" >> "$GITHUB_OUTPUT"
- name: Restore Playwright browser cache
id: playwright-cache
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ env.PLAYWRIGHT_BROWSERS_PATH }}
key: playwright-${{ runner.os }}-${{ steps.pw-version.outputs.version }}
- name: Install Playwright Chromium
if: steps.playwright-cache.outputs.cache-hit != 'true'
run: cd desktop && pnpm exec playwright install chromium
- name: Install Playwright system dependencies
run: cd desktop && ../scripts/ci-apt-retry.sh pnpm exec playwright install-deps chromium
- name: Save Playwright browser cache
if: steps.playwright-cache.outputs.cache-hit != 'true' && github.event_name == 'push' && matrix.shard == 1
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ env.PLAYWRIGHT_BROWSERS_PATH }}
key: playwright-${{ runner.os }}-${{ steps.pw-version.outputs.version }}
- name: Desktop E2E build
run: pnpm -C desktop build:e2e
- name: Wait for integration services
run: |
wait_healthy() {
local service="$1"
local container="$2"
for attempt in $(seq 1 60); do
status=$(docker inspect --format='{{.State.Health.Status}}' "${container}" 2>/dev/null || echo "not_found")
if [ "${status}" = "healthy" ]; then
echo "${service} is healthy"
return 0
fi
sleep 2
done
docker logs "${container}" || true
return 1
}
wait_healthy "Postgres" "buzz-postgres"
wait_healthy "Redis" "buzz-redis"
wait_healthy "MinIO" "buzz-minio"
- name: Download relay binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: desktop-e2e-relay-*
merge-multiple: true
path: target/ci
- name: Apply schema and seed deployment community
# MT: the relay resolves each request's tenant from the communities host
# map and fails closed on an unmapped host. The channel reconciler binds
# the deployment community ONCE at boot (outside its retry loop) and
# exits permanently on an unmapped host, so the 'localhost:3000'
# community MUST exist before the relay starts — the retry loop only
# handles late-seeded channels, not a late-seeded community. The relay
# migrates at boot via BUZZ_AUTO_MIGRATE, but that's too late for the
# pre-boot seed, so apply the schema here first (then drop AUTO_MIGRATE
# below). lower(host) is the unique index → ON CONFLICT target. psql
# isn't on PATH in hermit → exec into the buzz-postgres container.
env:
PGHOST: localhost
PGPORT: "5432"
PGUSER: buzz
PGPASSWORD: buzz_dev
PGDATABASE: buzz
# Use the already-running docker postgres for desired-state planning instead of
# downloading an embedded Postgres from Maven Central (transient-fetch flake source).
PGSCHEMA_PLAN_HOST: localhost
PGSCHEMA_PLAN_PORT: "5432"
PGSCHEMA_PLAN_DB: buzz
PGSCHEMA_PLAN_USER: buzz
PGSCHEMA_PLAN_PASSWORD: buzz_dev
run: |
./bin/pgschema apply --file schema/schema.sql --auto-approve
docker exec -i -e PGPASSWORD=buzz_dev buzz-postgres \
psql -U buzz -d buzz -v ON_ERROR_STOP=1 < scripts/reconcile-schema-after-pgschema.sql
docker exec -e PGPASSWORD=buzz_dev buzz-postgres \
psql -U buzz -d buzz -qtA -c "
INSERT INTO communities (id, host)
VALUES ('00000000-0000-4000-8000-00000000c0de', 'localhost:3000')
ON CONFLICT (lower(host)) DO NOTHING
;"
- name: Start relay
run: |
chmod +x ./target/ci/buzz-relay
nohup env \
DATABASE_URL="postgres://buzz:${BUZZ_TEST_POSTGRES_PASSWORD}@localhost:5432/buzz" \
REDIS_URL=redis://localhost:6379 \
RELAY_URL=ws://localhost:3000 \
BUZZ_BIND_ADDR=0.0.0.0:3000 \
BUZZ_RELAY_PRIVATE_KEY="$(openssl rand -hex 32)" \
BUZZ_REQUIRE_AUTH_TOKEN=false \
BUZZ_RECONCILE_CHANNELS=true \
BUZZ_RATE_LIMIT_HUMAN_MESSAGES_PER_MIN=100000 \
BUZZ_RATE_LIMIT_HUMAN_API_CALLS_PER_MIN=100000 \
BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC=10000 \
BUZZ_GIT_PROBE_WRITERS=8 \
SPROUT_REMINDER_SCHEDULER_INTERVAL_SECS=1 \
./target/ci/buzz-relay > /tmp/buzz-relay.log 2>&1 &
echo $! > /tmp/buzz-relay.pid
for attempt in $(seq 1 60); do
if ! kill -0 "$(cat /tmp/buzz-relay.pid)" 2>/dev/null; then
cat /tmp/buzz-relay.log
exit 1
fi
status_code=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:3000/_readiness || true)
if [ "${status_code}" = "200" ]; then
exit 0
fi
sleep 1
done
cat /tmp/buzz-relay.log
exit 1
- name: Seed desktop e2e data
run: bash scripts/setup-desktop-test-data.sh
- name: Desktop relay-backed e2e
run: cd desktop && pnpm exec playwright test --project=integration --shard=${{ matrix.shard }}/${{ strategy.job-total }}
- name: Summarize flaky tests
if: ${{ !cancelled() }}
run: node scripts/summarize-flaky-tests.mjs playwright-report.json "Desktop E2E Integration (${{ matrix.shard }}/${{ strategy.job-total }})"
working-directory: desktop
- name: Upload desktop integration artifacts
if: ${{ !cancelled() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: desktop-e2e-integration-artifacts-${{ matrix.shard }}
path: |
desktop/playwright-report
desktop/playwright-report.json
desktop/test-results
/tmp/buzz-relay.log
if-no-files-found: ignore
retention-days: 7
- name: Save pnpm store cache
if: github.event_name == 'push'
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
desktop-e2e-integration:
name: Desktop E2E Integration
runs-on: ubuntu-latest
timeout-minutes: 5
needs: [desktop-e2e-integration-shard]
if: always() && inputs.lane == 'required' && (github.event_name == 'push' || inputs.desktop || inputs.desktop_rust || inputs.rust)
permissions:
contents: read
steps:
- name: Check integration shards
run: |
if [ "${{ needs.desktop-e2e-integration-shard.result }}" != "success" ]; then
echo "Desktop E2E Integration shards finished with: ${{ needs.desktop-e2e-integration-shard.result }}"
exit 1
fi
echo "Desktop E2E Integration shards passed"
backend-integration:
name: Backend Integration (relay e2e)
runs-on: ubuntu-latest
timeout-minutes: 20
if: inputs.lane == 'required' && (github.event_name == 'push' || inputs.rust)
permissions:
contents: read
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- name: Install cargo-nextest
uses: taiki-e/install-action@0fd46367812ee04360509b4169d9f659d6892bb2 # v2.79.15
with:
tool: cargo-nextest@0.9.136
- name: Start integration services
run: |
for attempt in 1 2 3; do
if docker compose up -d postgres redis minio minio-init; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "docker compose up failed after 3 attempts" >&2
exit 1
fi
echo "docker compose up failed (attempt $attempt), retrying in $((attempt * 5))s..." >&2
sleep $((attempt * 5))
done
- name: Wait for integration services
run: |
wait_healthy() {
local service="$1"
local container="$2"
for attempt in $(seq 1 60); do
status=$(docker inspect --format='{{.State.Health.Status}}' "${container}" 2>/dev/null || echo "not_found")
if [ "${status}" = "healthy" ]; then
echo "${service} is healthy"
return 0
fi
sleep 2
done
docker logs "${container}" || true
return 1
}
wait_healthy "Postgres" "buzz-postgres"
wait_healthy "Redis" "buzz-redis"
wait_healthy "MinIO" "buzz-minio"
- name: Download relay artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: desktop-e2e-relay-*
merge-multiple: true
path: target/ci
- name: Apply schema and seed deployment community
# MT: the relay resolves each request's tenant from the communities host
# map and fails closed on an unmapped host. The reminder scheduler binds
# the deployment community ONCE at boot and exits permanently on an
# unmapped host (no retry, unlike the channel reconciler), so the
# 'localhost:3000' community MUST exist before the relay starts — seeding
# after boot leaves the scheduler dead. The relay migrates at boot via
# BUZZ_AUTO_MIGRATE, but that's too late for the pre-boot seed, so apply
# the schema here first (then drop AUTO_MIGRATE below). lower(host) is the
# unique index → ON CONFLICT target. psql isn't on PATH in hermit → exec
# into the buzz-postgres container.
env:
PGHOST: localhost
PGPORT: "5432"
PGUSER: buzz
PGPASSWORD: buzz_dev
PGDATABASE: buzz
# Use the already-running docker postgres for desired-state planning instead of
# downloading an embedded Postgres from Maven Central (transient-fetch flake source).
PGSCHEMA_PLAN_HOST: localhost
PGSCHEMA_PLAN_PORT: "5432"
PGSCHEMA_PLAN_DB: buzz
PGSCHEMA_PLAN_USER: buzz
PGSCHEMA_PLAN_PASSWORD: buzz_dev
run: |
./bin/pgschema apply --file schema/schema.sql --auto-approve
docker exec -i -e PGPASSWORD=buzz_dev buzz-postgres \
psql -U buzz -d buzz -v ON_ERROR_STOP=1 < scripts/reconcile-schema-after-pgschema.sql
docker exec -e PGPASSWORD=buzz_dev buzz-postgres \
psql -U buzz -d buzz -qtA -c "
INSERT INTO communities (id, host)
VALUES ('00000000-0000-4000-8000-00000000c0de', 'localhost:3000')
ON CONFLICT (lower(host)) DO NOTHING
;"
- name: Git default-branch route and clone regressions
env:
# The preceding step already applied and reconciled schema/schema.sql.
BUZZ_TEST_SCHEMA_MODE: desired
BUZZ_TEST_DATABASE_URL: postgres://buzz:${{ env.BUZZ_TEST_POSTGRES_PASSWORD }}@localhost:5432/buzz
BUZZ_TEST_REDIS_URL: redis://localhost:6379
BUZZ_TEST_S3_ENDPOINT: http://localhost:9000
BUZZ_TEST_S3_BUCKET: buzz-media
run: |
cargo nextest run \
--archive-file target/ci/backend-integration-tests.tar.zst \
-E 'package(buzz-relay) and test(/api::git::settings::tests::external_infra::/)' \
--run-ignored ignored-only
- name: Relay quota accounting and rejection regressions
env:
BUZZ_TEST_REDIS_URL: redis://localhost:6379
run: |
cargo nextest run \
--archive-file target/ci/backend-integration-tests.tar.zst \
-E 'package(buzz-relay) and (test(/^rejection::/) or test(/^admission::/))' \
--run-ignored all
- name: NIP-FI cross-pod command replay regressions
env:
BUZZ_TEST_REDIS_URL: redis://localhost:6379
run: |
cargo nextest run \
--archive-file target/ci/backend-integration-tests.tar.zst \
-E 'package(buzz-relay) and test(/^api::nip_fi::route_integration_tests::external_infra_redis::/)' \
--run-ignored ignored-only
- name: Workflow message provenance unit tests
# The relay's workflow_sink suite is not selected by the infra-free
# unit job. Its ignored database cases run in the isolated PostgreSQL
# lane; keep the pure provenance cases covered here without duplication.
run: |
cargo nextest run \
--archive-file target/ci/backend-integration-tests.tar.zst \
-E 'package(buzz-relay) and test(/workflow_sink/)'
- name: Start relay
run: |
chmod +x ./target/ci/buzz-relay
nohup env \
DATABASE_URL="postgres://buzz:${BUZZ_TEST_POSTGRES_PASSWORD}@localhost:5432/buzz" \
REDIS_URL=redis://localhost:6379 \
RELAY_URL=ws://localhost:3000 \
BUZZ_BIND_ADDR=0.0.0.0:3000 \
BUZZ_RELAY_PRIVATE_KEY="$(openssl rand -hex 32)" \
BUZZ_REQUIRE_AUTH_TOKEN=false \
BUZZ_RECONCILE_CHANNELS=true \
BUZZ_GIT_PROBE_WRITERS=8 \
SPROUT_REMINDER_SCHEDULER_INTERVAL_SECS=1 \
./target/ci/buzz-relay > /tmp/buzz-relay.log 2>&1 &
echo $! > /tmp/buzz-relay.pid
for attempt in $(seq 1 60); do
if ! kill -0 "$(cat /tmp/buzz-relay.pid)" 2>/dev/null; then
cat /tmp/buzz-relay.log
exit 1
fi
status_code=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:3000/_readiness || true)
if [ "${status_code}" = "200" ]; then
exit 0
fi
sleep 1
done
cat /tmp/buzz-relay.log
exit 1
- name: NIP-ER reminder e2e
# Feature e2e for NIP-ER (Event Reminders, kind:30300): write-path
# validation, author-only read filtering, and scheduler delivery against
# a live relay. The schema-drift / migration-version guarantee is owned
# by the buzz-db migration.rs unit tests, not this suite.
run: |
cargo nextest run \
--archive-file target/ci/backend-integration-tests.tar.zst \
-E 'binary(e2e_event_reminder) | binary(e2e_author_only_deletion) | binary(e2e_existence_oracle)' \
--run-ignored ignored-only
env:
RELAY_URL: ws://localhost:3000
# schema/schema.sql (applied above) indexes kind:5 content.
BUZZ_TEST_SEARCH_INDEXES_DELETIONS: "1"
- name: Upload relay log
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: backend-integration-relay-log
path: /tmp/buzz-relay.log
if-no-files-found: ignore
relay-e2e:
name: Relay E2E
runs-on: ubuntu-latest
timeout-minutes: 20
if: inputs.lane == 'required' && (github.event_name == 'push' || inputs.rust)
permissions:
contents: read
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: cashapp/activate-hermit@cea9af7913204a965fd488637a8d1811bba2e616 # v1
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32
with:
save-if: ${{ github.event_name != 'pull_request' }}
# Reuse the relay + git-credential-nostr built by Desktop E2E Relay
# instead of compiling them a second time.
- name: Download relay binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: desktop-e2e-relay-*
merge-multiple: true
path: target/ci
- name: Start relay
run: |
chmod +x ./target/ci/buzz-relay ./target/ci/git-credential-nostr
./scripts/start-relay-for-tests.sh --no-build
- name: Relay E2E tests
run: |
cargo test -p buzz-test-client --test e2e_persona --test e2e_team_catalog --test e2e_nostr_interop --test e2e_thread_roots --test e2e_project -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay invite -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip43_membership_snapshots_are_rejected -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip29_departure_wire -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip29_batch_put_user_wire -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip29_announce_channel_wire -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip29_unlisted_channel_wire -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip29_solo_share_and_convert_wire -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip29_member_raises_guest_agent -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip29_guest_is_read_only_in_normal_channel -- --ignored --nocapture
cargo test -p buzz-test-client --test e2e_relay nip29_relay_rejects_role_change_by_unprivileged_actor -- --ignored --nocapture
env:
RELAY_URL: ws://localhost:3000
GIT_CREDENTIAL_NOSTR_BIN: ${{ github.workspace }}/target/ci/git-credential-nostr
- name: Media read-auth e2e
# Reads require kind:24242 `t=get` auth, so these binaries are the only
# coverage that a real relay rejects bare reads and honours host- and
# hash-scoped tokens. They were #[ignore]d and selected by no CI job, so
# the lane never ran; select it here, where MinIO and the seeded
# 'localhost:3000' community already exist.
# --no-fail-fast: without it cargo stops after the first failing binary,
# so one broken case hides every later binary's result.
run: |
cargo test -p buzz-test-client --no-fail-fast --test e2e_media --test e2e_media_extended --test e2e_media_video -- --ignored --nocapture
env:
RELAY_URL: ws://localhost:3000
RELAY_HTTP_URL: http://localhost:3000
- name: Upload relay logs
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: relay-e2e-artifacts
path: /tmp/buzz-relay.log
if-no-files-found: ignore
results:
name: Results
if: ${{ always() }}
needs: [desktop-e2e-relay, postgres-tests, desktop-e2e-integration, backend-integration, relay-e2e]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
outputs:
desktop_e2e_relay_result: ${{ needs.desktop-e2e-relay.result }}
desktop_e2e_integration_result: ${{ needs.desktop-e2e-integration.result }}
backend_integration_result: ${{ needs.backend-integration.result }}
relay_e2e_result: ${{ needs.relay-e2e.result }}
postgres_tests_result: ${{ needs.postgres-tests.result }}
steps:
- run: echo "Captured Relay and PostgreSQL job results"