5 Commits

Author	SHA1	Message	Date
Luke Tornquist	d34c9c2c9b	ci: shorten the PR critical path (desktop lanes, smoke shards, parallel relay artifacts) (#8156) ... ## Summary The PR critical path on main push [run 37635224099](https://github.com/block/buzz/actions/runs/37635224099) (`e097c4299`) was about 42 minutes: - **Desktop Core** took 41 min. It ran frontend checks, two clippy graphs, the Tauri suite, the mesh-llm suite and every compiled-flag state one after another. The compiled-flag step alone took 16m51. - **Relay artifact producer → Desktop E2E Integration** took about 31 min. The producer built the relay binaries (5m07), the backend integration archive (4m41) and the PostgreSQL archive (8m18) one after another. - **Desktop Smoke E2E** took 19–24 min per shard across 4 shards, with 82 min of tests in total. This PR attacks all three. ### 1. Run each compiled-flag state once (`Justfile`) | Previous step | Cost | Why it's redundant | |---|---|---| | Clean state: full `buzz_lib` suite (`--lib`) | 3m27 run | `desktop-tauri-test` runs the same suite in the same compile state one step earlier. When `BUZZ_TEST_EXPECTED_AGENT_ACCESS_OWNER_ONLY` is unset, the expected value already defaults to `false` in non-owner-only builds (`runtime/test_fixtures.rs`, `agents_tests.rs`). | | Internal state: auto-connect and owner-only built separately | 1m22 + 1m06 + 56s across three recompiles | Internal release packaging always sets both capabilities together, so building owner-only without auto-connect tests a configuration that never ships. The extra 56s happens because a bare `cargo test <filter>` builds every target, not just the `--lib` harness. | | Demo state: `cargo test --workspace` | ~30s more than `--lib` | `buzz-terminal` and the `tests/*.rs` integration tests don't read any `BUZZ_BUILD_*` variable or `buzz_lib`, so they ran the same binaries a second time. | A fourth cost was hidden: every recipe invocation spent ~1m20 recompiling `buzz-desktop` to the *same* unit hash `desktop-tauri-test` had just built. `_ensure-sidecar-stubs` ran `touch` on every sidecar stub on each run. `tauri-build` emits `rerun-if-changed` for each `externalBin`, so the new mtimes reran `build.rs` and recompiled the crate. Stubs are now created only when they're missing. This also speeds up any local `desktop-tauri-*` recipe run back-to-back. After this change: - **Clean state:** runs only the two compiled-flag assertions (`compiled_flag_matches_expected`, `compiled_policy_matches_expected`) with `expected=false`. No recompile. - **Internal state:** one compile with `BUZZ_BUILD_AUTO_CONNECT_DEFAULT_RELAY=1` and `BUZZ_BUILD_AGENT_ACCESS_OWNER_ONLY=1`. It runs both assertions with `expected=true`, then the full `buzz_lib` suite. The suite now runs with auto-connect enabled as well, which matches the shipped internal build more closely than before. - **Auto-connect-only state (assertions only):** auto-connect `true`, owner-only `false`. The two capabilities are documented as independent. Without this state, a `build.rs` block that read the other capability's variable would pass, because every other state sets both or neither. - **Demo state:** one compile with only `BUZZ_BUILD_DEMO_SLUG`. It asserts the slug and asserts both capabilities stay `false`, then runs the full `buzz_lib` suite. - The recipe clears the three toggled `BUZZ_BUILD_*` capability variables and any caller `BUZZ_TEST_EXPECTED_*` values up front. Each state exports its own values in a subshell. Every invocation uses `--lib`, so each state compiles one test harness. - Each ignored-assertion run requires exactly one passing test per name filter. Previously, a renamed or un-ignored assertion matched nothing and passed silently. - The 32-character demo-name rejection check is unchanged. The full-suite reruns in the internal and demo states are kept on purpose. They are the only coverage that the whole suite holds up in those two shipped build shapes. Estimate: the compiled-flag work goes from about 17 min to about 11. ### 2. Split Desktop Core into parallel, path-gated lanes (`_ci-desktop.yml`) - **Desktop JS:** `desktop-check`, `desktop-test` and `desktop-build`. These compile no Rust, but some node:test cases read relay and DB crate sources to pin shared constants (for example `relayReconnectReplay.test.mjs`). So the job runs on desktop *and* Rust changes, in parallel with the Tauri lanes. `lefthook.yml` notes this as a deliberate local deviation. - **Desktop Tauri (core | mesh | compiled-flags):** a matrix of parallel Rust lanes. - `core` and `compiled-flags` share the existing `desktop-core` Rust cache key, so they start warm. `compiled-flags` restores it read-only. - `mesh` keeps its own cache because `mesh-llm` pulls a different dependency graph. - **Smoke E2E and the Windows frontend build** compile no Rust and read no crate sources, so they no longer run for Rust-only changes. - **Smoke E2E goes from 4 to 10 shards.** Playwright shards by test count, not duration. A simulation from the per-test durations in run 37635224099 reproduces the observed 4-shard split exactly. With 10 shards, the slowest shard runs about 11 min of tests. - **The `Desktop` gate is stricter.** Each selected job must succeed and each unselected job must be skipped, so a mis-gated job can't pass silently. The required check context (`Desktop`) is unchanged. `scripts/ci-required-gates.test.mjs` pins each job's `if:` to the gate's selection expression, and checks which jobs a Rust-only, desktop-only, Tauri, push, and empty selection pick. - **Path-filter fixes:** - The `desktop` filter now includes `_ci-desktop.yml`, `ci.yml` and `Justfile` (which holds the desktop recipes), so edits to them run the frontend lanes. - The `rust` filter now matches `Justfile`. It still named the pre-rename `justfile`, so a `Justfile`-only change selected no Rust suites. - `scripts/ci-selection.test.mjs` gains `Justfile` and desktop-workflow scenarios, and the shared-workflow scenario now expects `desktop`. - **Dead step removed:** the failure-only Playwright artifact upload in the old Core job never had anything to upload, because that job never ran Playwright. ### 3. Build relay artifacts in parallel (`_ci-relay.yml`) - **The producer is now a 3-way matrix** (`relay-binaries`, `backend-integration-tests`, `postgres-tests`). - Each lane has its own exact-source cache entry and uploads `desktop-e2e-relay-<artifact>`. - Consumers download `desktop-e2e-relay-*` into `target/ci`, so their paths are unchanged. - The required `Desktop E2E Relay` context still reads the producer job's aggregate result. - **Rust cache saving:** all lanes restore the existing Rust cache key, but only the widest lane (`postgres-tests`) saves it. Otherwise the fastest lane could overwrite it with a narrower target dir. - **Desktop E2E Integration goes from 2 to 3 shards.** Shard labels now use `strategy.job-total`. ### 4. Bound degraded apt steps (`scripts/ci-apt-retry.sh`) - With more shards, more runs hit the occasional degraded Ubuntu mirror. On this PR, a Tauri lane ran `apt-get install` for 27 minutes: apt's `Acquire` timeouts kept firing, but apt kept re-queuing the requests and never failed. Separately, three Smoke shards stalled silently in `playwright install-deps` (no output after `InRelease`) until the job timeout. Healthy apt steps across four recent runs finished in at most 238s. - Every apt-backed step in Desktop and Integration now runs through `scripts/ci-apt-retry.sh`. The script writes the apt options to `apt.conf.d`, so Playwright's own apt-get calls get them too. It kills each attempt after 7 minutes (90s for `apt-get update`). Between attempts it stops any `apt-get` or `dpkg` the attempt left running, repairs dpkg under its own 60s deadline, and retries up to 3 times. The stop matters for `playwright install-deps`, which runs `sudo sh -c "apt-get update && apt-get install ..."`: sudo puts its command in a new process group, so the deadline's signal stops `sh` and apt-get survives holding the dpkg lock (seen on Smoke shard 8 of run 37696815557, and reproduced with real sudo). - Downloaded `.deb`s are kept between attempts, so a retry resumes where the last one stopped. Against a mirror that stays degraded, this bounds each wrapped command at about 25 minutes rather than recovering it. The Tauri lanes wrap two commands, `apt-get update` (about 8.5 minutes worst case) and `apt-get install` (about 25), so about 34 minutes together. Job timeouts are sized to fit that bound plus the job's own work: Tauri 55m (lanes take up to ~15m), Integration 40m, and Smoke 45m. A degraded mirror therefore ends with the wrapper's error instead of a silent job timeout. - **The retry has already fired on this PR.** In [run 37703795342](https://github.com/block/buzz/actions/runs/37703795342), `playwright install-deps` on Integration shard 1/3 hit the 420s attempt deadline. The second attempt started about 1.4s later, downloaded and configured the real packages, and succeeded. The step took 526s (8m46), and that shard took 17m24 while the other two took about 5m. Healthy installs have taken at most 238s, so 420s leaves headroom, and a slow mirror costs one attempt instead of the job. - Between attempts the script stops **every** `apt-get`/`dpkg` on the machine, not only the ones it started. That's intentional: on a hosted runner any other one would hold the same dpkg lock. It assumes an ephemeral runner, and the script says not to use it on a self-hosted runner or a developer machine. - `scripts/ci-apt-retry.test.mjs` covers the following with a fake `sudo`: success, retry, the attempt budget, a stalled attempt, a stalled dpkg repair, an apt-get that escapes the attempt's process group (on a retry and on the final attempt), and usage. It needs Linux (GNU `timeout`, `flock`, `setsid`, procps), so it skips on other platforms. CI runs it on Ubuntu. ### What got simpler - **Compiled-flag recipe:** two full-suite runs instead of three, six cargo invocations instead of eight, and sidecar stubs no longer invalidate the desktop build cache. - **Path selection:** each desktop job declares only the paths that actually affect it. The stale `justfile` filter and a dead artifact step are gone. - **Apt options:** set once in `apt.conf.d` by one script, instead of as repeated `-o Acquire...` flags on each apt call. - **Shard counts:** the shard count is written once per matrix, not repeated in names and commands. ### Cost - **More runners:** a full run uses more runner jobs. Smoke E2E goes from 4 to 10 jobs, Desktop Core from 1 to 4, the relay producer from 1 to 3, and Integration from 2 to 3. - **Fewer runners on single-area PRs:** desktop-only PRs skip the Rust lanes, and Rust-only PRs skip Smoke E2E and the Windows build. - **One extra Rust cache entry:** `desktop-mesh`. ### Related issue None found. ### Testing Blox, compiled-flag recipe at `77b9874b`: - [x] Recipe passes with output redirected to a file. The log keeps all 4 state headers, the ignored assertions ran 2/2/2/3, and both full suites passed (`3423 passed`). - [x] A second back-to-back run shows no clean-state `Compiling buzz-desktop`. - [x] Mutation tests all fail as required: - `build.rs` auto-connect reads the owner-only var: rc=101. - The reverse swap: rc=101. - Rename `compiled_policy_matches_expected`: rc=1. - Wrong owner-only expectation: rc=101. - [ ] `just ci` Workflows: - [x] `actionlint` on `_ci-desktop.yml`, `_ci-relay.yml` and `ci.yml`: clean. - [x] `scripts/test-ci-required-context-isolation.sh`: passes. - [x] GitHub Actions: runs [37701003713](https://github.com/block/buzz/actions/runs/37701003713) and [37703795342](https://github.com/block/buzz/actions/runs/37703795342) took 27m09 and 28m51. Before this PR, PR runs had a median of 40.0m (n=21) and main pushes took about 42m. These are observations across mixed runner images, not a measured causal speedup. - [x] Selection, gate and apt suites (74/74 on Linux), `actionlint` and `bash -n` on Blox at tree `db9725fe`. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>	2026-10-08 11:33:37 -04:00
Luke Tornquist	d742fc5f30	Use Silo for bundled object storage (#8023)	2026-10-07 18:02:27 -04:00
Logan Johnson	ad9591c43b	fix(desktop): order unnamed roster members by full canonical npub (#7503) ... 🤖 ## Summary - Channel members appear in the Members sidebar. A member who has never set a display name is listed under an abbreviated form of their public key (npub), and the sidebar previously sorted those unnamed members by that short label. Short labels are not unique — different keys can share one — so the order of unnamed members could look arbitrary or unstable. Unnamed members now sort by their full public key, so the order is deterministic. - When two members display the same name, the previous tiebreak was membership order (who joined first), which is not visible to a reader and can shift as roster data loads in. The tiebreak is now the full public key, so identical display names always land in the same order. - Nothing gets noisier on screen: the full key is used only for sorting, and the sidebar still shows the compact abbreviated form. Priorities are unchanged — authored (custom) names still outrank fallback labels, and role/current-user grouping still applies. - Scope is the desktop app's Members sidebar and member management: the two existing sort comparators. Mobile and other lists in the app are untouched. ### Related issue Based on #7495 (introduced the abbreviated npub labels this follows up on). The original five presentation PRs remain independently reviewable. No closer duplicate found. ### Testing - 6469 desktop unit tests, typecheck, and check pass. - The 3 existing consumer-seam E2E tests still pass; a new E2E test asserts the sidebar lists unnamed members in full-key order, with fixture members deliberately inserted in the opposite order so incoming membership order cannot mask the sort. - Negative check: reverting only this change makes the new ordering assertion fail, so it genuinely binds the new sort. - CI has not run on this PR yet. Buzz provenance: channel 1f0e4a3d-7e01-4efe-bb16-843b357f85c9 / task 340c3de9b27dbedb8453c0c7652220f9080d30fcc70a7c4f6e27fdd4fa378056 --------- Signed-off-by: Logan Johnson <loganj@squareup.com> Co-authored-by: Larry <627498bd4bd1f281a16431e3c6cce3b5c25b6692798c78672298aefbf2f8f8b5@buzz.block.builderlab.xyz>	2026-09-09 13:32:10 -07:00
Mic Neale	44316ff72f	feat(mesh): upgrade to mesh-llm 0.76.0-rc8 and recommend Qwen3.8 27B (#6189) ... Upgrades Buzz's mesh-llm dependencies through the released `v0.76.0-rc8` tag (`2040765d`), including the Qwen3.8 curated recommendation and rc8's scheduler/runtime improvements. **Scope note:** the earlier open-relay/unenforced-admission mode has been removed from this PR at Mic's direction — it is not a product mode we want. Mesh admission remains roster/allowlist driven, exactly as on `main`: on a relay with no NIP-43 membership snapshot the mesh runs self-only. No NIP-11 mode probing, no mode-transition restarts. A future perimeter/admission strategy for open relays will be designed separately. This PR also: - seeds `BUZZ_AGENT_LLM_TIMEOUT_SECS=660` for mesh agents, above MeshLLM's 600-second backend timeout; - makes `desktop-tauri-clippy` lint both default and `mesh-llm` cfg graphs; - runs the feature-enabled desktop test suite in CI; - recommends Qwen3.8 27B Q4_K_M for 64 GB-and-larger machines, then ladders down through Gemma 4 E4B and Qwen 9B for smaller machines; - keeps stored shared-compute `auto` translated to MeshLLM's supported wire model `mesh`. RC8 verification: - `just ci` passed locally at `92ecc7ec933bdd4df804cc9f28a2b51efa5313c5`. - Pre-push differential gates passed, including both desktop Tauri cfg graphs and package tests. - A prior isolated runtime smoke used the RC8 binary's OpenAI endpoint for a Buzz-shaped system/user/tool/tool-result/final-response loop; all assertions passed and the isolated process was shut down. Perf previously measured on M5 Metal, Qwen3.8-27B-Q4_K_M: TTFT 0.22–0.32 s, ~25 tok/s streaming; agent-shaped turns ~1 s to first token after the first (prefix cache). --------- Signed-off-by: Michael Neale <michael.neale@gmail.com> Signed-off-by: Jimmy <1fe240cd1a8cf775f6f3060f115e5a303181f3abf28ad4cb0c2515f4a02b36a8@meshllm.communities.buzz.xyz> Signed-off-by: Alessandro Joabar <sandro@squareup.com> Co-authored-by: Michael Neale <michael.neale@gmail.com> Co-authored-by: Jimmy <1fe240cd1a8cf775f6f3060f115e5a303181f3abf28ad4cb0c2515f4a02b36a8@meshllm.communities.buzz.xyz> Co-authored-by: Mic Neale <305999590+micspiral@users.noreply.github.com> Co-authored-by: Alessandro Joabar <sandro@squareup.com>	2026-09-08 07:30:45 -07:00
Luke Tornquist	0e878664b0	ci: split CI into reusable workflows (#7168) ... ## Why The monolithic CI workflow is a frequent merge-conflict hotspot. Splitting cohesive domains into same-repository reusable workflows keeps one centrally filtered entry point while letting Rust, desktop, relay/PostgreSQL, client, and security CI evolve independently. ## What - Keep `ci.yml` as the only push/pull-request orchestrator with unchanged concurrency and path detection. - Move 18 execution jobs into five `workflow_call`-only domain workflows without changing their runners, steps, matrices, caches, artifacts, permissions, or timeouts. - Keep the relay artifact producer with desktop integration, the complete PostgreSQL lane, and relay E2E consumers. - Preserve all 12 existing required GitHub Actions contexts through lightweight top-level compatibility gates, so the repository ruleset does not need to change. - Update the Rust-cache contract to follow Unit Tests into `_ci-rust.yml`. ## Risk Assessment CI-only change with moderate workflow-orchestration risk. The main risks are reusable-workflow output propagation, skip behavior, and visible check naming; the old required names remain explicit top-level jobs, and the draft will stay open until an exact-head GitHub Actions run and independent review are complete. Generated with Codex --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> Signed-off-by: tornquist <tornquist@squareup.com>	2026-09-02 12:23:16 -04:00