mirror of
https://github.com/block/buzz.git
synced 2026-10-27 12:40:30 +00:00
be269eb340738c81920c9e35700015b4e6c5f2aa
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Luke Tornquist
|
c0ab789499 |
feat(buzz-db): retire the channel-TTL trigger (#8210)
## Summary This drops the channel-TTL event trigger. The application already refreshes ephemeral-channel deadlines in the event transaction (`event_follow_up::refresh_channel_ttls`, from #8135), so the trigger only repeats that work. This PR originally retired both follow-up triggers. #8352 has since retired the push-match trigger in migration 0063, so after the rebase this PR covers only the TTL half. Migration `0065_retire_channel_ttl_trigger.sql`: - `SET LOCAL lock_timeout = '5s'` (the same bound as 0063) - drops `events_refresh_channel_ttl` and `refresh_channel_ttl_after_event_insert()` Dropping the trigger on the partitioned parent also drops its partition clones. Published migrations, including 0022 and 0024, stay checksum-frozen. The same objects are removed from `schema/` (`schema/functions/…`, `schema/triggers/events.sql`, `schema/schema.sql`) and from the pgschema reconcile script. After this, migrated and desired-state databases have no event follow-up triggers. Tests: - `follow_up_retirement_tests.rs` (renamed from `push_retirement_tests.rs`): the push-retirement upgrade test now stops at 0063. A new test upgrades 0064 → 0065 and checks that the TTL trigger is gone from the parent and every partition, that the function is gone, and that the published checksums haven't changed. - `event_follow_up_postgres_tests.rs`: the contract now has two arms. `AppOnly` is the shipped schema, where both triggers are absent. `Dual` reconstructs the historical overlap of both triggers. The `PushRetired` arm (the push trigger dropped, the TTL trigger kept) was the shipped state before this PR. Its tests are removed because `Dual` and `AppOnly` together cover it. - `embedded_migrator_contains_consolidated_initial_schema` pins the 0065 statements and checks that both objects are absent from the desired schema. **Deploy gate:** do not run 0065 against a database still served by a relay older than #8135. After 0065, raw SQL writers no longer refresh TTL deadlines. `docs/push-gateway-deployment.md` is updated to match. ### What got simpler The trigger and the app no longer both refresh TTL deadlines; the app is now the only producer. One plpgsql function and one trigger are gone from both schema paths, and 8 per-partition drops are gone from the reconcile script. The follow-up contract drops from three arms to two. ### Related issue BUZZ-313 (Linear), part of the trigger-retirement work. ### Testing Head is `4f7d0baf0`. Results below are at `678927730` unless noted. `678927730..4f7d0baf0` changes only one test in `crates/buzz-db/src/store/event.rs`, described in the first bullet. - **Fixed test, `permanent_channel_event_commits_do_not_lock_the_channel_row`** (`4f7d0baf0`). - **Why it changed:** after 0065, its raw-pool inserts no longer ran any TTL refresh, so the test passed whatever `refresh_channel_ttls` did. This was raised in review. - **What it does now:** it inserts through the production `AdmittedTx` path and keeps the barrier and the tuple-lock probe. It commits all N transactions while an external transaction holds `FOR SHARE` on the channel row. It also asserts that no `buzz_db_channel_ttl_refresh_failures_total` is counted. - **Mutation check:** with `SELECT … FOR UPDATE` injected into `refresh_channel_ttl`, the new test fails (`must not wait on the channel row`, timeout at 10s). The old test passes under the same mutation. - **Blox:** fmt and `buzz-db` clippy pass. - **GitHub CI at `4f7d0baf0`:** every required check passes (56 pass, 0 fail). CI builds the merge ref with `main` `93ffc37cb`. - **New and renamed tests, checked by name in the CI logs at `4f7d0baf0`.** These pass in PostgreSQL Domain / PostgreSQL Tests ([117033725006](https://github.com/block/buzz/actions/runs/38915998692/job/117033725006)): - `permanent_channel_event_commits_do_not_lock_the_channel_row` - `migration_schema_ttl_retirement_drops_every_partition_trigger_and_keeps_checksums` - `migration_schema_push_retirement_preserves_ttl_and_published_checksums` - `event_insert_ttl_refresh_handles_permanent_ephemeral_duplicate_and_activation_race` - `{desired,migration_schema}_event_follow_up_contract_{dual,app_only}` (four tests) `embedded_migrator_contains_consolidated_initial_schema` passes in Rust / Unit Tests ([117024125423](https://github.com/block/buzz/actions/runs/38915998692/job/117024125423)). No `*_push_retired` test runs anywhere. - **Blox gates at `678927730`** (private PG16 + Redis, on a workstation shared with other sessions): - `cargo fmt --check`, workspace `clippy -D warnings` and `buzz-db --lib` all pass. - The `buzz-admin`, `buzz-audit`, `buzz-deletion`, `buzz-push-gateway`, `buzz-search` and `buzz-workflow` Postgres packages pass. - `buzz-db`: 517 of 519 tests passed. The 2 that hung, both `writer_pool_*` tests, were killed. `buzz-relay`: 413 of 426 passed. The 13 failures are covered below. - **Uncontended reruns at `4f7d0baf0`** on a fresh, idle workstation (PG16.15, `competing=0` for every arm): - **`writer_pool_rejects_non_read_committed_database_default` predates this PR.** It timed out 3/3 at head and 3/3 at `main` `93ffc37cb`, with a 300s timeout per run. - **`writer_pool_metrics_stop_after_session_timeout_setup_failure` also predates this PR.** It timed out 2/3 at head and 2/3 at `main`, and passed in the remaining run on each side. - Both tests pass in CI. The gate accepts them as a named exception for this PR, as a pre-existing hang on the PG16 test environment. - **`buzz-db`:** 518 of 519 tests passed. The single failure is `writer_pool_rejects_non_read_committed_database_default`, killed after it hung (the exception above). - **`buzz-relay`: still open.** 413 of 426 tests passed. The same 13 `handlers::group_state_postgres_tests` failed as on the shared box, each panicking at once in its fixture's `ensure community` step with `Sqlx(PoolTimedOut)`. Contention does not explain them. The fixture's community setup is in `buzz-db`, which this PR changes. A head/`main` comparison of the full `buzz-relay` package on the same idle box is running to find out whether this predates the PR. All 13 pass in CI at this head. - **E2E (Gimli, Blox)** at `050e593af`. `050e593af..678927730` changes only tests and docs; the migration, schema and runtime code are byte-identical. Gimli's audited verdict: **EXERCISED AND WORKS**, for DB and relay runtime only. Gimli ran it at `050e593af`; this head's runtime is byte-identical but was not freshly executed. - **Upgrade:** a real relay startup took the schema from 0064 to 0065. That removed the TTL function and the trigger from the parent table and from all 8 partitions. Checksums for migrations 1–64 are unchanged. - **Lock-timeout rollback:** with a real lock held, the migration failed with `55P03` at 5.0007s and the schema rolled back fully to 64. Once the lock was released, a retry applied 65. - **Fresh desired schema:** after pgschema plus reconcile, the retired objects are absent. - **Real signed kind-9 sends:** on both schema paths, these refreshed ephemeral-channel deadlines to about +600s. Permanent channels stayed NULL. - **Raw SQL insert:** at 64 it refreshed the deadline. After 65 it left the deadline unchanged, on both the upgraded and the fresh schema. - **Not covered:** native or device testing, and deploy. - **Review:** Legolas found 0 P0 and 0 P1. His three optional P2s are fixed in `678927730`. - **No human app test:** this is a DB-only change. - **Migration numbering:** #8237 and #8372 also add a `0065_*` migration. Whichever PR merges second must renumber. --------- Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> |
||
|
Tom Brow
|
6766174673 |
chore(push): retire push notification database trigger (#8352)
Buzz must schedule mobile push notifications when it saves an eligible message. The relay, the server that receives and stores messages, now does this in application code. An older database trigger still exists. This trigger is a rule that runs automatically when a message is saved. This PR removes the old trigger and its supporting database function. The relay remains responsible for scheduling notifications. Migration 0063 makes this change in existing databases; updated setup scripts apply it to new databases. The change does not turn on push notifications. Production migration and deployment need separate approval. Before deployment, verify the release images and all pending database migrations. If push needs to be stopped after release, disable it with the feature flag and keep the qualified application version and database schema. Do not downgrade the application image. ### Validation - Local checks passed: 519 isolated database integration tests, 166 database unit tests, 17 source-policy tests, and the relay push-runtime tests. - Tests verify that the migration removes the old trigger, preserves earlier migration records, and lets the application schedule push work correctly. A concurrency test exercises the real application path; deliberately removing its lock makes the test fail. - [Current-head CI passed on retry](https://github.com/block/buzz/actions/runs/38752849383/attempts/2). The first attempt had one database-connection timeout during test setup; the failed jobs passed when rerun on the unchanged commit. - Initial local CI and push checks hit unrelated ACP test failures, which passed when run separately. Only the Rust-test push hook was skipped under the repository's local-check exception; other applicable hooks passed. ### Before merge - [x] Migration 0063 is available and the push trigger remains on [main at |
||
|
Tom Brow
|
868e2f8f32 |
refactor(push): one app profile per gateway (#8206)
Buzz push currently selects an application profile, such as `dogfood`, during client registration. We want each gateway to serve one Apple app configured by the operator instead. This removes profile selection from the client, relay, and gateway, while allowing internal and public gateways to use their own Apple identities. This PR implements that model in the Buzz source and upstream deployment chart. Discovery, enrollment, grants, and leases no longer carry an application-profile selector. Each gateway has one configured Apple application and APNs transport. The protocol changes are incompatible with the old registration format, so relays advertise `buzz-push-v1` instead of `nip-pl`. Old clients do not discover the new push capability. Updated clients use separate Keychain records and register again. The [push notes](https://github.com/block/buzz/blob/push-capability-v1/docs/buzz-push.md) describe the experimental contract. Landing this PR is the first step. Publish the matching gateway image and upstream chart, then land the [shared chart update](https://github.com/squareup/cash-helm-charts/pull/5740) and [gateway deployment update](https://github.com/squareup/builderbot-platform-core-infrastructure/pull/1085). Before cutover, stop old gateway instances and check legacy registration state. Migrations refuse to proceed if old gateway installations or active relay leases remain; they do not delete that state. Physical-device verification remains necessary before production activation. ### Validation Gateway and focused Flutter push tests pass, as do the Swift package, simulator build, and native notification/enrichment tests. A local relay advertises `buzz-push-v1` without `nip-pl` or `app_profiles`. The PostgreSQL suite passed 868 tests; its two migration-test failures were corrected and pass a focused rerun together with the gateway cutover guard. Chart rendering and release-contract checks pass. Desktop JavaScript tests, desktop and web builds, native desktop compilation, mobile analysis/tests, and desktop native tests pass. The full Rust unit lane and prescribed `just test` command pass, including workspace integration tests against task-isolated local services. The production overlay now requires an explicitly supplied APNs topic, with omission and rendered-value regression coverage. The corrected Swift checksum assertion passes the complete package suite: 78 XCTest tests and 42 Swift Testing tests. No physical-device enrollment or production cutover is claimed. Local merge validation: the embedded migration test and formatting pass after reserving profile migration 0060 behind rollback migration 0059. The local pre-push Rust lane failed ACP timing/steering tests (`idle_resets_on_stdout_activity`, `keepalive_resets_idle_past_deadline`, and the failed/missing-outcome steering tests). ACP and its dependency manifests are unchanged by this branch; all four pass in an isolated single-thread rerun. The documented local-check exception skips only the Rust pre-push lane for publication; required CI remains mandatory. Mobile and desktop pre-push lanes passed. Combined migration validation: after rollback 0059 landed, the main merge preserves profile migration 0060 and both embedded migration assertions. The focused migration test and formatting pass. The next local push again failed only the unchanged ACP idle/keepalive timing tests (1053 passed, 2 failed, 3 ignored); other lanes passed. Publication uses the same documented local Rust-lane exception; required CI remains mandatory. Follow-up review fix: removed the legacy profile column from the copied-lease fixture. All four event-follow-up contract tests pass on isolated PostgreSQL databases, covering desired and migrated schemas in dual and app-only modes. Formatting passes. Publication retains the prior passing local-lane evidence and uses the authorized pre-push exception for the known unrelated ACP timing failures; required CI remains mandatory. --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tom Brow
|
5b907b838a |
fix(push): isolate push enqueue and stop workers on rollback (#8216)
Push enqueueing previously ran inside the message transaction, so a push lock or database error could block or reject a message. Configured relay writers now submit bounded best-effort work only after commit. A dedicated connection and bounded queue isolate enqueue waits and failures from message-write connections. Overload, errors, process loss, or shutdown may drop a notification; the committed message remains authoritative. `BUZZ_PUSH_ENABLED=false` disables the application producer and workers. Updated writer connections always bypass any surviving enqueue trigger. The overlap migration updates an existing function without resurrecting retired objects. Apply it before relying on isolation with a surviving trigger, and replace every old enabled process and writer connection: missing settings preserve legacy behavior during overlap. SIGTERM/Ctrl-C cancels enqueue, matcher, and delivery work before listener drain. Durable queues, claims, leases, preferences, and delivery configuration remain intact. Claims expire normally; gateway requests already accepted may still deliver. Reactivation requires an explicit operator decision about retained work. There is no automatic purge, startup interlock, or backfill of messages written while disabled. Built on merged [#8135](https://github.com/block/buzz/pull/8135), including [#8178](https://github.com/block/buzz/pull/8178). Trigger retirement [#8210](https://github.com/block/buzz/pull/8210) remains open; this change is tested with the trigger present and after removal. Operational details are in [the rollback and reactivation guide](https://github.com/block/buzz/blob/relay-push-rollback/docs/push-gateway-deployment.md#relay-rollback-and-reactivation). ### Validation [ |
||
|
Tom Brow
|
781d39510c |
feat(push): support configurable HTTP(S) delivery URLs (#7877)
The relay can use a configurable HTTP(S) gateway delivery URL and signs the same URL it sends. The gateway verifies NIP-98 against the received request URL, including authority, port, path and query. App Attest, scoped grants, replay checks and delivery quotas are unchanged. Forwarding proxies must preserve Host and supply the original request scheme through X-Forwarded-Proto. The NIP relay-delivery wording now defines the configured HTTP(S) URL as the signed URL and labels the public URL as an example. <!-- buzz-review-completed --> --------- Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tom Brow
|
0ef7a2222e |
fix(mobile-infra): render push grant lifetimes as decimal in chart 0.3.2 (#7820)
A valid 30-day push gateway grant lifetime could render in scientific notation (e.g. `2.592e+06`) in the chart, which the gateway binary does not accept. Always ormat the integer as decimal text, preserving the strict gateway parser. ### Testing Added regression coverage that feeds actual Helm-rendered environment values into the production config parser. Covers defaults, values files, command-line overrides, exact lifetime boundaries, and invalid input rejection. The default case failed before the fix and passes afterward. Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tom Brow
|
7e96f91229 |
feat(mobile-infra): support development App Attest (#7744)
Development-signed iOS apps cannot enroll with the current gateway because their Apple App Attest AAGUID is rejected. Add an explicit personal development build feature and environment setting so a development gateway can use a developer's own application identity and sandbox APNs credentials. Ordinary gateway builds remain production-only. Special builds also default to production, enforce exactly the selected signed AAGUID, and retain certificate-chain, pinned-root, nonce, app-ID, key, credential-ID, and counter verification. Document the matching mobile signing configuration and expose the feature through an optional Docker build argument. ### Validation The development image deployed successfully to a personal Railway stack; private readiness, enrollment challenges, authenticated relay access, and APNs sandbox TLS/HTTP2 connectivity passed. Signed-off-by: Tom Brow <tomb@block.xyz> |
||
|
Tom Brow
|
76f7e5ef0c |
feat(push): make gateway origin configurable (#7261)
Make the push gateway origin configurable across the relay, gateway, mobile client, and deployment chart. This is a fresh deployment. The gateway has only run on personal development infrastructure, and push has no existing enabled users. Legacy migration, configured-to-unconfigured mobile transitions, and rolling upgrades from pre-launch gateway binaries are not supported. Populated pre-launch databases are rejected with an informative error before migrations run; their data is untouched. Mobile builds may omit the gateway URL, leaving push unavailable. Once push is enabled, subsequent builds must retain that configuration. ### Validation - All applicable CI checks pass at `5f546cfaf403f0386f146d213adfb4392f4ec29c`, including Rust, mobile/Swift, desktop/integration, security, image builds, and Helm checks. An unrelated agent cancellation test passed on retry. - PostgreSQL CI passed 384 tests, including the production-entry-point regression for refusing populated pre-launch databases before migrations, fresh initialization, and subsequent initialized deployments. - Focused local validation passed all 10 gateway PostgreSQL tests under the CI discovery filter and all 3 unconfigured-mobile tests. Repository-wide PostgreSQL discovery validation and the complete pre-push hooks passed. - Physical-device APNs/App Attest behavior and deployment to a live cluster were not verified in this work. --------- Signed-off-by: Tom Brow <tomb@block.xyz> Co-authored-by: Codex <noreply@openai.com> |
||
|
Tom Brow
|
42b42447b0 |
feat(mobile): prepare buzz-push-gateway for deployment (#7158)
A few things to prepare for deploying the Buzz push gateway as implemented in #6269: - Cut `buzz-push-gateway` chart `0.2.0` from the current config (certificate auth, dogfood client) - Datadog-compatible metrics - Align the desired schema, metrics docs, and production ingress guidance with the current dogfood deployment contract ## Links - [BUZZ-17: Publish the MVP-compatible gateway chart](https://linear.app/squareup/issue/BUZZ-17/publish-the-mvp-compatible-gateway-chart) - [BUZZ-26: Add Datadog-compatible push-gateway metrics and correct source drift](https://linear.app/squareup/issue/BUZZ-26/add-datadog-compatible-push-gateway-metrics-and-correct-source-drift) --------- Signed-off-by: Tom Brow <tomb@block.xyz> Co-authored-by: Codex <noreply@openai.com> |
||
|
Luke Tornquist
|
3ed623bb21 |
feat(db): configurable writer session timeouts (lock, idle-txn, statement) (#6229)
## Why A wedged relay boot pod holding a relation lock can park every other writer in the fleet behind it: DB load pins at pool capacity in `Lock:relation` waits while CPU stays flat, and nothing server-side releases the lock until the holder dies. We hit exactly this in production — ~1,400 sessions queued behind one crash-looping pod's boot transaction for ~20 minutes until kubelet killed the container. ## What Applies session-level Postgres timeouts to every **writer** connection inside the existing single `after_connect` hook in `buzz-db`, all env-tunable through the same `Config::from_env → DbConfig` path as the existing pool-size knobs: | Env var | GUC | Default | Effect | |---|---|---|---| | `BUZZ_DB_LOCK_TIMEOUT_MS` | `lock_timeout` | 5000 | statements waiting on any lock fail fast instead of parking behind a wedged holder | | `BUZZ_DB_IDLE_TXN_TIMEOUT_MS` | `idle_in_transaction_session_timeout` | 60000 | reaps wedged clients idling inside an open transaction while holding locks | | `BUZZ_DB_STATEMENT_TIMEOUT_MS` | `statement_timeout` | 0 (off) | opt-in runaway-statement cap; off by default because startup migrations/backfills legitimately run long statements | `0` disables a timeout (Postgres semantics) and deliberately passes through the env parsing — unlike the pool-size knobs where `0` falls back to the default. The reader pool is untouched: replica sessions never take contended locks and already fail acquire in 150 ms. Deployers tune these via plain env vars (`.env`, or `relay.extraEnv` in the Helm chart) — no code changes needed. ## Behavior change to note With the 5 s default `lock_timeout`, a boot-time migration or backfill that waits >5 s on a lock now errors (surfacing in logs / crash-looping the pod) instead of stalling silently. That is the intended visible-failure-over-fleet-stall tradeoff; deployers with slow contended migrations can set `BUZZ_DB_LOCK_TIMEOUT_MS=0`. ## Testing - `cargo test -p buzz-db -p buzz-relay` — buzz-db green; buzz-relay has 9 failures that also fail on clean `main` in this environment (api::admin/api::media/mesh_demo — unrelated, pre-existing). - New config test covers override / `0`-passthrough / invalid-fallback for all three env vars. - Extended the existing `writer_pool_safety_hook_is_single_and_composed` source-shape test so the timeouts can't drift out of the single `after_connect` hook (SQLx replaces hooks — a second hook would silently disarm the floor guard). - `cargo fmt --check` and `cargo clippy --all-targets` clean for the touched crates. Closest existing PR/issue: none found. --- **Update Aug 28, 17:06 EDT:** Rebased onto `main` at `a3730784fc` and addressed the latest correctness review. - Ported the timeout policy onto the refactored `buzz-db::runtime` pool constructor and kept the shared env overlay for relay, admin, deletion, and audit writers. - Migration/schema-destruction connections now disable `lock_timeout` and `statement_timeout` for their intentional long wait/DDL path. This supersedes the earlier “Behavior change to note”: contended boot migrations wait for the current migration owner rather than crash-looping after five seconds. - The audit worker now preserves and retries the same entry on PostgreSQL `55P03` lock timeouts, using exponential backoff capped at one second. Other database errors retain the existing terminal error behavior, and retries emit `buzz_audit_log_lock_retries_total`. - Added CI-backed PostgreSQL regressions for writer GUC installation/migration exemption, audit-pool lock timeouts, and worker recovery. The worker regression holds the real audit advisory lock past `lock_timeout`, observes a retry, releases the lock, and proves the original entry is appended exactly once. Current verification supersedes the earlier testing notes: workspace Rust clippy passed with warnings denied; all nine infrastructure-free backend unit-test lanes passed; all three focused PostgreSQL regressions passed against PostgreSQL 17; formatting, diff checks, file-size guards, and desktop frontend checks passed. The Linux Blox workstation could not run the unrelated Tauri native lane because `glib-2.0` is absent, so that platform check is left to PR CI. --- **Update Aug 31, 11:09 EDT:** Rebased onto current `main` at `c3132c3ee9` and reran the requested audit-lock contention scenario on Blox at head `896c3fe9ed`. - `git range-diff` reports both PR commits unchanged by the rebase; the branch remains two commits and the worktree is clean. - `cargo fmt --all -- --check` and clippy with warnings denied passed for `buzz-db`, `buzz-relay`, `buzz-admin`, and `buzz-deletion`. - All three focused PostgreSQL 17 regressions passed: writer session timeout/migration exemption, audit writer timeout bounds, and audit worker recovery of the original entry exactly once. - Live protocol verification used a head-built relay and CLI, native PostgreSQL 17/Redis, `BUZZ_DB_LOCK_TIMEOUT_MS=300`, and an eight-second hold on the community audit advisory lock. The real message was accepted and persisted once while the lock was held; its audit-row count remained zero during contention while retries accumulated. After release, exactly one `event_created` audit row appeared and remained exactly one after an additional two-second duplicate check. The run recorded nine lock-timeout retries, zero audit failures, and event ID `7f8c4ffae28e78555fcf2d56396d6e6c01b3712e5411288dc79e9a54af9d9444`. Generated with Codex --------- Signed-off-by: Luke Tornquist <tornquist@squareup.com> |
||
|
Tom Brow
|
c432a111ca |
feat(mobile): push notifications MVP (#6269)
This PR implements MVP, iOS-only,
[NIP-PL](
|
||
|
Tyler
|
bffbc5f22c |
feat(push): deliver accepted relay events as wakes (#1866)
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> |
||
|
Tyler
|
1c006822e4 |
feat(push): add public APNs gateway (#1770)
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> |