Files
buzz/.env.example
Luke Tornquist 3ed623bb21 feat(db): configurable writer session timeouts (lock, idle-txn, statement) (#6229)
## Why

A wedged relay boot pod holding a relation lock can park every other
writer in the fleet behind it: DB load pins at pool capacity in
`Lock:relation` waits while CPU stays flat, and nothing server-side
releases the lock until the holder dies. We hit exactly this in
production — ~1,400 sessions queued behind one crash-looping pod's boot
transaction for ~20 minutes until kubelet killed the container.

## What

Applies session-level Postgres timeouts to every **writer** connection
inside the existing single `after_connect` hook in `buzz-db`, all
env-tunable through the same `Config::from_env → DbConfig` path as the
existing pool-size knobs:

| Env var | GUC | Default | Effect |
|---|---|---|---|
| `BUZZ_DB_LOCK_TIMEOUT_MS` | `lock_timeout` | 5000 | statements waiting
on any lock fail fast instead of parking behind a wedged holder |
| `BUZZ_DB_IDLE_TXN_TIMEOUT_MS` | `idle_in_transaction_session_timeout`
| 60000 | reaps wedged clients idling inside an open transaction while
holding locks |
| `BUZZ_DB_STATEMENT_TIMEOUT_MS` | `statement_timeout` | 0 (off) |
opt-in runaway-statement cap; off by default because startup
migrations/backfills legitimately run long statements |

`0` disables a timeout (Postgres semantics) and deliberately passes
through the env parsing — unlike the pool-size knobs where `0` falls
back to the default. The reader pool is untouched: replica sessions
never take contended locks and already fail acquire in 150 ms.

Deployers tune these via plain env vars (`.env`, or `relay.extraEnv` in
the Helm chart) — no code changes needed.

## Behavior change to note

With the 5 s default `lock_timeout`, a boot-time migration or backfill
that waits >5 s on a lock now errors (surfacing in logs / crash-looping
the pod) instead of stalling silently. That is the intended
visible-failure-over-fleet-stall tradeoff; deployers with slow contended
migrations can set `BUZZ_DB_LOCK_TIMEOUT_MS=0`.

## Testing

- `cargo test -p buzz-db -p buzz-relay` — buzz-db green; buzz-relay has
9 failures that also fail on clean `main` in this environment
(api::admin/api::media/mesh_demo — unrelated, pre-existing).
- New config test covers override / `0`-passthrough / invalid-fallback
for all three env vars.
- Extended the existing `writer_pool_safety_hook_is_single_and_composed`
source-shape test so the timeouts can't drift out of the single
`after_connect` hook (SQLx replaces hooks — a second hook would silently
disarm the floor guard).
- `cargo fmt --check` and `cargo clippy --all-targets` clean for the
touched crates.

Closest existing PR/issue: none found.

---
**Update Aug 28, 17:06 EDT:** Rebased onto `main` at `a3730784fc` and
addressed the latest correctness review.

- Ported the timeout policy onto the refactored `buzz-db::runtime` pool
constructor and kept the shared env overlay for relay, admin, deletion,
and audit writers.
- Migration/schema-destruction connections now disable `lock_timeout`
and `statement_timeout` for their intentional long wait/DDL path. This
supersedes the earlier “Behavior change to note”: contended boot
migrations wait for the current migration owner rather than
crash-looping after five seconds.
- The audit worker now preserves and retries the same entry on
PostgreSQL `55P03` lock timeouts, using exponential backoff capped at
one second. Other database errors retain the existing terminal error
behavior, and retries emit `buzz_audit_log_lock_retries_total`.
- Added CI-backed PostgreSQL regressions for writer GUC
installation/migration exemption, audit-pool lock timeouts, and worker
recovery. The worker regression holds the real audit advisory lock past
`lock_timeout`, observes a retry, releases the lock, and proves the
original entry is appended exactly once.

Current verification supersedes the earlier testing notes: workspace
Rust clippy passed with warnings denied; all nine infrastructure-free
backend unit-test lanes passed; all three focused PostgreSQL regressions
passed against PostgreSQL 17; formatting, diff checks, file-size guards,
and desktop frontend checks passed. The Linux Blox workstation could not
run the unrelated Tauri native lane because `glib-2.0` is absent, so
that platform check is left to PR CI.


---
**Update Aug 31, 11:09 EDT:** Rebased onto current `main` at
`c3132c3ee9` and reran the requested audit-lock contention scenario on
Blox at head `896c3fe9ed`.

- `git range-diff` reports both PR commits unchanged by the rebase; the
branch remains two commits and the worktree is clean.
- `cargo fmt --all -- --check` and clippy with warnings denied passed
for `buzz-db`, `buzz-relay`, `buzz-admin`, and `buzz-deletion`.
- All three focused PostgreSQL 17 regressions passed: writer session
timeout/migration exemption, audit writer timeout bounds, and audit
worker recovery of the original entry exactly once.
- Live protocol verification used a head-built relay and CLI, native
PostgreSQL 17/Redis, `BUZZ_DB_LOCK_TIMEOUT_MS=300`, and an eight-second
hold on the community audit advisory lock. The real message was accepted
and persisted once while the lock was held; its audit-row count remained
zero during contention while retries accumulated. After release, exactly
one `event_created` audit row appeared and remained exactly one after an
additional two-second duplicate check. The run recorded nine
lock-timeout retries, zero audit failures, and event ID
`7f8c4ffae28e78555fcf2d56396d6e6c01b3712e5411288dc79e9a54af9d9444`.

Generated with Codex

---------

Signed-off-by: Luke Tornquist <tornquist@squareup.com>
2026-08-31 11:59:13 -04:00

327 lines
17 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# =============================================================================
# Buzz Backend — Local Development Environment
# =============================================================================
# Copy this file to .env and adjust as needed:
# cp .env.example .env
#
# All defaults here work with `docker compose up` out of the box.
#
# Service ports (defaults):
# Postgres → localhost:5432
# Redis → localhost:6379
# Typesense → localhost:8108
# Adminer → localhost:8082 (DB browser UI)
#
# Note: If port 8082 conflicts, change the adminer port in docker-compose.yml
# =============================================================================
# -----------------------------------------------------------------------------
# Database (Postgres 17)
# -----------------------------------------------------------------------------
DATABASE_URL=postgres://buzz:buzz_dev@localhost:5432/buzz
# Optional read-replica URL; unset/blank keeps all reads on the writer.
# READ_DATABASE_URL=postgres://buzz:buzz_dev@localhost:5433/buzz
PGHOST=localhost
PGPORT=5432
PGUSER=buzz
PGPASSWORD=buzz_dev
PGDATABASE=buzz
# -----------------------------------------------------------------------------
# Redis 7
# -----------------------------------------------------------------------------
REDIS_URL=redis://localhost:6379
# Max connections in the relay's shared Redis pool (default 16).
# BUZZ_REDIS_POOL_SIZE=16
# Max connections in each of the relay's Postgres pools — writer and, when
# READ_DATABASE_URL is set, reader (default 50).
# BUZZ_DB_POOL_SIZE=50
# Writer-session Postgres timeouts for buzz-db-backed pools and the relay audit
# pool, all in milliseconds; 0 disables. The separately deployed push gateway
# owns its own database and session policy and does not consume these knobs.
# lock_timeout: fail a statement that waits this long on any lock instead of
# parking behind a wedged holder (default 5000).
# BUZZ_DB_LOCK_TIMEOUT_MS=5000
# idle_in_transaction_session_timeout: reap sessions idle inside an open
# transaction — bounds how long a wedged client can hold locks (default 60000).
# BUZZ_DB_IDLE_TXN_TIMEOUT_MS=60000
# statement_timeout: cap any single statement's runtime. Off by default —
# startup migrations/backfills legitimately run long statements. Warning: a
# pathologically low value (e.g. 1) also times out connection setup and can
# prevent any DB connection from establishing.
# BUZZ_DB_STATEMENT_TIMEOUT_MS=0
# -----------------------------------------------------------------------------
# Typesense (search)
# -----------------------------------------------------------------------------
TYPESENSE_API_KEY=buzz_dev_key
TYPESENSE_URL=http://localhost:8108
# -----------------------------------------------------------------------------
# Relay (WebSocket server)
# -----------------------------------------------------------------------------
# Bind address for the relay (host:port)
BUZZ_BIND_ADDR=0.0.0.0:3000
# Public WebSocket URL — used in NIP-42 auth challenges
RELAY_URL=ws://localhost:3000
# Stable relay signing key (required). `just bootstrap` generates a random key in
# the gitignored .env file. Preserve that value across restarts and backups.
# BUZZ_RELAY_PRIVATE_KEY=<32-byte hex private key>
# Optional: path to the web UI dist directory. When set, the relay serves
# the web frontend at / for browser requests. Leave unset for local dev
# (use `just web` for Vite HMR instead).
# BUZZ_WEB_DIR=./web/dist
# NIP-PL mobile push is an explicit deployment opt-in. A gateway URL alone
# never enables it. When enabled and the URL is absent, the canonical
# https://push.buzz.xyz/v1/deliveries/apns endpoint is used.
BUZZ_PUSH_ENABLED=false
# BUZZ_PUSH_GATEWAY_DELIVERY_URL=https://push.buzz.xyz/v1/deliveries/apns
# -----------------------------------------------------------------------------
# Admin Dashboard (private moderation surface)
# -----------------------------------------------------------------------------
# Host name that serves the moderation dashboard and its /api/admin/v1
# endpoints. Leave unset to keep the admin surface absent.
# BUZZ_ADMIN_HOST=admin.localhost:3000
#
# Authentication mode. Accepted values: nip98 (default), disabled.
# Any other value is a startup error. Token authentication was removed:
# BUZZ_ADMIN_TOKEN is ignored with a startup warning — remove it from the environment.
# BUZZ_ADMIN_AUTH=nip98
#
# Option A — BUZZ_ADMIN_AUTH=nip98 (Nostr pubkey-based auth, default):
# NIP-98 HTTP Auth. Each request must carry an Authorization: Nostr header
# with a signed kind-27235 event. Authorized principals are resolved from:
# 1. RELAY_OPERATOR_PUBKEYS — comma-separated 64-char hex pubkeys (config Operators).
# 2. RELAY_OWNER_PUBKEY — implicit Operator fallback when RELAY_OPERATOR_PUBKEYS is unset.
# 3. relay_operators table — DB-managed Operator/Moderator roster.
# The dashboard requires a NIP-07 browser extension.
# Setting RELAY_OPERATOR_PUBKEYS for the admin console does NOT require
# RELAY_OPERATOR_API_ORIGIN; that origin is only for community provisioning
# (see below). When BUZZ_ADMIN_HOST is set, the relay advertises the admin
# origin in its NIP-11 document (`admin_api` field) so clients can auto-discover
# the console without manual URL entry.
# RELAY_OPERATOR_PUBKEYS=<64-char hex pubkey>[,<64-char hex pubkey>...]
#
# Option B — BUZZ_ADMIN_AUTH=disabled (network-layer auth only):
# Set only when the admin API is already protected at the network layer
# (VPN, private ingress). The relay logs a WARN on every startup.
# `just admin` defaults to this mode for local review.
#
# Directory holding the built dashboard assets (`pnpm -C admin-web build`).
# BUZZ_ADMIN_WEB_DIR=./admin-web/dist
#
# Canonical origin (http(s)://host[:port], no path) that community-provisioning
# NIP-98 requests are verified against. Required only to USE the provisioning
# endpoints (POST /operator/communities) — not for the admin console. When
# RELAY_OPERATOR_PUBKEYS is set but this is unset, the relay boots with a WARN
# and provisioning requests fail closed until it is set.
# RELAY_OPERATOR_API_ORIGIN=http://127.0.0.1:3000
# Optional relay-owned KLIPY key. When set, NIP-11 advertises GIF search and
# authenticated desktop clients use this relay as the metadata/search proxy.
# Keep the real value in your deployment's secret manager; never commit it.
# BUZZ_KLIPY_API_KEY=
# Shared Redis-backed admission limits. Defaults shown below; each value must
# be a positive integer.
# BUZZ_RATE_LIMIT_HUMAN_MESSAGES_PER_MIN=60
# BUZZ_RATE_LIMIT_GIF_SEARCHES_PER_MIN=30
# BUZZ_RATE_LIMIT_HUMAN_API_CALLS_PER_MIN=300
# BUZZ_RATE_LIMIT_HUMAN_WS_EVENTS_PER_SEC=10
# BUZZ_RATE_LIMIT_AGENT_STANDARD_MESSAGES_PER_MIN=120
# BUZZ_RATE_LIMIT_AGENT_STANDARD_API_CALLS_PER_MIN=600
# BUZZ_RATE_LIMIT_AGENT_ELEVATED_MESSAGES_PER_MIN=300
# BUZZ_RATE_LIMIT_AGENT_PLATFORM_MESSAGES_PER_MIN=600
# -----------------------------------------------------------------------------
# Git (NIP-34 bare repositories)
# -----------------------------------------------------------------------------
# Root directory for ephemeral Git workspaces and the disposable pack cache.
# Default: ./repos (relative to CWD).
# BUZZ_GIT_REPO_PATH=./repos
# BUZZ_GIT_MAX_PACK_BYTES=524288000
# BUZZ_GIT_MAX_REPO_BYTES=1048576000
# Process-local immutable pack/index cache. Zero disables retention.
# BUZZ_GIT_PACK_CACHE_PATH=./repos/.pack-cache
# BUZZ_GIT_PACK_CACHE_MAX_BYTES=5368709120
# BUZZ_GIT_PACK_CACHE_MAX_CONCURRENT_POPULATIONS=2
# -----------------------------------------------------------------------------
# S3-Compatible Object Storage (media + Git/CAS)
# -----------------------------------------------------------------------------
# The local MinIO container is reachable from host processes at localhost:9000.
# Path style keeps the bucket in the URL path and is required by this local DNS
# setup. Use `virtual` only when the provider requires bucket-as-subdomain URLs.
BUZZ_S3_ENDPOINT=http://localhost:9000
BUZZ_S3_ACCESS_KEY=buzz_dev
BUZZ_S3_SECRET_KEY=buzz_dev_secret
BUZZ_S3_BUCKET=buzz-media
BUZZ_S3_REGION=us-east-1
BUZZ_S3_ADDRESSING_STYLE=path
# -----------------------------------------------------------------------------
# Media Upload Admission
# -----------------------------------------------------------------------------
# Bound image/video parser and storage work admitted by one relay process.
# BUZZ_MEDIA_MAX_CONCURRENT_UPLOADS=8
# BUZZ_MEDIA_MAX_CONCURRENT_UPLOADS_PER_PUBKEY=2
# BUZZ_MEDIA_UPLOADS_PER_MINUTE=30
# GET/HEAD /media/* always require Blossom t=get auth and relay membership.
# BUZZ_REQUIRE_MEDIA_GET_AUTH and BUZZ_REQUIRE_MEDIA_READ_AUTH are no longer
# read; setting either (including to false) changes nothing and the relay warns
# about it at startup.
# -----------------------------------------------------------------------------
# Ephemeral Channels (TTL testing)
# -----------------------------------------------------------------------------
# Override the TTL for all ephemeral channels (in seconds). When set, any
# channel created with a TTL tag will use this value instead of the
# client-provided one. Unset to use the client-provided TTL.
# BUZZ_EPHEMERAL_TTL_OVERRIDE=60
# How often the reaper checks for expired ephemeral channels (default: 60s).
# BUZZ_REAPER_INTERVAL_SECS=5
# -----------------------------------------------------------------------------
# Logging / Tracing
# -----------------------------------------------------------------------------
RUST_LOG=buzz_relay=debug,buzz_datastore=info,buzz_db=debug,buzz_auth=debug,buzz_pubsub=debug,tower_http=debug
# Optional OpenTelemetry-only target filter. This is deliberately independent
# from RUST_LOG so log verbosity changes cannot break trace parentage.
# BUZZ_OTEL_FILTER=buzz_relay=info,buzz_datastore=info
# OTLP tracing endpoint (optional — leave unset to disable)
# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
# -----------------------------------------------------------------------------
# ACP (Agent Communication Protocol — buzz-acp harness)
# -----------------------------------------------------------------------------
# The ACP harness bridges Buzz events to AI agents. Each env var below maps
# to a CLI flag of the same name (lowercase, hyphens → underscores). All values
# are optional unless noted; defaults are shown in comments.
#
# Quick start:
# BUZZ_PRIVATE_KEY=<hex> BUZZ_RELAY_URL=ws://localhost:3000 buzz-acp
# ── Identity & auth ──────────────────────────────────────────────────────────
# Nostr private key (hex or bech32). REQUIRED — identifies the agent on the relay.
# BUZZ_PRIVATE_KEY=<32-byte hex or nsec1… private key>
# Relay WebSocket URL the harness connects to.
# Note: the relay itself uses RELAY_URL (above); this is the ACP harness's
# connection target — they happen to point at the same place in local dev.
# BUZZ_RELAY_URL=ws://localhost:3000
# ── Agent subprocess ─────────────────────────────────────────────────────────
# Binary to spawn as the AI agent (e.g. "goose", "codex-acp", "claude-code").
# BUZZ_ACP_AGENT_COMMAND=goose
# Comma-separated arguments passed to the agent binary.
# Goose default: "acp". Codex/Claude default: "" (empty).
# BUZZ_ACP_AGENT_ARGS=acp
# Binary for an optional MCP server sidecar (e.g. buzz-dev-mcp for buzz-agent).
# BUZZ_ACP_MCP_COMMAND=
# Number of parallel agent subprocesses (1–32).
# BUZZ_ACP_AGENTS=1
# Desired LLM model ID. Applied to every new ACP session.
# Use `buzz-acp models` to discover available model IDs.
# BUZZ_ACP_MODEL=
# Optional Databricks model-picker visibility filter. Discovery-only; this does
# not grant inference access. Comma-separated full-string * / ? patterns are
# OR-matched against raw workspace endpoint and Unity Catalog model-service IDs.
# Unset or blank shows every catalog entry. A nonblank value with no usable patterns is invalid.
# DATABRICKS_MODEL_FILTER=databricks-*,data_tools.goose.*
# ── Timeouts & sessions ──────────────────────────────────────────────────────
# Max seconds per agent turn before timeout (default 320 = ~5 min).
# BUZZ_ACP_TURN_TIMEOUT=320
# Max turns per session before proactive rotation. 0 = disabled (rotate only
# on MaxTokens / MaxTurnRequests). Recommended: 50 for long-running agents.
# BUZZ_ACP_MAX_TURNS_PER_SESSION=0
# ── Prompts ──────────────────────────────────────────────────────────────────
# System prompt injected into every agent session (inline text).
# BUZZ_ACP_SYSTEM_PROMPT=
# Path to a file containing the system prompt (mutually exclusive with above).
# BUZZ_ACP_SYSTEM_PROMPT_FILE=
# Message sent to the agent immediately after session creation.
# BUZZ_ACP_INITIAL_MESSAGE=
# ── Heartbeat ────────────────────────────────────────────────────────────────
# Seconds between heartbeat prompts. 0 = disabled. Must be 0 or ≥10.
# Recommended: 60 for long-running agents to prevent idle session timeouts.
# BUZZ_ACP_HEARTBEAT_INTERVAL=0
# Heartbeat prompt text (inline). Mutually exclusive with file variant.
# BUZZ_ACP_HEARTBEAT_PROMPT=
# Path to a file containing the heartbeat prompt.
# BUZZ_ACP_HEARTBEAT_PROMPT_FILE=
# ── Desktop development ──────────────────────────────────────────────────────
# DEV-only: replay first-run onboarding and the Welcome Team kickoff on each
# app launch while keeping the current identity and relay data.
# VITE_BUZZ_FORCE_FRESH_ONBOARDING=true
# ── Subscription & filtering ─────────────────────────────────────────────────
# Subscribe mode: "mentions" (default), "all", or "config" (rule-based).
# BUZZ_ACP_SUBSCRIBE=mentions
# Comma-separated event kind numbers to subscribe to (overrides mode defaults).
# BUZZ_ACP_KINDS=
# Comma-separated channel UUIDs to limit subscription scope.
# BUZZ_ACP_CHANNELS=
# Set to true to disable the @-mention filter in mentions mode.
# BUZZ_ACP_NO_MENTION_FILTER=false
# Path to TOML config file for rule-based subscriptions (config mode).
# BUZZ_ACP_CONFIG=./buzz-acp.toml
# ── Dedup & self-ignore ──────────────────────────────────────────────────────
# How to handle duplicate events: "queue" (default) or "drop".
# BUZZ_ACP_DEDUP=queue
# Set to true to process the agent's own messages (default: ignore self).
# BUZZ_ACP_NO_IGNORE_SELF=false
# ── Context ──────────────────────────────────────────────────────────────────
# Max context messages fetched for thread replies and DMs (0–100). 0 = disabled.
# BUZZ_ACP_CONTEXT_MESSAGE_LIMIT=12
# ── Presence & typing ────────────────────────────────────────────────────────
# Set to true to disable automatic online/offline presence status.
# BUZZ_ACP_NO_PRESENCE=false
# Set to true to disable typing indicators while the agent is processing.
# BUZZ_ACP_NO_TYPING=false
# ── Advanced tuning ──────────────────────────────────────────────────────────
# Event channel buffer capacity (WebSocket → harness). Increase for
# high-throughput agents. Minimum 1.
# BUZZ_ACP_EVENT_BUFFER=256
# ── Legacy aliases ───────────────────────────────────────────────────────────
# These are accepted for backward compatibility but the canonical names above
# are preferred:
# BUZZ_ACP_PRIVATE_KEY → BUZZ_PRIVATE_KEY
# Optional relay join policy. Markdown is served by the relay so every join
# surface can present the same documents. Each document and the independent age
# attestation are optional; configuring any one enables policy acceptance.
# BUZZ_TERMS_OF_SERVICE_MARKDOWN="# Terms of Service\n\nFull terms here."
# BUZZ_PRIVACY_POLICY_MARKDOWN="# Privacy Policy\n\nFull policy here."
# BUZZ_AGE_ATTESTATION_REQUIRED=true