fix(relay): advance the partition catch-all by default (#8464)

## Summary

The migrations leave `events` and `delivery_log` with a catch-all
partition that already covers every month from 2026-07. Only catch-all
advancement replaces it with monthly partitions. Advancement was off by
default, so any deployment that left out
`BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` routed every new row into the
catch-all. An operator then had to repair it by hand: drain the
catch-all, swap it for monthly partitions, and re-attach. This has
already happened in one staging environment.

- **Advancement on by default.**
`BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` now defaults to `true`, and an
explicit `false` still disables it.
`BUZZ_PARTITION_MANAGER_CREATE_ENABLED=false` remains the kill switch
for all automatic partition DDL.
- **Prompt retry while the current month routes to the catch-all.** If a
startup or periodic pass leaves the current month routed to an empty
catch-all that the maintenance planner would replace, the audit loop
retries on its existing backoff. That is 5 s, doubling to a 60 s cap,
for at most 8 retries (about 5 minutes). After that it returns to
`BUZZ_PARTITION_AUDIT_INTERVAL_SECS`. Before this change, a failed first
pass, such as a lock timeout during a rollout, waited the full 15
minutes.
- The routine monthly advance, where only the last lookahead month falls
into the catch-all, stays on the normal interval. Each attempt queues an
`ACCESS EXCLUSIVE` request on the parent, so it shouldn't be retried
without urgency.
  - The budget resets once the catch-all no longer awaits advancement.
  - A failed attempt keeps the last known state.
- Layouts the planner refuses, such as a catch-all that holds rows, need
an operator, so they are never retried early. Neither is a policy with
create or advance disabled.
- The check is `PartitionTableAudit::awaits_catch_all_advance(policy)`,
plus a `PartitionAudit` helper that applies it to every table. It is
derived from `plan_maintenance`, so it can't drift from the planner's
rules.
- **Fresh-install test.** A new `#[ignore = "requires Postgres"]` test
runs every migration in a dedicated database. It checks that both tables
start out waiting on the catch-all and that create-only maintenance does
not fix them. It then checks that the first pass with advancement
produces canonical monthlies, an empty `{table}_p_future`, trigger
parity and a non-degraded audit, and that a second pass is a no-op.

### Related issue

None found.

### Testing

**Automated:**
- `cargo test -p buzz-db --lib partition`, including `--ignored` against
local Postgres: 21 unit tests and 55 Postgres tests pass.
- `cargo test -p buzz-relay` for the config tests and the partition
schedule tests passes.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings`
and `cargo fmt --check` are clean.
- Mutation checks on `awaits_catch_all_advance`. Each fails the new unit
test:
  - matching any window month instead of the current month;
  - replacing the planner check with the policy flags alone;
  - short-circuiting the current-month gate.

**Local relay** (`BUZZ_AUTO_MIGRATE=true`, no advance env var):
- **Fresh database:** startup committed `advanced` for both tables,
creating `p2026_07` through `p2027_04` plus `_p_future`.
- **Failed first pass:** I held `LOCK TABLE ONLY communities IN SHARE
ROW EXCLUSIVE MODE` for 4 s across startup. The startup pass failed with
`lock_timeout` at 17:07:20.8, and the first periodic retry committed
`advanced` for both tables at 17:07:29.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ian Oberst
2026-10-20 09:20:18 -07:00
committed by GitHub
parent 039e1c8d59
commit ca3be55018
6 changed files with 328 additions and 38 deletions

View File

@@ -52,10 +52,10 @@ REDIS_URL=redis://localhost:6379
# Emergency kill switch for all automatic partition DDL, including catch-all
# advancement; audits remain enabled.
# BUZZ_PARTITION_MANAGER_CREATE_ENABLED=true
# Opt in to replacing an empty partition catch-all with six months of dedicated
# monthlies, at startup and on every audit interval (default false). Has no
# effect while the kill switch above is off.
# BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED=false
# Replace an empty partition catch-all with six months of dedicated monthlies,
# at startup and on every audit interval (default true). Has no effect while the
# kill switch above is off.
# BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED=true
# Writer-session Postgres timeouts for buzz-db-backed pools and the relay audit
# pool, all in milliseconds; 0 disables. The separately deployed push gateway

View File

@@ -315,6 +315,38 @@ fn plan_maintenance(
})
}
impl PartitionTableAudit {
/// Whether rows timestamped now route to an empty catch-all that a
/// maintenance pass under `policy` would replace.
///
/// Only the current month counts: a catch-all that serves just the far
/// end of the lookahead window is the routine monthly advance, which can
/// wait for the next pass. Any layout the planner refuses, such as an
/// occupied catch-all, needs an operator and is false.
pub fn awaits_catch_all_advance(&self, policy: PartitionMaintenancePolicy) -> bool {
self.months
.first()
.is_some_and(|month| month.kind == MonthCoverageKind::CoveredByCatchAll)
&& matches!(
plan_maintenance(self, policy),
MaintenancePlan::Apply(MaintenanceChange {
catch_all: Some(_),
..
})
)
}
}
impl PartitionAudit {
/// Whether any managed parent awaits catch-all advancement. See
/// [`PartitionTableAudit::awaits_catch_all_advance`].
pub fn awaits_catch_all_advance(&self, policy: PartitionMaintenancePolicy) -> bool {
self.tables
.iter()
.any(|table| table.awaits_catch_all_advance(policy))
}
}
/// Audit, then issue only the DDL `policy` permits, one bounded transaction per table.
///
/// A table that is already at the desired layout, or whose required DDL is

View File

@@ -126,6 +126,83 @@ fn advancement_requires_its_own_policy_flag() {
);
}
/// The repaired layout once January 2027 arrives: the current month and the
/// whole lookahead route to the empty catch-all, as on a fresh install.
fn catch_all_serving_now_audit() -> PartitionTableAudit {
let mut audit = repaired_audit(false);
audit.months = (0..=PARTITION_MANAGER_MONTHS_AHEAD as i32)
.map(|offset| {
let (year, m) = add_months(2027, 1, offset).unwrap();
MonthCoverage {
start: month(year, m),
kind: MonthCoverageKind::CoveredByCatchAll,
}
})
.collect();
audit
}
fn with_catch_all(
mut audit: PartitionTableAudit,
edit: impl Fn(&mut PartitionChildAudit),
) -> PartitionTableAudit {
for child in &mut audit.children {
if child.kind == PartitionChildKind::CatchAll {
edit(child);
}
}
audit
}
#[test]
fn only_a_catch_all_serving_the_current_month_awaits_advance() {
let serving = catch_all_serving_now_audit();
assert!(serving.awaits_catch_all_advance(ADVANCE));
// The routine monthly advance: the planner would advance, but only the end
// of the window routes to the catch-all, so it waits for the next pass.
let routine = repaired_audit(false);
assert!(matches!(
plan_maintenance(&routine, ADVANCE),
MaintenancePlan::Apply(_)
));
assert!(!routine.awaits_catch_all_advance(ADVANCE));
// Advancement disabled, including by the create kill switch.
for policy in [
CREATE_ONLY,
PartitionMaintenancePolicy {
create_enabled: false,
advance_enabled: true,
},
PartitionMaintenancePolicy::default(),
] {
assert!(!serving.awaits_catch_all_advance(policy), "{policy:?}");
}
// Layouts the planner refuses need an operator, not prompt retries.
let occupied = with_catch_all(serving.clone(), |child| {
child.catch_all_nonempty = Some(true);
});
let child_triggers = with_catch_all(serving.clone(), |child| {
child.extra_triggers = vec!["child_only_probe".to_string()];
});
for refused in [&occupied, &child_triggers] {
assert!(matches!(
plan_maintenance(refused, ADVANCE),
MaintenancePlan::Refuse(_)
));
assert!(!refused.awaits_catch_all_advance(ADVANCE));
}
let catalog = |tables| PartitionAudit {
audited_at: month(2027, 1),
tables,
};
assert!(catalog(vec![routine.clone(), serving]).awaits_catch_all_advance(ADVANCE));
assert!(!catalog(vec![routine, occupied]).awaits_catch_all_advance(ADVANCE));
}
#[test]
fn full_monthly_runway_is_a_noop_even_when_disabled() {
let mut audit = repaired_audit(false);
@@ -1306,4 +1383,86 @@ mod postgres_tests {
.any(|(name, ..)| name == "events_p_future_next"));
assert_ne!(after, before, "delivery_log still advances");
}
/// Audits taken in a freshly migrated database before and after its first passes.
struct FreshInstallPasses {
migrated: PartitionAudit,
create_only: PartitionAudit,
advanced: PartitionAudit,
repeated: PartitionAudit,
}
/// Runs every migration in a dedicated database, then the passes a relay
/// would run at `fixed_now()`. The database is dropped before returning.
async fn fresh_install_passes() -> Result<FreshInstallPasses> {
let base_url = crate::test_support::database_url();
let admin = PgPool::connect(&base_url)
.await
.expect("connect admin database");
let database = format!("buzz_partition_fresh_{}", uuid::Uuid::new_v4().simple());
sqlx::query(sqlx::AssertSqlSafe(format!("CREATE DATABASE {database}")))
.execute(&admin)
.await
.expect("create fresh-install database");
let (base_prefix, _) = base_url.rsplit_once('/').expect("database url has a path");
let passes = async {
let pool = PgPool::connect(&format!("{base_prefix}/{database}")).await?;
let result = async {
crate::migration::run_migrations(&pool).await?;
let now = fixed_now();
Ok(FreshInstallPasses {
migrated: audit_partition_catalog_at(
&pool,
PARTITION_MANAGER_MONTHS_AHEAD,
now,
)
.await?,
create_only: maintain_partitions_at(
&pool,
PARTITION_MANAGER_MONTHS_AHEAD,
CREATE_ONLY,
now,
)
.await?,
advanced: maintain(&pool).await?,
repeated: maintain(&pool).await?,
})
}
.await;
pool.close().await;
result
}
.await;
sqlx::query(sqlx::AssertSqlSafe(format!(
"DROP DATABASE {database} WITH (FORCE)"
)))
.execute(&admin)
.await
.expect("drop fresh-install database");
passes
}
#[tokio::test]
#[ignore = "requires Postgres"]
async fn fresh_install_catch_all_advances_on_the_first_pass() {
let passes = fresh_install_passes().await.expect("fresh-install passes");
// The migrations end every managed table in an empty catch-all that
// already serves months, and creation alone cannot replace it.
for audit in [&passes.migrated, &passes.create_only] {
assert_eq!(audit.tables.len(), PARTITIONED_TABLES.len());
assert!(audit.awaits_catch_all_advance(ADVANCE), "{audit:?}");
for table in &audit.tables {
assert!(table.awaits_catch_all_advance(ADVANCE), "{}", table.table);
}
}
assert_advanced(&passes.advanced);
assert!(!passes.advanced.awaits_catch_all_advance(ADVANCE));
assert!(passes.advanced.serving_safe());
for table in &passes.advanced.tables {
assert!(!table.degraded(), "{}: {table:?}", table.table);
}
assert_eq!(passes.repeated, passes.advanced);
}
}

View File

@@ -211,7 +211,7 @@ pub struct Config {
pub partition_manager_create_enabled: bool,
/// Whether the partition manager may replace an empty right-edge catch-all
/// with dedicated monthlies. Also enables periodic (not only startup) DDL.
/// Requires `partition_manager_create_enabled`.
/// On by default; requires `partition_manager_create_enabled`.
pub partition_manager_advance_enabled: bool,
/// When true, NIP-42 pubkey-only authentication (no API token) is
@@ -953,9 +953,10 @@ impl Config {
);
let partition_manager_create_enabled =
parse_bool("BUZZ_PARTITION_MANAGER_CREATE_ENABLED", true)?;
// Off by default: each environment opts in after its canary.
// On by default: a fresh install's migrated catch-all covers months
// from 2026-07, and only advancement replaces it with monthlies.
let partition_manager_advance_enabled =
parse_bool("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", false)?;
parse_bool("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", true)?;
let s3_addressing_style = match std::env::var("BUZZ_S3_ADDRESSING_STYLE") {
Ok(value) => value.parse().map_err(ConfigError::InvalidValue)?,
@@ -1676,7 +1677,7 @@ mod tests {
assert_eq!(config.max_frame_bytes, DEFAULT_MAX_FRAME_BYTES);
assert_eq!(config.partition_audit_interval, Duration::from_secs(900));
assert!(config.partition_manager_create_enabled);
assert!(!config.partition_manager_advance_enabled);
assert!(config.partition_manager_advance_enabled);
assert!(config.slow_client_grace_limit > 0);
assert!(
!config.pubkey_allowlist_enabled,
@@ -2182,7 +2183,7 @@ mod tests {
}
#[test]
fn partition_manager_advance_is_opt_in_and_parses_strictly() {
fn partition_manager_advance_defaults_on_and_parses_strictly() {
let _guards = env_guards();
let previous = std::env::var_os("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED");
@@ -2190,6 +2191,8 @@ mod tests {
let default = Config::from_env().expect("default config");
std::env::set_var("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", " TRUE ");
let enabled = Config::from_env().expect("enabled config");
std::env::set_var("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", "false");
let disabled = Config::from_env().expect("disabled config");
std::env::set_var("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", "enable-maybe");
let invalid = Config::from_env();
@@ -2199,8 +2202,9 @@ mod tests {
std::env::remove_var("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED");
}
assert!(!default.partition_manager_advance_enabled);
assert!(default.partition_manager_advance_enabled);
assert!(enabled.partition_manager_advance_enabled);
assert!(!disabled.partition_manager_advance_enabled);
assert!(matches!(
invalid,
Err(ConfigError::InvalidValue(ref message))

View File

@@ -295,44 +295,75 @@ where
}
}
/// Prompt retries allowed while rows timestamped now route to an empty catch-all
/// awaiting advancement. With the 5 s to 60 s backoff they span about five
/// minutes, after which the normal period applies until that changes.
const PARTITION_ADVANCE_RETRY_LIMIT: u32 = 8;
/// Retry missing diagnostics promptly, without tying audit recovery to readiness.
/// Delays start after each attempt finishes, so slow audits cannot cause bursts.
/// Also retries promptly, a bounded number of times, while the current month
/// routes to an empty catch-all awaiting advancement, so a failed first pass on
/// a fresh install does not wait a full period. Delays start after each attempt finishes, so slow audits
/// cannot cause bursts.
struct PartitionAuditSchedule {
period: std::time::Duration,
next_delay: std::time::Duration,
retry_delay: std::time::Duration,
has_completed_audit: bool,
advance_pending: bool,
advance_retries_left: u32,
}
impl PartitionAuditSchedule {
fn new(period: std::time::Duration, has_completed_audit: bool) -> Self {
Self {
/// `startup` is the startup attempt's outcome, as for [`Self::record_attempt`].
fn new(period: std::time::Duration, startup: Option<bool>) -> Self {
let mut schedule = Self {
period,
next_delay: if has_completed_audit {
period
} else {
std::time::Duration::ZERO
},
next_delay: std::time::Duration::ZERO,
retry_delay: std::time::Duration::from_secs(5),
has_completed_audit,
has_completed_audit: false,
advance_pending: false,
advance_retries_left: PARTITION_ADVANCE_RETRY_LIMIT,
};
if startup.is_some() {
schedule.record_attempt(startup);
}
schedule
}
async fn wait(&self) {
tokio::time::sleep(self.next_delay).await;
}
/// A completed audit counts even when its verdict is unsafe.
fn record_attempt(&mut self, completed: bool) {
self.has_completed_audit |= completed;
if self.has_completed_audit {
self.next_delay = self.period;
/// Records one attempt: `None` when no audit completed, otherwise whether
/// catch-all advancement is still pending. A completed audit counts even
/// when its verdict is unsafe. A failed attempt keeps the last known
/// pending state.
fn record_attempt(&mut self, advance_pending: Option<bool>) {
if let Some(pending) = advance_pending {
self.has_completed_audit = true;
self.advance_pending = pending;
}
if !self.advance_pending {
self.advance_retries_left = PARTITION_ADVANCE_RETRY_LIMIT;
}
let retry = if !self.has_completed_audit {
true
} else if self.advance_pending && self.advance_retries_left > 0 {
self.advance_retries_left -= 1;
true
} else {
false
};
if retry {
self.next_delay = self.retry_delay;
self.retry_delay = self
.retry_delay
.saturating_mul(2)
.min(std::time::Duration::from_secs(60));
} else {
self.next_delay = self.period;
self.retry_delay = std::time::Duration::from_secs(5);
}
}
}
@@ -906,7 +937,9 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
)
.await;
}
let has_startup_partition_audit = startup_partition_audit.is_some();
let startup_partition_attempt = startup_partition_audit
.as_ref()
.map(|audit| audit.awaits_catch_all_advance(partition_policy));
if let Some(audit) = startup_partition_audit {
state.record_partition_audit(audit);
}
@@ -919,7 +952,7 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
let audit_interval = state.config.partition_audit_interval;
tokio::spawn(async move {
let mut schedule =
PartitionAuditSchedule::new(audit_interval, has_startup_partition_audit);
PartitionAuditSchedule::new(audit_interval, startup_partition_attempt);
loop {
schedule.wait().await;
let db = &partition_state.db;
@@ -941,7 +974,12 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
counted_partition_audit(db.audit_partitions(PARTITION_MANAGER_MONTHS_AHEAD))
.await
};
schedule.record_attempt(result.is_ok());
schedule.record_attempt(
result
.as_ref()
.ok()
.map(|audit| audit.awaits_catch_all_advance(partition_policy)),
);
match result {
Ok(audit) => partition_state.record_partition_audit(audit),
Err(error) => warn!(%error, "Periodic partition catalog audit failed"),
@@ -3196,12 +3234,12 @@ mod tests {
let period = Duration::from_secs(900);
let immediate_start = tokio::time::Instant::now();
let mut schedule = PartitionAuditSchedule::new(period, false);
let mut schedule = PartitionAuditSchedule::new(period, None);
schedule.wait().await;
assert_eq!(tokio::time::Instant::now(), immediate_start);
// Failure of the first periodic attempt must not delay recovery by 15 minutes.
schedule.record_attempt(false);
schedule.record_attempt(None);
schedule.wait().await;
assert_eq!(
tokio::time::Instant::now() - immediate_start,
@@ -3209,18 +3247,27 @@ mod tests {
);
let delayed_start = tokio::time::Instant::now();
let schedule = PartitionAuditSchedule::new(period, true);
let schedule = PartitionAuditSchedule::new(period, Some(false));
schedule.wait().await;
assert_eq!(tokio::time::Instant::now() - delayed_start, period);
// A startup pass that left an empty catch-all awaiting advance retries soon.
let pending_start = tokio::time::Instant::now();
let schedule = PartitionAuditSchedule::new(period, Some(true));
schedule.wait().await;
assert_eq!(
tokio::time::Instant::now() - pending_start,
Duration::from_secs(5)
);
}
#[tokio::test(start_paused = true)]
async fn partition_audit_recovery_backoff_is_bounded_and_completion_ends_retries() {
let period = Duration::from_secs(900);
let mut schedule = PartitionAuditSchedule::new(period, false);
let mut schedule = PartitionAuditSchedule::new(period, None);
schedule.wait().await;
for seconds in [5, 10, 20, 40, 60, 60, 60] {
schedule.record_attempt(false);
schedule.record_attempt(None);
let start = tokio::time::Instant::now();
schedule.wait().await;
assert_eq!(
@@ -3231,8 +3278,8 @@ mod tests {
// Completion, not a serving-safe verdict, ends recovery. Later failures
// retain the cached audit and use the normal cadence.
for completed in [true, false, false] {
schedule.record_attempt(completed);
for attempt in [Some(false), None, None] {
schedule.record_attempt(attempt);
let start = tokio::time::Instant::now();
schedule.wait().await;
assert_eq!(tokio::time::Instant::now() - start, period);
@@ -3241,10 +3288,10 @@ mod tests {
#[tokio::test(start_paused = true)]
async fn partition_audit_retries_wait_after_slow_attempts() {
let mut schedule = PartitionAuditSchedule::new(Duration::from_secs(900), false);
let mut schedule = PartitionAuditSchedule::new(Duration::from_secs(900), None);
schedule.wait().await;
tokio::time::advance(Duration::from_secs(120)).await;
schedule.record_attempt(false);
schedule.record_attempt(None);
let finished = tokio::time::Instant::now();
schedule.wait().await;
assert_eq!(
@@ -3253,6 +3300,54 @@ mod tests {
);
}
/// Waits one scheduled delay and returns how long it was.
async fn partition_audit_delay(schedule: &PartitionAuditSchedule) -> Duration {
let start = tokio::time::Instant::now();
schedule.wait().await;
tokio::time::Instant::now() - start
}
#[tokio::test(start_paused = true)]
async fn partition_advance_retries_are_bounded_and_reset_once_advanced() {
let period = Duration::from_secs(900);
let mut schedule = PartitionAuditSchedule::new(period, Some(true));
let mut delays = vec![partition_audit_delay(&schedule).await.as_secs()];
// Failed attempts keep the pending state and spend the same budget.
for attempt in [
None,
Some(true),
None,
Some(true),
Some(true),
Some(true),
Some(true),
] {
schedule.record_attempt(attempt);
delays.push(partition_audit_delay(&schedule).await.as_secs());
}
assert_eq!(delays, [5, 10, 20, 40, 60, 60, 60, 60]);
assert_eq!(
usize::try_from(super::PARTITION_ADVANCE_RETRY_LIMIT).ok(),
Some(delays.len())
);
// The budget is spent: a catch-all that never advances falls back to
// the normal period instead of retrying forever.
for attempt in [Some(true), None, Some(true)] {
schedule.record_attempt(attempt);
assert_eq!(partition_audit_delay(&schedule).await, period);
}
// Advancing resets the budget and backoff for a later pending catch-all.
schedule.record_attempt(Some(false));
assert_eq!(partition_audit_delay(&schedule).await, period);
schedule.record_attempt(Some(true));
assert_eq!(
partition_audit_delay(&schedule).await,
Duration::from_secs(5)
);
}
fn partition_audit_ok() -> buzz_db::Result<buzz_db::partition::PartitionAudit> {
Ok(buzz_db::partition::PartitionAudit {
audited_at: chrono::Utc::now(),

View File

@@ -65,8 +65,8 @@ run with a new outcome:
- `buzz_partition_create_attempts_total{table, outcome}` counts per-month
decisions in each maintenance pass: `created`, `error`, or `skipped_covered`
for a month that was already covered. Maintenance runs at startup, and on
every periodic audit when `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` is on, so
`skipped_covered` then grows on every pass in steady state.
every periodic audit when `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` is on (the
default), so `skipped_covered` then grows on every pass in steady state.
## Alerts