mirror of
https://github.com/block/buzz.git
synced 2026-10-27 04:30:52 +00:00
fix(relay): advance the partition catch-all by default (#8464)
## Summary
The migrations leave `events` and `delivery_log` with a catch-all
partition that already covers every month from 2026-07. Only catch-all
advancement replaces it with monthly partitions. Advancement was off by
default, so any deployment that left out
`BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` routed every new row into the
catch-all. An operator then had to repair it by hand: drain the
catch-all, swap it for monthly partitions, and re-attach. This has
already happened in one staging environment.
- **Advancement on by default.**
`BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` now defaults to `true`, and an
explicit `false` still disables it.
`BUZZ_PARTITION_MANAGER_CREATE_ENABLED=false` remains the kill switch
for all automatic partition DDL.
- **Prompt retry while the current month routes to the catch-all.** If a
startup or periodic pass leaves the current month routed to an empty
catch-all that the maintenance planner would replace, the audit loop
retries on its existing backoff. That is 5 s, doubling to a 60 s cap,
for at most 8 retries (about 5 minutes). After that it returns to
`BUZZ_PARTITION_AUDIT_INTERVAL_SECS`. Before this change, a failed first
pass, such as a lock timeout during a rollout, waited the full 15
minutes.
- The routine monthly advance, where only the last lookahead month falls
into the catch-all, stays on the normal interval. Each attempt queues an
`ACCESS EXCLUSIVE` request on the parent, so it shouldn't be retried
without urgency.
- The budget resets once the catch-all no longer awaits advancement.
- A failed attempt keeps the last known state.
- Layouts the planner refuses, such as a catch-all that holds rows, need
an operator, so they are never retried early. Neither is a policy with
create or advance disabled.
- The check is `PartitionTableAudit::awaits_catch_all_advance(policy)`,
plus a `PartitionAudit` helper that applies it to every table. It is
derived from `plan_maintenance`, so it can't drift from the planner's
rules.
- **Fresh-install test.** A new `#[ignore = "requires Postgres"]` test
runs every migration in a dedicated database. It checks that both tables
start out waiting on the catch-all and that create-only maintenance does
not fix them. It then checks that the first pass with advancement
produces canonical monthlies, an empty `{table}_p_future`, trigger
parity and a non-degraded audit, and that a second pass is a no-op.
### Related issue
None found.
### Testing
**Automated:**
- `cargo test -p buzz-db --lib partition`, including `--ignored` against
local Postgres: 21 unit tests and 55 Postgres tests pass.
- `cargo test -p buzz-relay` for the config tests and the partition
schedule tests passes.
- `cargo clippy -p buzz-db -p buzz-relay --all-targets -- -D warnings`
and `cargo fmt --check` are clean.
- Mutation checks on `awaits_catch_all_advance`. Each fails the new unit
test:
- matching any window month instead of the current month;
- replacing the planner check with the policy flags alone;
- short-circuiting the current-month gate.
**Local relay** (`BUZZ_AUTO_MIGRATE=true`, no advance env var):
- **Fresh database:** startup committed `advanced` for both tables,
creating `p2026_07` through `p2027_04` plus `_p_future`.
- **Failed first pass:** I held `LOCK TABLE ONLY communities IN SHARE
ROW EXCLUSIVE MODE` for 4 s across startup. The startup pass failed with
`lock_timeout` at 17:07:20.8, and the first periodic retry committed
`advanced` for both tables at 17:07:29.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Ian Oberst <ioberst@block.xyz>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -52,10 +52,10 @@ REDIS_URL=redis://localhost:6379
|
||||
# Emergency kill switch for all automatic partition DDL, including catch-all
|
||||
# advancement; audits remain enabled.
|
||||
# BUZZ_PARTITION_MANAGER_CREATE_ENABLED=true
|
||||
# Opt in to replacing an empty partition catch-all with six months of dedicated
|
||||
# monthlies, at startup and on every audit interval (default false). Has no
|
||||
# effect while the kill switch above is off.
|
||||
# BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED=false
|
||||
# Replace an empty partition catch-all with six months of dedicated monthlies,
|
||||
# at startup and on every audit interval (default true). Has no effect while the
|
||||
# kill switch above is off.
|
||||
# BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED=true
|
||||
|
||||
# Writer-session Postgres timeouts for buzz-db-backed pools and the relay audit
|
||||
# pool, all in milliseconds; 0 disables. The separately deployed push gateway
|
||||
|
||||
@@ -315,6 +315,38 @@ fn plan_maintenance(
|
||||
})
|
||||
}
|
||||
|
||||
impl PartitionTableAudit {
|
||||
/// Whether rows timestamped now route to an empty catch-all that a
|
||||
/// maintenance pass under `policy` would replace.
|
||||
///
|
||||
/// Only the current month counts: a catch-all that serves just the far
|
||||
/// end of the lookahead window is the routine monthly advance, which can
|
||||
/// wait for the next pass. Any layout the planner refuses, such as an
|
||||
/// occupied catch-all, needs an operator and is false.
|
||||
pub fn awaits_catch_all_advance(&self, policy: PartitionMaintenancePolicy) -> bool {
|
||||
self.months
|
||||
.first()
|
||||
.is_some_and(|month| month.kind == MonthCoverageKind::CoveredByCatchAll)
|
||||
&& matches!(
|
||||
plan_maintenance(self, policy),
|
||||
MaintenancePlan::Apply(MaintenanceChange {
|
||||
catch_all: Some(_),
|
||||
..
|
||||
})
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
impl PartitionAudit {
|
||||
/// Whether any managed parent awaits catch-all advancement. See
|
||||
/// [`PartitionTableAudit::awaits_catch_all_advance`].
|
||||
pub fn awaits_catch_all_advance(&self, policy: PartitionMaintenancePolicy) -> bool {
|
||||
self.tables
|
||||
.iter()
|
||||
.any(|table| table.awaits_catch_all_advance(policy))
|
||||
}
|
||||
}
|
||||
|
||||
/// Audit, then issue only the DDL `policy` permits, one bounded transaction per table.
|
||||
///
|
||||
/// A table that is already at the desired layout, or whose required DDL is
|
||||
|
||||
@@ -126,6 +126,83 @@ fn advancement_requires_its_own_policy_flag() {
|
||||
);
|
||||
}
|
||||
|
||||
/// The repaired layout once January 2027 arrives: the current month and the
|
||||
/// whole lookahead route to the empty catch-all, as on a fresh install.
|
||||
fn catch_all_serving_now_audit() -> PartitionTableAudit {
|
||||
let mut audit = repaired_audit(false);
|
||||
audit.months = (0..=PARTITION_MANAGER_MONTHS_AHEAD as i32)
|
||||
.map(|offset| {
|
||||
let (year, m) = add_months(2027, 1, offset).unwrap();
|
||||
MonthCoverage {
|
||||
start: month(year, m),
|
||||
kind: MonthCoverageKind::CoveredByCatchAll,
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
audit
|
||||
}
|
||||
|
||||
fn with_catch_all(
|
||||
mut audit: PartitionTableAudit,
|
||||
edit: impl Fn(&mut PartitionChildAudit),
|
||||
) -> PartitionTableAudit {
|
||||
for child in &mut audit.children {
|
||||
if child.kind == PartitionChildKind::CatchAll {
|
||||
edit(child);
|
||||
}
|
||||
}
|
||||
audit
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_a_catch_all_serving_the_current_month_awaits_advance() {
|
||||
let serving = catch_all_serving_now_audit();
|
||||
assert!(serving.awaits_catch_all_advance(ADVANCE));
|
||||
|
||||
// The routine monthly advance: the planner would advance, but only the end
|
||||
// of the window routes to the catch-all, so it waits for the next pass.
|
||||
let routine = repaired_audit(false);
|
||||
assert!(matches!(
|
||||
plan_maintenance(&routine, ADVANCE),
|
||||
MaintenancePlan::Apply(_)
|
||||
));
|
||||
assert!(!routine.awaits_catch_all_advance(ADVANCE));
|
||||
|
||||
// Advancement disabled, including by the create kill switch.
|
||||
for policy in [
|
||||
CREATE_ONLY,
|
||||
PartitionMaintenancePolicy {
|
||||
create_enabled: false,
|
||||
advance_enabled: true,
|
||||
},
|
||||
PartitionMaintenancePolicy::default(),
|
||||
] {
|
||||
assert!(!serving.awaits_catch_all_advance(policy), "{policy:?}");
|
||||
}
|
||||
|
||||
// Layouts the planner refuses need an operator, not prompt retries.
|
||||
let occupied = with_catch_all(serving.clone(), |child| {
|
||||
child.catch_all_nonempty = Some(true);
|
||||
});
|
||||
let child_triggers = with_catch_all(serving.clone(), |child| {
|
||||
child.extra_triggers = vec!["child_only_probe".to_string()];
|
||||
});
|
||||
for refused in [&occupied, &child_triggers] {
|
||||
assert!(matches!(
|
||||
plan_maintenance(refused, ADVANCE),
|
||||
MaintenancePlan::Refuse(_)
|
||||
));
|
||||
assert!(!refused.awaits_catch_all_advance(ADVANCE));
|
||||
}
|
||||
|
||||
let catalog = |tables| PartitionAudit {
|
||||
audited_at: month(2027, 1),
|
||||
tables,
|
||||
};
|
||||
assert!(catalog(vec![routine.clone(), serving]).awaits_catch_all_advance(ADVANCE));
|
||||
assert!(!catalog(vec![routine, occupied]).awaits_catch_all_advance(ADVANCE));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn full_monthly_runway_is_a_noop_even_when_disabled() {
|
||||
let mut audit = repaired_audit(false);
|
||||
@@ -1306,4 +1383,86 @@ mod postgres_tests {
|
||||
.any(|(name, ..)| name == "events_p_future_next"));
|
||||
assert_ne!(after, before, "delivery_log still advances");
|
||||
}
|
||||
|
||||
/// Audits taken in a freshly migrated database before and after its first passes.
|
||||
struct FreshInstallPasses {
|
||||
migrated: PartitionAudit,
|
||||
create_only: PartitionAudit,
|
||||
advanced: PartitionAudit,
|
||||
repeated: PartitionAudit,
|
||||
}
|
||||
|
||||
/// Runs every migration in a dedicated database, then the passes a relay
|
||||
/// would run at `fixed_now()`. The database is dropped before returning.
|
||||
async fn fresh_install_passes() -> Result<FreshInstallPasses> {
|
||||
let base_url = crate::test_support::database_url();
|
||||
let admin = PgPool::connect(&base_url)
|
||||
.await
|
||||
.expect("connect admin database");
|
||||
let database = format!("buzz_partition_fresh_{}", uuid::Uuid::new_v4().simple());
|
||||
sqlx::query(sqlx::AssertSqlSafe(format!("CREATE DATABASE {database}")))
|
||||
.execute(&admin)
|
||||
.await
|
||||
.expect("create fresh-install database");
|
||||
let (base_prefix, _) = base_url.rsplit_once('/').expect("database url has a path");
|
||||
let passes = async {
|
||||
let pool = PgPool::connect(&format!("{base_prefix}/{database}")).await?;
|
||||
let result = async {
|
||||
crate::migration::run_migrations(&pool).await?;
|
||||
let now = fixed_now();
|
||||
Ok(FreshInstallPasses {
|
||||
migrated: audit_partition_catalog_at(
|
||||
&pool,
|
||||
PARTITION_MANAGER_MONTHS_AHEAD,
|
||||
now,
|
||||
)
|
||||
.await?,
|
||||
create_only: maintain_partitions_at(
|
||||
&pool,
|
||||
PARTITION_MANAGER_MONTHS_AHEAD,
|
||||
CREATE_ONLY,
|
||||
now,
|
||||
)
|
||||
.await?,
|
||||
advanced: maintain(&pool).await?,
|
||||
repeated: maintain(&pool).await?,
|
||||
})
|
||||
}
|
||||
.await;
|
||||
pool.close().await;
|
||||
result
|
||||
}
|
||||
.await;
|
||||
sqlx::query(sqlx::AssertSqlSafe(format!(
|
||||
"DROP DATABASE {database} WITH (FORCE)"
|
||||
)))
|
||||
.execute(&admin)
|
||||
.await
|
||||
.expect("drop fresh-install database");
|
||||
passes
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "requires Postgres"]
|
||||
async fn fresh_install_catch_all_advances_on_the_first_pass() {
|
||||
let passes = fresh_install_passes().await.expect("fresh-install passes");
|
||||
|
||||
// The migrations end every managed table in an empty catch-all that
|
||||
// already serves months, and creation alone cannot replace it.
|
||||
for audit in [&passes.migrated, &passes.create_only] {
|
||||
assert_eq!(audit.tables.len(), PARTITIONED_TABLES.len());
|
||||
assert!(audit.awaits_catch_all_advance(ADVANCE), "{audit:?}");
|
||||
for table in &audit.tables {
|
||||
assert!(table.awaits_catch_all_advance(ADVANCE), "{}", table.table);
|
||||
}
|
||||
}
|
||||
|
||||
assert_advanced(&passes.advanced);
|
||||
assert!(!passes.advanced.awaits_catch_all_advance(ADVANCE));
|
||||
assert!(passes.advanced.serving_safe());
|
||||
for table in &passes.advanced.tables {
|
||||
assert!(!table.degraded(), "{}: {table:?}", table.table);
|
||||
}
|
||||
assert_eq!(passes.repeated, passes.advanced);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -211,7 +211,7 @@ pub struct Config {
|
||||
pub partition_manager_create_enabled: bool,
|
||||
/// Whether the partition manager may replace an empty right-edge catch-all
|
||||
/// with dedicated monthlies. Also enables periodic (not only startup) DDL.
|
||||
/// Requires `partition_manager_create_enabled`.
|
||||
/// On by default; requires `partition_manager_create_enabled`.
|
||||
pub partition_manager_advance_enabled: bool,
|
||||
|
||||
/// When true, NIP-42 pubkey-only authentication (no API token) is
|
||||
@@ -953,9 +953,10 @@ impl Config {
|
||||
);
|
||||
let partition_manager_create_enabled =
|
||||
parse_bool("BUZZ_PARTITION_MANAGER_CREATE_ENABLED", true)?;
|
||||
// Off by default: each environment opts in after its canary.
|
||||
// On by default: a fresh install's migrated catch-all covers months
|
||||
// from 2026-07, and only advancement replaces it with monthlies.
|
||||
let partition_manager_advance_enabled =
|
||||
parse_bool("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", false)?;
|
||||
parse_bool("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", true)?;
|
||||
|
||||
let s3_addressing_style = match std::env::var("BUZZ_S3_ADDRESSING_STYLE") {
|
||||
Ok(value) => value.parse().map_err(ConfigError::InvalidValue)?,
|
||||
@@ -1676,7 +1677,7 @@ mod tests {
|
||||
assert_eq!(config.max_frame_bytes, DEFAULT_MAX_FRAME_BYTES);
|
||||
assert_eq!(config.partition_audit_interval, Duration::from_secs(900));
|
||||
assert!(config.partition_manager_create_enabled);
|
||||
assert!(!config.partition_manager_advance_enabled);
|
||||
assert!(config.partition_manager_advance_enabled);
|
||||
assert!(config.slow_client_grace_limit > 0);
|
||||
assert!(
|
||||
!config.pubkey_allowlist_enabled,
|
||||
@@ -2182,7 +2183,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn partition_manager_advance_is_opt_in_and_parses_strictly() {
|
||||
fn partition_manager_advance_defaults_on_and_parses_strictly() {
|
||||
let _guards = env_guards();
|
||||
let previous = std::env::var_os("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED");
|
||||
|
||||
@@ -2190,6 +2191,8 @@ mod tests {
|
||||
let default = Config::from_env().expect("default config");
|
||||
std::env::set_var("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", " TRUE ");
|
||||
let enabled = Config::from_env().expect("enabled config");
|
||||
std::env::set_var("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", "false");
|
||||
let disabled = Config::from_env().expect("disabled config");
|
||||
std::env::set_var("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED", "enable-maybe");
|
||||
let invalid = Config::from_env();
|
||||
|
||||
@@ -2199,8 +2202,9 @@ mod tests {
|
||||
std::env::remove_var("BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED");
|
||||
}
|
||||
|
||||
assert!(!default.partition_manager_advance_enabled);
|
||||
assert!(default.partition_manager_advance_enabled);
|
||||
assert!(enabled.partition_manager_advance_enabled);
|
||||
assert!(!disabled.partition_manager_advance_enabled);
|
||||
assert!(matches!(
|
||||
invalid,
|
||||
Err(ConfigError::InvalidValue(ref message))
|
||||
|
||||
@@ -295,44 +295,75 @@ where
|
||||
}
|
||||
}
|
||||
|
||||
/// Prompt retries allowed while rows timestamped now route to an empty catch-all
|
||||
/// awaiting advancement. With the 5 s to 60 s backoff they span about five
|
||||
/// minutes, after which the normal period applies until that changes.
|
||||
const PARTITION_ADVANCE_RETRY_LIMIT: u32 = 8;
|
||||
|
||||
/// Retry missing diagnostics promptly, without tying audit recovery to readiness.
|
||||
/// Delays start after each attempt finishes, so slow audits cannot cause bursts.
|
||||
/// Also retries promptly, a bounded number of times, while the current month
|
||||
/// routes to an empty catch-all awaiting advancement, so a failed first pass on
|
||||
/// a fresh install does not wait a full period. Delays start after each attempt finishes, so slow audits
|
||||
/// cannot cause bursts.
|
||||
struct PartitionAuditSchedule {
|
||||
period: std::time::Duration,
|
||||
next_delay: std::time::Duration,
|
||||
retry_delay: std::time::Duration,
|
||||
has_completed_audit: bool,
|
||||
advance_pending: bool,
|
||||
advance_retries_left: u32,
|
||||
}
|
||||
|
||||
impl PartitionAuditSchedule {
|
||||
fn new(period: std::time::Duration, has_completed_audit: bool) -> Self {
|
||||
Self {
|
||||
/// `startup` is the startup attempt's outcome, as for [`Self::record_attempt`].
|
||||
fn new(period: std::time::Duration, startup: Option<bool>) -> Self {
|
||||
let mut schedule = Self {
|
||||
period,
|
||||
next_delay: if has_completed_audit {
|
||||
period
|
||||
} else {
|
||||
std::time::Duration::ZERO
|
||||
},
|
||||
next_delay: std::time::Duration::ZERO,
|
||||
retry_delay: std::time::Duration::from_secs(5),
|
||||
has_completed_audit,
|
||||
has_completed_audit: false,
|
||||
advance_pending: false,
|
||||
advance_retries_left: PARTITION_ADVANCE_RETRY_LIMIT,
|
||||
};
|
||||
if startup.is_some() {
|
||||
schedule.record_attempt(startup);
|
||||
}
|
||||
schedule
|
||||
}
|
||||
|
||||
async fn wait(&self) {
|
||||
tokio::time::sleep(self.next_delay).await;
|
||||
}
|
||||
|
||||
/// A completed audit counts even when its verdict is unsafe.
|
||||
fn record_attempt(&mut self, completed: bool) {
|
||||
self.has_completed_audit |= completed;
|
||||
if self.has_completed_audit {
|
||||
self.next_delay = self.period;
|
||||
/// Records one attempt: `None` when no audit completed, otherwise whether
|
||||
/// catch-all advancement is still pending. A completed audit counts even
|
||||
/// when its verdict is unsafe. A failed attempt keeps the last known
|
||||
/// pending state.
|
||||
fn record_attempt(&mut self, advance_pending: Option<bool>) {
|
||||
if let Some(pending) = advance_pending {
|
||||
self.has_completed_audit = true;
|
||||
self.advance_pending = pending;
|
||||
}
|
||||
if !self.advance_pending {
|
||||
self.advance_retries_left = PARTITION_ADVANCE_RETRY_LIMIT;
|
||||
}
|
||||
let retry = if !self.has_completed_audit {
|
||||
true
|
||||
} else if self.advance_pending && self.advance_retries_left > 0 {
|
||||
self.advance_retries_left -= 1;
|
||||
true
|
||||
} else {
|
||||
false
|
||||
};
|
||||
if retry {
|
||||
self.next_delay = self.retry_delay;
|
||||
self.retry_delay = self
|
||||
.retry_delay
|
||||
.saturating_mul(2)
|
||||
.min(std::time::Duration::from_secs(60));
|
||||
} else {
|
||||
self.next_delay = self.period;
|
||||
self.retry_delay = std::time::Duration::from_secs(5);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -906,7 +937,9 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
|
||||
)
|
||||
.await;
|
||||
}
|
||||
let has_startup_partition_audit = startup_partition_audit.is_some();
|
||||
let startup_partition_attempt = startup_partition_audit
|
||||
.as_ref()
|
||||
.map(|audit| audit.awaits_catch_all_advance(partition_policy));
|
||||
if let Some(audit) = startup_partition_audit {
|
||||
state.record_partition_audit(audit);
|
||||
}
|
||||
@@ -919,7 +952,7 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
|
||||
let audit_interval = state.config.partition_audit_interval;
|
||||
tokio::spawn(async move {
|
||||
let mut schedule =
|
||||
PartitionAuditSchedule::new(audit_interval, has_startup_partition_audit);
|
||||
PartitionAuditSchedule::new(audit_interval, startup_partition_attempt);
|
||||
loop {
|
||||
schedule.wait().await;
|
||||
let db = &partition_state.db;
|
||||
@@ -941,7 +974,12 @@ async fn run_relay_main(boot: BootTracker) -> anyhow::Result<()> {
|
||||
counted_partition_audit(db.audit_partitions(PARTITION_MANAGER_MONTHS_AHEAD))
|
||||
.await
|
||||
};
|
||||
schedule.record_attempt(result.is_ok());
|
||||
schedule.record_attempt(
|
||||
result
|
||||
.as_ref()
|
||||
.ok()
|
||||
.map(|audit| audit.awaits_catch_all_advance(partition_policy)),
|
||||
);
|
||||
match result {
|
||||
Ok(audit) => partition_state.record_partition_audit(audit),
|
||||
Err(error) => warn!(%error, "Periodic partition catalog audit failed"),
|
||||
@@ -3196,12 +3234,12 @@ mod tests {
|
||||
let period = Duration::from_secs(900);
|
||||
|
||||
let immediate_start = tokio::time::Instant::now();
|
||||
let mut schedule = PartitionAuditSchedule::new(period, false);
|
||||
let mut schedule = PartitionAuditSchedule::new(period, None);
|
||||
schedule.wait().await;
|
||||
assert_eq!(tokio::time::Instant::now(), immediate_start);
|
||||
|
||||
// Failure of the first periodic attempt must not delay recovery by 15 minutes.
|
||||
schedule.record_attempt(false);
|
||||
schedule.record_attempt(None);
|
||||
schedule.wait().await;
|
||||
assert_eq!(
|
||||
tokio::time::Instant::now() - immediate_start,
|
||||
@@ -3209,18 +3247,27 @@ mod tests {
|
||||
);
|
||||
|
||||
let delayed_start = tokio::time::Instant::now();
|
||||
let schedule = PartitionAuditSchedule::new(period, true);
|
||||
let schedule = PartitionAuditSchedule::new(period, Some(false));
|
||||
schedule.wait().await;
|
||||
assert_eq!(tokio::time::Instant::now() - delayed_start, period);
|
||||
|
||||
// A startup pass that left an empty catch-all awaiting advance retries soon.
|
||||
let pending_start = tokio::time::Instant::now();
|
||||
let schedule = PartitionAuditSchedule::new(period, Some(true));
|
||||
schedule.wait().await;
|
||||
assert_eq!(
|
||||
tokio::time::Instant::now() - pending_start,
|
||||
Duration::from_secs(5)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn partition_audit_recovery_backoff_is_bounded_and_completion_ends_retries() {
|
||||
let period = Duration::from_secs(900);
|
||||
let mut schedule = PartitionAuditSchedule::new(period, false);
|
||||
let mut schedule = PartitionAuditSchedule::new(period, None);
|
||||
schedule.wait().await;
|
||||
for seconds in [5, 10, 20, 40, 60, 60, 60] {
|
||||
schedule.record_attempt(false);
|
||||
schedule.record_attempt(None);
|
||||
let start = tokio::time::Instant::now();
|
||||
schedule.wait().await;
|
||||
assert_eq!(
|
||||
@@ -3231,8 +3278,8 @@ mod tests {
|
||||
|
||||
// Completion, not a serving-safe verdict, ends recovery. Later failures
|
||||
// retain the cached audit and use the normal cadence.
|
||||
for completed in [true, false, false] {
|
||||
schedule.record_attempt(completed);
|
||||
for attempt in [Some(false), None, None] {
|
||||
schedule.record_attempt(attempt);
|
||||
let start = tokio::time::Instant::now();
|
||||
schedule.wait().await;
|
||||
assert_eq!(tokio::time::Instant::now() - start, period);
|
||||
@@ -3241,10 +3288,10 @@ mod tests {
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn partition_audit_retries_wait_after_slow_attempts() {
|
||||
let mut schedule = PartitionAuditSchedule::new(Duration::from_secs(900), false);
|
||||
let mut schedule = PartitionAuditSchedule::new(Duration::from_secs(900), None);
|
||||
schedule.wait().await;
|
||||
tokio::time::advance(Duration::from_secs(120)).await;
|
||||
schedule.record_attempt(false);
|
||||
schedule.record_attempt(None);
|
||||
let finished = tokio::time::Instant::now();
|
||||
schedule.wait().await;
|
||||
assert_eq!(
|
||||
@@ -3253,6 +3300,54 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// Waits one scheduled delay and returns how long it was.
|
||||
async fn partition_audit_delay(schedule: &PartitionAuditSchedule) -> Duration {
|
||||
let start = tokio::time::Instant::now();
|
||||
schedule.wait().await;
|
||||
tokio::time::Instant::now() - start
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn partition_advance_retries_are_bounded_and_reset_once_advanced() {
|
||||
let period = Duration::from_secs(900);
|
||||
let mut schedule = PartitionAuditSchedule::new(period, Some(true));
|
||||
let mut delays = vec![partition_audit_delay(&schedule).await.as_secs()];
|
||||
// Failed attempts keep the pending state and spend the same budget.
|
||||
for attempt in [
|
||||
None,
|
||||
Some(true),
|
||||
None,
|
||||
Some(true),
|
||||
Some(true),
|
||||
Some(true),
|
||||
Some(true),
|
||||
] {
|
||||
schedule.record_attempt(attempt);
|
||||
delays.push(partition_audit_delay(&schedule).await.as_secs());
|
||||
}
|
||||
assert_eq!(delays, [5, 10, 20, 40, 60, 60, 60, 60]);
|
||||
assert_eq!(
|
||||
usize::try_from(super::PARTITION_ADVANCE_RETRY_LIMIT).ok(),
|
||||
Some(delays.len())
|
||||
);
|
||||
|
||||
// The budget is spent: a catch-all that never advances falls back to
|
||||
// the normal period instead of retrying forever.
|
||||
for attempt in [Some(true), None, Some(true)] {
|
||||
schedule.record_attempt(attempt);
|
||||
assert_eq!(partition_audit_delay(&schedule).await, period);
|
||||
}
|
||||
|
||||
// Advancing resets the budget and backoff for a later pending catch-all.
|
||||
schedule.record_attempt(Some(false));
|
||||
assert_eq!(partition_audit_delay(&schedule).await, period);
|
||||
schedule.record_attempt(Some(true));
|
||||
assert_eq!(
|
||||
partition_audit_delay(&schedule).await,
|
||||
Duration::from_secs(5)
|
||||
);
|
||||
}
|
||||
|
||||
fn partition_audit_ok() -> buzz_db::Result<buzz_db::partition::PartitionAudit> {
|
||||
Ok(buzz_db::partition::PartitionAudit {
|
||||
audited_at: chrono::Utc::now(),
|
||||
|
||||
@@ -65,8 +65,8 @@ run with a new outcome:
|
||||
- `buzz_partition_create_attempts_total{table, outcome}` counts per-month
|
||||
decisions in each maintenance pass: `created`, `error`, or `skipped_covered`
|
||||
for a month that was already covered. Maintenance runs at startup, and on
|
||||
every periodic audit when `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` is on, so
|
||||
`skipped_covered` then grows on every pass in steady state.
|
||||
every periodic audit when `BUZZ_PARTITION_MANAGER_ADVANCE_ENABLED` is on (the
|
||||
default), so `skipped_covered` then grows on every pass in steady state.
|
||||
|
||||
## Alerts
|
||||
|
||||
|
||||
Reference in New Issue
Block a user