fix(wallet-monitor): post only on state changes, drop reminder re-alerts

Slack posts now fire only when a wallet newly drops below its threshold
or recovers above it; a wallet that stays low never re-posts. The
reminder_hours mechanism is removed entirely — per-wallet reminder
clocks drifted apart (Ogmios blips skip wallets from a run), so with
many chronically-low wallets some clock came due nearly every run,
collapsing the 6h cadence to the hourly cron interval.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
david
2026-07-28 11:12:29 +02:00
parent 0b253c73bb
commit b10706cf69
5 changed files with 28 additions and 45 deletions

View File

@@ -9,10 +9,9 @@ SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXXX/YYYY/ZZZZ
# Defaults to the mainnet public endpoint (also pinned in the compose file).
# OGMIOS_URL=https://ogmios.vector.mainnet.apexfusion.org
# OPTIONAL — override the wallets.json default threshold / reminder cadence
# without editing the mounted wallets file.
# OPTIONAL — override the wallets.json default threshold without editing the
# mounted wallets file.
# DEFAULT_MIN_AP3X=10
# REMINDER_HOURS=6
# OPTIONAL — paths inside the container (defaults shown; bind-mounted by compose).
# WALLETS_PATH=/repo/wallet-monitor/wallets.json

View File

@@ -2,12 +2,12 @@
One-shot wallet-balance Slack alerter for the `vector-marketplace` mainnet box.
Checks every wallet in `wallets.json` against a threshold (default **10 APEX**),
posts a combined Slack message when any wallet is below it, re-reminds every
`reminder_hours` while a wallet stays low, and posts a recovery message when it
climbs back above. Each alert lists **every** monitored wallet — low (🔴) and
healthy (🟢) — with its full, copy-pasteable address; it stays fully silent
when all wallets are healthy. Balances are read from Ogmios via
Checks every wallet in `wallets.json` against a threshold (default **10 APEX**)
and posts a combined Slack message only when something **changed** since the
last run — a wallet newly dropped below its threshold, or climbed back above
it. A wallet that merely stays low never re-posts. Each alert lists **every**
monitored wallet — low (🔴) and healthy (🟢) — with its full, copy-pasteable
address; it stays fully silent while nothing changes. Balances are read from Ogmios via
`ReadOnlyOgmiosProvider` (`@marketplace/shared/chain`) — the same client behind
the buyer's `/v1/wallet/balance` and `buyer/scripts/monitor-wallets.ts`.

View File

@@ -8,12 +8,11 @@
* STATE_PATH — path to the dedup state file (read/written each run).
* Default /repo/wallet-monitor/data/state.json.
* DEFAULT_MIN_AP3X — (optional) overrides wallets.json default_min_ap3x.
* REMINDER_HOURS — (optional) overrides wallets.json reminder_hours.
* TEST — "1" posts a one-line "configured" message to Slack and exits.
*
* wallets.json owns the wallet list + the default threshold / reminder cadence;
* the env overrides exist only so an operator can retune without editing the
* mounted wallets file.
* wallets.json owns the wallet list + the default threshold; the env override
* exists only so an operator can retune without editing the mounted wallets
* file.
*/
const DEFAULT_OGMIOS_URL = "https://ogmios.vector.mainnet.apexfusion.org";
@@ -21,7 +20,6 @@ const DEFAULT_WALLETS_PATH = "/repo/wallet-monitor/wallets.json";
const DEFAULT_STATE_PATH = "/repo/wallet-monitor/data/state.json";
const FALLBACK_DEFAULT_MIN_AP3X = 10;
const FALLBACK_REMINDER_HOURS = 6;
export interface EnvConfig {
slackWebhookUrl: string;
@@ -30,8 +28,6 @@ export interface EnvConfig {
statePath: string;
/** When set, overrides the wallets-file default_min_ap3x. */
defaultMinAp3xOverride: number | null;
/** When set, overrides the wallets-file reminder_hours. */
reminderHoursOverride: number | null;
testMode: boolean;
}
@@ -44,7 +40,6 @@ export interface WalletEntry {
export interface WalletsFile {
defaultMinAp3x: number;
reminderHours: number;
wallets: WalletEntry[];
}
@@ -81,7 +76,6 @@ export function loadConfig(env: Record<string, string | undefined>): EnvConfig {
walletsPath: nonEmpty(env, "WALLETS_PATH", DEFAULT_WALLETS_PATH),
statePath: nonEmpty(env, "STATE_PATH", DEFAULT_STATE_PATH),
defaultMinAp3xOverride: parseOptionalPositiveNumber(env.DEFAULT_MIN_AP3X, "DEFAULT_MIN_AP3X"),
reminderHoursOverride: parseOptionalPositiveNumber(env.REMINDER_HOURS, "REMINDER_HOURS"),
testMode: env.TEST === "1",
};
}
@@ -113,11 +107,6 @@ export function parseWalletsFile(raw: unknown): WalletsFile {
"default_min_ap3x",
FALLBACK_DEFAULT_MIN_AP3X,
);
const reminderHours = parsePositiveNumberField(
raw.reminder_hours,
"reminder_hours",
FALLBACK_REMINDER_HOURS,
);
const walletsRaw = raw.wallets;
if (!Array.isArray(walletsRaw) || walletsRaw.length === 0) {
@@ -152,5 +141,5 @@ export function parseWalletsFile(raw: unknown): WalletsFile {
return entry;
});
return { defaultMinAp3x, reminderHours, wallets };
return { defaultMinAp3x, wallets };
}

View File

@@ -3,9 +3,9 @@
*
* Runs once and exits (invoked hourly by cron via `docker compose run --rm`).
* For each wallet in wallets.json: query its on-chain balance via Ogmios, and
* if it's below threshold post a combined Slack alert. A persisted state file
* dedups so a still-low wallet only re-alerts every `reminder_hours`, and a
* recovery message fires when a wallet climbs back above its threshold.
* post a combined Slack alert ONLY when something changed since the last run —
* a wallet newly dropped below threshold, or climbed back above it. A wallet
* that merely stays low never re-posts (state persisted across runs).
*
* Reuses ReadOnlyOgmiosProvider from @marketplace/shared/chain — the same
* client behind the buyer's /v1/wallet/balance and buyer/scripts/monitor-wallets.ts.
@@ -19,7 +19,6 @@ import { loadConfig, parseWalletsFile, type WalletEntry } from "./config.js";
import { postSlack } from "./slack.js";
const LOVELACE_PER_AP3X = 1_000_000;
const MS_PER_HOUR = 3_600_000;
type Status = "ok" | "low";
@@ -94,7 +93,6 @@ async function main(): Promise<void> {
JSON.parse(readFileSync(cfg.walletsPath, "utf-8")) as unknown,
);
const defaultMinAp3x = cfg.defaultMinAp3xOverride ?? walletsFile.defaultMinAp3x;
const reminderMs = (cfg.reminderHoursOverride ?? walletsFile.reminderHours) * MS_PER_HOUR;
const provider = new ReadOnlyOgmiosProvider({ ogmiosUrl: cfg.ogmiosUrl });
@@ -119,11 +117,11 @@ async function main(): Promise<void> {
const now = Date.now();
const nextState: StateFile = { ...prevState };
// Classify each successfully-queried wallet, advancing dedup state. The
// decision to POST is driven only by transitions (a newly-low wallet, a low
// wallet past its reminder window, or a recovery). But when we DO post, the
// message lists EVERY monitored wallet for full context (built below).
let anyDueLow = false;
// Classify each successfully-queried wallet. The decision to POST is driven
// only by state CHANGES (a wallet newly dropping low, or recovering) — a
// wallet that stays low never re-posts. But when we DO post, the message
// lists EVERY monitored wallet for full context (built below).
let anyNewlyLow = false;
let anyRecovered = false;
const recoveredNames = new Set<string>();
@@ -136,14 +134,12 @@ async function main(): Promise<void> {
const isLow = c.lovelace < c.minLovelace;
if (isLow) {
const wasLow = prev?.status === "low";
const due = !wasLow || now - (prev?.lastAlertMs ?? 0) >= reminderMs;
if (due) {
anyDueLow = true;
nextState[c.name] = { status: "low", lastAlertMs: now };
if (prev?.status === "low") {
// Still low — already alerted when it dropped; stay silent.
nextState[c.name] = prev;
} else {
// Still low but the reminder window hasn't elapsed — keep prev as-is.
nextState[c.name] = prev ?? { status: "low", lastAlertMs: now };
anyNewlyLow = true;
nextState[c.name] = { status: "low", lastAlertMs: now };
}
} else {
if (prev?.status === "low") {
@@ -154,7 +150,7 @@ async function main(): Promise<void> {
}
}
const needPost = anyDueLow || anyRecovered;
const needPost = anyNewlyLow || anyRecovered;
if (needPost) {
const anyLow = checked.some(
@@ -200,12 +196,12 @@ async function main(): Promise<void> {
).length;
console.log(
lowCount > 0
? `[wallet-monitor] ${lowCount} wallet(s) low but already alerted within the reminder window — no Slack post`
? `[wallet-monitor] ${lowCount} wallet(s) still low (already alerted) — no change, no Slack post`
: "[wallet-monitor] all wallets healthy — no Slack post",
);
}
// Persist the new state (records ok/low statuses + advanced reminder stamps).
// Persist the new state (records ok/low statuses + alert stamps).
// Only reached when there was nothing to post or the post succeeded.
saveState(cfg.statePath, nextState);

View File

@@ -1,6 +1,5 @@
{
"default_min_ap3x": 10,
"reminder_hours": 6,
"wallets": [
{ "name": "buyer", "address": "addr1_REPLACE_ME" },
{ "name": "supplier-ollama", "address": "addr1_REPLACE_ME" },