buyer: PDF book summarizer (/book) — chunk → map-reduce over suppliers

Adds an authenticated upload flow to the buyer web app: an operator
uploads a PDF, the server extracts + paragraph-aligned-chunks it
(~2.2k tokens), then runs a hierarchical map-reduce summarization by
paying marketplace suppliers per chunk — one on-chain escrow per map
call and per reduce node. One book upload fans out into dozens of real
mainnet escrows (the intended marketplace traffic) while producing a
genuine full-book summary.

Server module (buyer/src/pdf/): PDF extraction (text-only, rejects
scanned), token-aware chunker, supplier discovery/round-robin filtered
to capable models (kimi/deepseek/gpt; excludes qwen2.5:0.5b), pre-spend
cost estimate + wallet-floor guard, and a JobStore orchestrator that
reuses submitPrompt → findSubmittedEscrowRef → runAccept per call.
v1 serializes PostEscrow under a mutex (the shared tx builder can't pin
wallet inputs, so parallel escrows would double-spend); retries failed
calls on a different supplier then continues with gaps. 6 /v1/pdf-*
routes (multipart upload, estimate, confirm-gated start, SSE progress,
result, .md download) behind the existing session auth + page/chunk/
balance caps. SPA adds a 4-step BookSummarizer wizard.

Tests: chunk/estimate/supplier-pool/extract unit tests + orchestrator
(happy/gap/failover) + routes integration. Full buyer suite green
(181 tests); SPA build + typecheck clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
david
2026-06-03 11:37:49 +02:00
parent 2025ab7411
commit c552c4caa3
25 changed files with 2518 additions and 0 deletions

View File

@@ -38,6 +38,29 @@ NETWORK_ID=0
# along over loopback. Production behind TLS should leave this unset / "1".
# COOKIE_SECURE=1
# ─── Optional: PDF Book Summarizer (/book page, /v1/pdf-* routes) ─────────
# Upload a PDF → chunk it → pay suppliers to summarize each chunk (one escrow
# per map + reduce call) → full-book summary. All knobs are optional; defaults
# shown. Tune the caps to bound spend/abuse on mainnet.
#
# Hard caps (a job is refused if it would exceed these):
# PDF_MAX_PAGES=400 # reject books larger than this
# PDF_MAX_CHUNKS=200 # reject books that chunk into more calls
# PDF_MAX_PDF_BYTES=25000000 # multipart upload size limit (bytes)
# Refuse to START a job if projected spend would drop the wallet below this:
# PDF_WALLET_FLOOR_LOVELACE=50000000 # 50 AP3X
# Chunking + reduce shape:
# PDF_CHUNK_TARGET_TOKENS=2200 # target tokens per chunk
# PDF_CHUNK_MAX_TOKENS=3000 # hard ceiling per chunk
# PDF_REDUCE_FANIN=8 # summaries collapsed per reduce node
# PDF_RETRY_K=2 # per-call retries on a different supplier
# Supplier selection (comma-separated model-name substrings):
# PDF_MODEL_ALLOWLIST=kimi,deepseek,gpt
# PDF_MODEL_DENYLIST=qwen2.5:0.5b,qwen2.5-0.5b
# Concurrency. v1 must stay 1 (serial PostEscrow — the shared tx builder
# can't pin wallet inputs yet, so parallel escrows would double-spend):
# PDF_LANE_COUNT=1
# ─── Optional: CIP-33 reference scripts ───────────────────────────────────
# When set, buyer-signed script-spending txs (Accept, Reclaim) read the
# escrow validator from an on-chain reference UTxO instead of inlining