mirror of
https://github.com/block/buzz.git
synced 2026-11-10 11:21:05 +00:00
test(relay): give every media e2e upload its own blob hash (#9272)
## Summary Split out of the `flakey-monday` flake-fix series. Two Relay E2E media tests shared one deterministic blob hash across every test in their binaries. In enforce mode (`BUZZ_MEDIA_READ_MODE=enforce`, on for the Relay E2E lane since #9077), a rejected publish of that shared hash left a sticky publication marker that made ranged GETs from the other tests return 404. Every media e2e upload now has its own blob hash. Commits (cherry-picked from `flakey-monday`, original messages kept): - `test(media): give each video e2e test its own MP4 blob` - `test(media): publish images of their own in the media e2e tests` ### Root cause and fix per test **`test_video_range_request_206` / `test_video_range_request_416`** (`crates/buzz-test-client/tests/e2e_media_video.rs`): a ranged GET returned 404 instead of 206/416 in 9 of 55 Relay E2E jobs. - Root cause (confirmed): `build_test_mp4()` was deterministic, so every test in the binary uploaded the same sha256, and the tests run in parallel against one relay. In enforce mode, `validate_media_references` records a sticky `media_publications` marker for every hash an event publishes, before dispatch, so a later rejection does not undo it. `test_video_poster_imeta_rejects_video_as_poster` published the shared hash and was then rejected by imeta validation. From then on `media_access` gave an uploader with no readable reference `Restricted`, served as 404, until the accepted poster test stored its event in an open channel. - Fix: `build_test_mp4()` appends a top-level `free` box holding a random UUID, so each call yields a distinct, still valid fast-start MP4. The upload validator allows `free` and does not inspect its payload. The relay is unchanged: the pre-dispatch marker is the documented fail-closed contract in `docs/operator-private-media.md`. **`test_video_poster_imeta_accepted_via_ws`** and **`e2e_media_extended::test_ws_valid_imeta`**: hardening, not a fix for an observed failure. Both published a fixed 1x1 JPEG byte-identical to `tiny_jpeg()` in `e2e_media.rs`, which `test_upload_and_get` reads back through an uploader preview. That currently passes (a readable live reference wins before the marker matters), but a publishing test that shares a fixed blob with other media tests is one rejected or private event away from the same flake. Both now publish an 8x8 PNG of random pixels from a new `unique_png()` helper; `test_ws_valid_imeta`'s imeta declares `image/png` and the real size; `tiny_jpeg()` is removed from `e2e_media_video.rs`. ### Related issue N/A (flake fixes; none found). ### Testing Human testing is still required. This PR does not carry `buzz-review-completed`; the author has not yet run these tests themselves on this branch. #### Verification These changes were verified locally on the `flakey-monday` branch before the split, against an enforce-mode relay with docker Postgres/MinIO: - Old code, fresh enforce-mode relay: reject test then range tests failed both range tests with 404 at the CI lines (reproduction). - Whole `e2e_media_video` binary, `--test-threads=8`: 20/20 passed. - Reject test then range tests against the polluted hash: 3/3 passed. - The CI "Media read-auth e2e" command (`e2e_media`, `e2e_media_extended`, `e2e_media_video`, `--ignored`), 5 back-to-back runs on the same scratch database (the persistent-relay case): 5/5 runs, 40/40 tests each time. - `cargo fmt --check` and `clippy --tests`: clean. Nothing was re-run on this split branch: the workstation was overloaded, so no tests, builds or linters were executed after cherry-picking. CI on this PR is the verification for the split branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: mtoohey <mtoohey@squareup.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Matt Toohey
@@ -127,6 +127,19 @@ fn tiny_jpeg() -> Vec<u8> {
|
||||
]
|
||||
}
|
||||
|
||||
/// An 8x8 PNG of random pixels, so every call has its own blob hash.
|
||||
/// Publishing a hash leaves a sticky publication marker. Until the hash has a
|
||||
/// readable reference (never, if the event is rejected), the relay refuses
|
||||
/// uploader previews of it. A blob of its own keeps a publishing test from
|
||||
/// reaching the fixed blobs, like `tiny_jpeg`, that other media tests read.
|
||||
fn unique_png() -> Vec<u8> {
|
||||
let mut png = Vec::new();
|
||||
image::RgbImage::from_fn(8, 8, |_, _| image::Rgb(rand::random()))
|
||||
.write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png)
|
||||
.expect("encode PNG");
|
||||
png
|
||||
}
|
||||
|
||||
fn tiny_png() -> Vec<u8> {
|
||||
// Valid 2x2 red PNG generated by ffmpeg, with ffmpeg's pHYs chunk stripped:
|
||||
// `validate_png_metadata_free` rejects pHYs as an identity channel, so the
|
||||
@@ -658,10 +671,10 @@ async fn test_ws_valid_imeta() {
|
||||
let channel_id = channel_uuid.to_string();
|
||||
println!("Channel: {channel_id}");
|
||||
|
||||
// Upload a JPEG to get a valid sha256
|
||||
let jpeg = tiny_jpeg();
|
||||
let sha256 = hex::encode(Sha256::digest(&jpeg));
|
||||
let resp = upload(&http, &keys, &jpeg).await;
|
||||
// Upload an image of its own to get a valid sha256
|
||||
let png = unique_png();
|
||||
let sha256 = hex::encode(Sha256::digest(&png));
|
||||
let resp = upload(&http, &keys, &png).await;
|
||||
assert_eq!(resp.status(), 200);
|
||||
|
||||
// Connect via WebSocket
|
||||
@@ -675,10 +688,10 @@ async fn test_ws_valid_imeta() {
|
||||
Tag::parse(["h", &channel_id]).unwrap(),
|
||||
Tag::parse([
|
||||
"imeta",
|
||||
&format!("url http://localhost:3000/media/{sha256}.jpg"),
|
||||
"m image/jpeg",
|
||||
&format!("url http://localhost:3000/media/{sha256}.png"),
|
||||
"m image/png",
|
||||
&format!("x {sha256}"),
|
||||
"size 347",
|
||||
&format!("size {}", png.len()),
|
||||
])
|
||||
.unwrap(),
|
||||
])
|
||||
|
||||
@@ -79,10 +79,18 @@ fn blossom_auth_header(event: &nostr::Event) -> String {
|
||||
)
|
||||
}
|
||||
|
||||
/// Build a minimal but structurally valid fast-start MP4 (H.264, 1s, 320×240).
|
||||
/// Build a minimal but structurally valid fast-start MP4 (H.264, 1s, 320×240)
|
||||
/// that is unique to this call.
|
||||
///
|
||||
/// Layout: ftyp | moov(mvhd + trak(tkhd + mdia(mdhd + hdlr + minf(vmhd + dinf + stbl)))) | mdat
|
||||
/// Layout: ftyp | moov(mvhd + trak(tkhd + mdia(mdhd + hdlr + minf(vmhd + dinf + stbl)))) | mdat | free
|
||||
/// This is enough for `infer` to detect video/mp4 and for the `mp4` crate to parse.
|
||||
///
|
||||
/// The trailing `free` box carries a random nonce so every test uploads its own
|
||||
/// blob. Tests run in parallel against one relay, and under
|
||||
/// `BUZZ_MEDIA_READ_MODE=enforce` any event that publishes a hash records a
|
||||
/// sticky publication marker, even if the event is later rejected. That marker
|
||||
/// ends uploader-preview reads of the hash, so a shared blob let one test's
|
||||
/// publish turn another test's GET into a 404.
|
||||
fn build_test_mp4() -> Vec<u8> {
|
||||
fn box_wrap(fourcc: &[u8; 4], payload: &[u8]) -> Vec<u8> {
|
||||
let size = (8 + payload.len()) as u32;
|
||||
@@ -263,8 +271,9 @@ fn build_test_mp4() -> Vec<u8> {
|
||||
let moov_payload = [&mvhd[..], &trak].concat();
|
||||
let moov = box_wrap(b"moov", &moov_payload);
|
||||
let mdat = box_wrap(b"mdat", &[]);
|
||||
let nonce = box_wrap(b"free", uuid::Uuid::new_v4().as_bytes());
|
||||
|
||||
[ftyp, moov, mdat].concat()
|
||||
[ftyp, moov, mdat, nonce].concat()
|
||||
}
|
||||
|
||||
/// Upload a valid MP4 video via Blossom, verify the BlobDescriptor includes
|
||||
@@ -481,34 +490,17 @@ fn relay_ws_url() -> String {
|
||||
.replace("https://", "wss://")
|
||||
}
|
||||
|
||||
/// Minimal valid JPEG (1x1 pixel) — used as a poster frame blob.
|
||||
fn tiny_jpeg() -> Vec<u8> {
|
||||
vec![
|
||||
0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00, 0x01, 0x01, 0x00, 0x00,
|
||||
0x01, 0x00, 0x01, 0x00, 0x00, 0xFF, 0xDB, 0x00, 0x43, 0x00, 0x08, 0x06, 0x06, 0x07, 0x06,
|
||||
0x05, 0x08, 0x07, 0x07, 0x07, 0x09, 0x09, 0x08, 0x0A, 0x0C, 0x14, 0x0D, 0x0C, 0x0B, 0x0B,
|
||||
0x0C, 0x19, 0x12, 0x13, 0x0F, 0x14, 0x1D, 0x1A, 0x1F, 0x1E, 0x1D, 0x1A, 0x1C, 0x1C, 0x20,
|
||||
0x24, 0x2E, 0x27, 0x20, 0x22, 0x2C, 0x23, 0x1C, 0x1C, 0x28, 0x37, 0x29, 0x2C, 0x30, 0x31,
|
||||
0x34, 0x34, 0x34, 0x1F, 0x27, 0x39, 0x3D, 0x38, 0x32, 0x3C, 0x2E, 0x33, 0x34, 0x32, 0xFF,
|
||||
0xC0, 0x00, 0x0B, 0x08, 0x00, 0x01, 0x00, 0x01, 0x01, 0x01, 0x11, 0x00, 0xFF, 0xC4, 0x00,
|
||||
0x1F, 0x00, 0x00, 0x01, 0x05, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x00, 0x00, 0x00, 0x00,
|
||||
0x00, 0x00, 0x00, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0A, 0x0B,
|
||||
0xFF, 0xC4, 0x00, 0xB5, 0x10, 0x00, 0x02, 0x01, 0x03, 0x03, 0x02, 0x04, 0x03, 0x05, 0x05,
|
||||
0x04, 0x04, 0x00, 0x00, 0x01, 0x7D, 0x01, 0x02, 0x03, 0x00, 0x04, 0x11, 0x05, 0x12, 0x21,
|
||||
0x31, 0x41, 0x06, 0x13, 0x51, 0x61, 0x07, 0x22, 0x71, 0x14, 0x32, 0x81, 0x91, 0xA1, 0x08,
|
||||
0x23, 0x42, 0xB1, 0xC1, 0x15, 0x52, 0xD1, 0xF0, 0x24, 0x33, 0x62, 0x72, 0x82, 0x09, 0x0A,
|
||||
0x16, 0x17, 0x18, 0x19, 0x1A, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2A, 0x34, 0x35, 0x36, 0x37,
|
||||
0x38, 0x39, 0x3A, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4A, 0x53, 0x54, 0x55, 0x56,
|
||||
0x57, 0x58, 0x59, 0x5A, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68, 0x69, 0x6A, 0x73, 0x74, 0x75,
|
||||
0x76, 0x77, 0x78, 0x79, 0x7A, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8A, 0x92, 0x93,
|
||||
0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9A, 0xA2, 0xA3, 0xA4, 0xA5, 0xA6, 0xA7, 0xA8, 0xA9,
|
||||
0xAA, 0xB2, 0xB3, 0xB4, 0xB5, 0xB6, 0xB7, 0xB8, 0xB9, 0xBA, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6,
|
||||
0xC7, 0xC8, 0xC9, 0xCA, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD7, 0xD8, 0xD9, 0xDA, 0xE1, 0xE2,
|
||||
0xE3, 0xE4, 0xE5, 0xE6, 0xE7, 0xE8, 0xE9, 0xEA, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7,
|
||||
0xF8, 0xF9, 0xFA, 0xFF, 0xDA, 0x00, 0x08, 0x01, 0x01, 0x00, 0x00, 0x3F, 0x00, 0x7B, 0x94,
|
||||
0x11, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
|
||||
0xFF, 0xD9,
|
||||
]
|
||||
/// An 8x8 PNG of random pixels, so every call has its own blob hash.
|
||||
/// Publishing a hash leaves a sticky publication marker. Until the hash has a
|
||||
/// readable reference (never, if the event is rejected), the relay refuses
|
||||
/// uploader previews of it. A blob of its own keeps a publishing test from
|
||||
/// reaching the fixed blobs, like `tiny_jpeg`, that other media tests read.
|
||||
fn unique_png() -> Vec<u8> {
|
||||
let mut png = Vec::new();
|
||||
image::RgbImage::from_fn(8, 8, |_, _| image::Rgb(rand::random()))
|
||||
.write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png)
|
||||
.expect("encode PNG");
|
||||
png
|
||||
}
|
||||
|
||||
/// Upload a video + poster, then send a message with imeta `image` field
|
||||
@@ -561,8 +553,8 @@ async fn test_video_poster_imeta_accepted_via_ws() {
|
||||
let video_desc: serde_json::Value = video_resp.json().await.unwrap();
|
||||
let video_size = video_desc["size"].as_u64().unwrap();
|
||||
|
||||
// 3. Upload poster (tiny JPEG)
|
||||
let poster = tiny_jpeg();
|
||||
// 3. Upload poster (unique PNG)
|
||||
let poster = unique_png();
|
||||
let poster_sha = hex::encode(Sha256::digest(&poster));
|
||||
let poster_auth = sign_blossom_auth(&keys, &poster_sha);
|
||||
let poster_resp = client
|
||||
@@ -593,7 +585,7 @@ async fn test_video_poster_imeta_accepted_via_ws() {
|
||||
"m video/mp4",
|
||||
&format!("x {video_sha}"),
|
||||
&format!("size {video_size}"),
|
||||
&format!("image {base}/media/{poster_sha}.jpg"),
|
||||
&format!("image {base}/media/{poster_sha}.png"),
|
||||
])
|
||||
.unwrap(),
|
||||
])
|
||||
|
||||
Reference in New Issue
Block a user