Bulkhead Queues
Isolated BullMQ pools for scrape, ingest, and each publish destination.
Why bulkheads
A bulkhead is a separate queue + concurrency limit. Scrape storms (hundreds of URLs) must not starve publish. Amazon’s slow restore rate must not hold the Shopify GraphQL worker.
Think ship compartments: one flooded hold, the rest of the hull still floats.
Worker pools
Typical split:
- Scrape / URL ingest
- Document ingest
- Publish per destination (Shopify, Amazon, eBay, TikTok, Woo, Medusa, Walmart)
Concurrency is capped per pool. Adding a destination does not steal workers from an existing healthy pool.
Dead-letter queue
Jobs that exhaust retries land in the DLQ. Poison messages (malformed JSON that will never succeed) stay there instead of tight-looping.
Admin → System: **Replay DLQ** requeues; **Purge DLQ** drops them. Read the payload before purge.
Operator actions
If publishes are “stuck,” check:
- Is this channel’s breaker open?
- Is this channel’s queue backing off on 429?
- Is the DLQ full of the same mapper error?
Fix the catalog or credentials, then replay. Do not increase global concurrency as the first lever.