Scraper Architecture

When to use URL ingest

Use the scraper when you have a public (or lab) product URL and you want eListSync to build the canonical product for you. Dashboard: **/import**. API: POST /api/import/url or POST /api/scrape with persist: true when you want rows in Postgres.

URL ingest is the right default for storefronts that emit Schema.org. It is the wrong default for a 10,000-SKU supplier spreadsheet — use the document importer instead.

JSON-LD extraction

The scraper looks for Schema.org / JSON-LD blocks first (Product, Offer, image arrays). That path has the highest fidelity: title, price, currency, availability, and often variants without brittle CSS selectors.

If the page ships JSON-LD, you should see a complete preview before you sync. If price or currency is missing in JSON-LD, the DOM fallback may still fill it from OpenGraph or visible offer nodes.

DOM fallback

When JSON-LD is absent or incomplete, Cheerio (and Puppeteer/stealth when the host blocks naive fetch) reads:

• OpenGraph og:title, og:image, og:price:amount
• Marketplace-shaped trees (Shopify storefronts, Amazon product pages, eBay listings)

DOM coverage is “earth-wide” in intent, not a guarantee that every theme works. Always preview a new supplier domain before a bulk job.

Proxies and geo

Outbound fetches can rotate proxies when configured. Use this for geo-priced storefronts or hosts that rate-limit a single datacenter IP. Failures emit telemetry to the admin SSE stream so you can see timeout vs 403 vs empty body.

Do not point the scraper at a site you are not allowed to copy. Respect robots and supplier contracts.

SSRF and local ingest

By default, private, link-local, and metadata IPs are rejected. That stops an operator from making eListSync fetch http://127.0.0.1:6379/ or cloud instance metadata.

Lab stores on localhost (Medusa on :9000, a fixture storefront) require TOKSYNC_ALLOW_LOCAL_INGEST=1 in the app environment. Leave that off in production.

Extension payloads

The browser extension is not a second scraper architecture. It POSTs **already captured** fields to /api/extension/ingest. SSRF checks still apply to any URLs inside the payload (images, source url).

Use the extension when you are logged into a supplier site that the server-side fetch cannot see (session cookies never leave your browser).

Quality checklist

Before you publish a newly ingested product:

• SKUs unique per variant
• Images are absolute HTTPS URLs
• Currency matches the offer (do not mix USD list price with a EUR JSON-LD block)
• Stock is an integer, not a string like “in stock”
• Title is the product title, not the site name from og:site_name

If preview is empty, see Troubleshooting → DOM Timeouts.