Dashboard

Four idempotent stages move data through one S3 bucket (layers are key roots: raw/ structured/ export/ export-images/ logs/) and Postgres:

1. fetch — crawls each site, saves raw responses to raw/. Runs only on disposable scraper boxes, never from here, so our IPs aren't blacklisted. Their runs still show up under Runs.
2. parse — turns a site's latest raw run into structured/ parquet (pure, no network).
3. merge — normalizes INCI, dedupes, and upserts canonical products into Postgres. A scraped change that would overwrite a manual edit goes to the Merge queue instead.
4. export — writes qualifying products (name + full INCI + ≥1 image) to export/ parquet and their images to export-images/. Use Reuse existing images to skip re-downloading.

Editing a product's INCI fields auto-normalizes them and rebuilds its ingredient rows. Control matching (add/correct/delete overrides, then re-normalize) on the INCI page.