The control plane for a retail price-scraping platform. It schedules scrape jobs onto a Redis queue, spins worker nodes up and down to match the backlog, ingests what the workers send back, and keeps a daily price history for every tracked item.
- Cron schedules that pair a vendor with a search query
- Autoscaling workers on SaladCloud or local Docker
- Idempotent result ingest with a dead-letter queue
- Price history per listing, with auctions told apart from Buy It Now
- A dashboard for jobs, nodes, schedules and price trends