Projects

pricescraping-orchestrator Distributed scraping platform

The control plane for a retail price-scraping platform. It schedules scrape jobs onto a Redis queue, spins worker nodes up and down to match the backlog, ingests what the workers send back, and keeps a daily price history for every tracked item.

  • Cron schedules that pair a vendor with a search query
  • Autoscaling workers on SaladCloud or local Docker
  • Idempotent result ingest with a dead-letter queue
  • Price history per listing, with auctions told apart from Buy It Now
  • A dashboard for jobs, nodes, schedules and price trends
Python Flask, SQLAlchemy, APScheduler, Redis Streams, Docker, TimescaleDB View repository