Back to projects

Crawler & PriceWatch: Hyperlocal Price & Delivery Intelligence

An asynchronous crawler and monitor that polls retail and quick-commerce APIs to track price swings, local darkstore inventory, and delivery promise SLAs.

Role
Architect & Developer
Environment
Self-Hosted Homelab (Docker / Python)
Timeline
2026
Stack
Python, AsyncIO, HTTPX, SQLite/PostgreSQL, Docker
Codebase
Private (Active HomeLab infrastructure)

The problem

Modern quick-commerce platforms (JioMart, Blinkit, Instamart) deliver dynamic pricing and fluctuating inventory based on the user's exact street address and darkstore availability. Traditional web scrapers fail because they rely on static page parsing rather than navigating complex session cookies, coordinate-based logistics APIs, and regional token refreshes.

What I built

I engineered PriceWatch as an automated pipeline that emulates native client logistics flows. It resolves geographic coordinates to local fulfillment darkstores, manages authenticated session transports, and polls targeted product SKUs at scheduled intervals. Price histories and stock availability are indexed into local databases to provide trend graphs and instant restock notifications.

Architecture

┌────────────────────────────────────────────────────────┐
│             Scheduled Poller & Cron Dispatcher         │
└───────────────────────────┬────────────────────────────┘
                            │ Async SKU Target Queue
┌───────────────────────────▼────────────────────────────┐
│            Crawler Engine (Python AsyncIO / HTTPX)     │
│  ├── Dynamic Geolocation & Logistics Resolver          │
│  ├── Session Header & Token Emulation Transport        │
│  └── SKU Availability & Price Extractor                │
└───────────────────────────┬────────────────────────────┘
                            │ Normalised Data Pipeline
             ┌──────────────┴──────────────┐
             ▼                             ▼
   Time-Series DB / SQLite        Webhook Alert Triggers

Engineering challenges & trade-offs

Quick-commerce endpoints undergo frequent schema updates and rate limits. By isolating platform scrapers behind modular interfaces and implementing robust retry semantics with exponential backoff and localized user-agent rotations, the crawler runs unattended 24/7 without getting rate-limited or blocked.

Key takeaways

Understanding internal platform APIs beats scraping HTML every time. Emulating actual mobile/web client logistics requests produces reliable, structured JSON payloads in milliseconds with minimal CPU and bandwidth usage.

See other projects