The Problem
Monitoring a handful of websites for new content is a weekend script. Monitoring 850+ websites, every day, without gaps, false data, or a slow bleed of engineering hours spent patching broken scrapers, is a different problem entirely.
The client needed a system that could reliably collect structured data from a large and constantly changing set of websites — many of them protected by Cloudflare and other web application firewalls specifically designed to block automated access. On top of that, the data itself needed to come out clean and consistent: correctly parsed dates, deduplicated content, and structured fields — not a pile of raw HTML that a human still had to sort through.
Two failure modes had to be avoided:
- Getting blocked. A scraper that trips anti-bot defenses stops collecting data entirely, often without any clear warning.
- Getting garbage. A scraper that technically "works" but returns malformed dates, duplicate entries, or half-parsed content is often worse than no data at all, because it looks trustworthy while quietly being wrong.
The brief, in short: build something that keeps running, keeps up with sites that change, and keeps the output clean — at a scale where manual babysitting isn't an option.
The Approach
Automations Limited's founder, Mustafa Haider, took on this build as a backend/data engineering project, working directly with Python, Scrapy, Playwright, and Google Cloud Platform.
The core design decision was to treat "fetching a page" and "extracting data from it" as two separate problems, each solved with layered, fallback-driven logic rather than a single fixed method. Anti-bot protection and site layout changes are both moving targets, so a system built around one rigid technique breaks the first time a target site updates anything. A system built around tiers and fallbacks keeps working while individual layers get bypassed, blocked, or updated.
That single principle — never rely on exactly one method when methods can fail — shows up throughout the architecture.
How It Works
A tiered fetch chain. Instead of sending every request straight through a full browser (slow, resource-heavy, and the first thing a WAF looks for), the system tries the cheapest, fastest method first and only escalates when it has to:
- REST — where a site offers a direct API or accessible endpoint, use it. Fastest, cheapest, least likely to be blocked.
- RSS — many sites still expose structured feeds for their own content; when available, this is a clean, low-friction source.
- Scrapy — a standard HTTP-based scrape for sites without an API or feed, handling the majority of straightforward pages efficiently.
- Playwright — a full browser-based fetch, reserved for sites protected by Cloudflare/WAF or that require JavaScript rendering to reveal their content.
Each site is only pushed as far down this chain as it needs to go. A site that works fine on REST never touches the expensive browser-rendering layer — which matters enormously once you're monitoring hundreds of sites daily.
An ensemble extractor. Once a page is fetched, pulling out the actual data (headlines, dates, body content, structured fields) is handled by three independent parsers running in parallel, with a scoring system that selects the best result. This matters because any single parser, however well built, will eventually misread a page — a layout change, an unusual template, or a malformed element. Instead of that failure silently producing bad data, the ensemble compares outputs and picks the one most likely to be correct, so no single point of failure can quietly corrupt the dataset.
A 7-layer date parser. Dates on the open web appear in dozens of inconsistent, sometimes ambiguous formats, across multiple languages. The system works through seven layers of parsing logic in sequence, falling back to an LLM only for the genuinely ambiguous cases a rules-based parser can't resolve — keeping the common cases fast and cheap while still handling the edge cases correctly.
Deduplication at scale. With hundreds of sources publishing overlapping or republished content, duplicate detection has to run fast and cheap at high volume. The system uses SimHash — a technique for generating compact fingerprints of content that make near-duplicate detection efficient even across a large daily volume of pages, without doing expensive full-text comparisons on every pair of documents.
Per-domain strategy caching. Once the system determines which tier of the fetch chain and which extraction strategy works for a given domain, that decision is cached and reused rather than re-evaluated on every run. Sites don't change their anti-bot defenses or layout every day, so re-testing every possible strategy on every visit is wasted compute. Caching the working strategy per domain cuts that repeated overhead significantly.
Tools & Stack Used
- Python — core language for the platform
- Scrapy — primary scraping framework for standard HTTP-based collection
- Playwright — browser-based fetching for JavaScript-heavy or heavily protected sites
- Google Cloud Platform (GCP) — infrastructure and hosting
- Tiered fetch chain (REST → RSS → Scrapy → Playwright) for resilient, cost-aware data collection
- Ensemble extraction with score-based parser selection for reliable structured output
- 7-layer, multi-language date parser with LLM fallback for ambiguous cases
- SimHash deduplication for efficient near-duplicate detection at scale
- Per-domain strategy caching to avoid redundant strategy evaluation
Outcome
The platform now monitors 850+ websites daily, handling Cloudflare and WAF-protected sites as a matter of routine rather than a recurring emergency. Per-domain strategy caching alone reduced compute usage by approximately 30%, by avoiding repeated re-evaluation of fetch and extraction strategies on domains whose behavior hasn't changed.
Lessons Learned
The biggest lesson from this build wasn't a specific library or technique — it was a design philosophy: fallback chains beat single solutions at scale. Any individual scraping method, however sophisticated, will eventually fail on some subset of sites — because of a layout change, a new anti-bot rule, or a format nobody anticipated. Building the system around graceful escalation (try the cheap method, fall back to the expensive one only when needed) rather than a single "best" method turned out to be more robust and more efficient than trying to build one technique to handle every case.
The second lesson was closely related: caching decisions, not just data, is where a lot of the efficiency gain hides. Most of the compute cost in a system like this comes from repeatedly figuring out how to handle a site, not from handling it once you know how. Remembering that answer per domain, instead of re-deriving it on every run, is a small architectural choice with an outsized effect on cost at scale.
Considering Something Similar?
If your business depends on data that lives on other people's websites — competitor pricing, industry listings, regulatory updates, or anything else that changes daily — this same approach can be adapted to your use case. Book a free automation audit and we'll tell you honestly what a system like this would take to build for your situation.