The Problem
An institutional research client needed accurate, up-to-date contact information pulled from 27 institutional websites — the kind of sites that are inconsistent in structure, inconsistent in how they present staff or contact details, and often actively hostile to automated access. Some blocked scrapers outright. Some varied wildly by region. A single fixed scraping script would break constantly, and a manual research process couldn't keep pace with 27 sites that all change on their own schedule.
The real requirement wasn't just "get the data." It was: get the data reliably, at reasonable cost, without a person babysitting the process every time a site changed its defenses — and don't hand back names and emails that look plausible but aren't actually valid.
Automations Limited's founder, Mustafa Haider, was brought on as the AI/data engineer to design and build the system from scratch.
The Approach
Rather than writing one scraper and hoping it held up, he designed the system around a simple idea: not every site needs the same amount of effort to access, and the system should learn which amount of effort works where.
That became a 6-tier waterfall fetcher — a sequence of increasingly capable (and increasingly expensive) ways to retrieve a page, tried in order until one succeeds:
- httpx — a fast, lightweight HTTP client, tried first because most sites don't need anything more.
- curl_cffi — a client that mimics real browser fingerprints, for sites that block obvious bot traffic.
- Proxy rotation — for sites that rate-limit or block by IP.
- Playwright — full browser automation, for sites that require JavaScript rendering to show any content at all.
- BrightData CDP — a managed browser infrastructure layer, for sites with stronger anti-bot defenses.
- Scrappey — a specialized unblocking service, used as the last resort for the most defended sites.
Each tier costs more time and money than the one before it, so the fetcher only escalates when it has to — it never reaches for the expensive option if the cheap one works.
On top of the fetcher sits a validation layer, because a name or email extracted from a page is only useful if it's real: NLP-based name validation and DNS-validated email checking sit between "extracted" and "accepted."
How It Works
The waterfall, and why order matters. Each of the 27 sites has its own defenses, and those defenses aren't static — a site that was easy to reach last month might tighten up next month. Instead of guessing, the system tracks which tier actually succeeded for each region on every run. The next time it runs, it reorders its attempts to try the tier that worked last time first, for that region specifically. Over repeated runs, this means the system converges on the cheapest method that still works for each site — it gets faster and cheaper without anyone manually retuning it.
Recovery strategies for blocked sites. Even with six tiers, some sites still refuse direct access, or only refuse it intermittently. Rather than giving up, the engine falls back to seven recovery strategies, including deep crawling from linked pages, using cached versions of a page, pulling historical snapshots from the Wayback Machine, and reading a site's sitemap directly to find pages a normal crawl would miss. These are last-resort paths for the sites that resist the waterfall entirely.
NLP and LLM name validation. Contact pages return a lot of text that looks like it could be a name but isn't — job titles, department labels, boilerplate. Extracted candidates are checked with Spacy for named-entity recognition, with an LLM layer applied to resolve ambiguous cases, so the output is people's names, not noise that happened to be capitalized correctly.
DNS-validated emails. An email address that's formatted correctly can still be dead. Every extracted email is checked at the DNS level before it's accepted, filtering out addresses that would otherwise pass a simple format check but bounce in the real world.
Atomic budget tracking with a circuit breaker. Because the later tiers cost real money (proxies, managed browsers, unblocking services), the system tracks spend atomically as it runs and trips a circuit breaker before it can run away on a site that's proving unusually expensive to access.
SCD Type 2 audit trail. Contact data changes — people change roles, leave, get replaced. Rather than overwriting old records, the system stores changes using Slowly Changing Dimension Type 2 logic, keeping a full history of what was known and when, instead of only ever showing the current snapshot.
All of this runs behind a FastAPI server, giving the client a clean interface to trigger runs and pull results without touching the underlying pipeline.
Tools & Stack Used
- Python — core language for the entire pipeline
- FastAPI — server layer exposing the extraction engine
- Spacy — NLP-based name validation
- LLM layer — resolving ambiguous name candidates
- PostgreSQL — data storage, including the SCD Type 2 audit trail
- httpx, curl_cffi, Playwright — the lighter-to-heavier fetch tiers
- Proxy rotation, BrightData CDP, Scrappey — escalation tiers for defended sites
- DNS-validated email checking — filtering non-deliverable addresses
- Atomic budget tracking with circuit breaker — cost control across expensive tiers
- Recovery strategies — deep crawl, cache, Wayback Machine, sitemap parsing, and others, for sites that resist direct access
Outcome
The finished system covers all 27 institutional sites through a single pipeline rather than 27 one-off scripts. Because it tracks which tier succeeds per region and reorders its own attempts accordingly, it's a self-improving system — every run makes the next run more efficient, not just a static tool that degrades as sites change their defenses. The result is a production codebase of roughly 9,272 lines of Python handling fetching, recovery, NLP validation, email validation, budget control, and historical auditing in one place.
Lessons Learned
The most useful design decision in this project wasn't any single tier — it was making the tier order learned rather than fixed. A hardcoded "always try Playwright for site X" approach works until the site changes, and then someone has to notice and update the config by hand. Letting the system track outcomes per region and adjust its own ordering means it adapts to a moving target without manual intervention — no one has to notice a site changed its defenses and go update a config file.
The second lesson: validation is not optional overhead, it's half the value. A scraper that returns a name and an email address has extracted data. A scraper that confirms the name is actually a person's name and the email actually resolves has delivered usable contacts. For a client who needs to act on this data — not just look at it — that distinction is the entire point of the system.
If your business depends on contact or data extraction from sites that are inconsistent, defended, or constantly changing, we can help you design a pipeline that's built to adapt rather than break. Book a free consultation to talk through what you need.