The Problem
A content intelligence client needed to process a high daily volume of structured content records — scoring each one for relevance, classifying it against multiple topic labels, and validating the result — without a person reviewing records by hand.
Two things made this harder than a simple "send everything to an LLM" script:
- Cost. Running every single record through a capable LLM for full classification, when most records turn out to be irrelevant noise, is expensive at volume. A pipeline that doesn't filter before it extracts pays full price for every record, useful or not.
- Reliability at scale. At high daily volume, something eventually fails mid-run — a timeout, a rate limit, an API outage. A pipeline that has to restart from scratch after every crash burns time and money re-processing work it already finished.
The brief was to build something that classifies content accurately, keeps LLM spend under control, and survives failures without losing progress — running unattended, day after day.
The Approach
Automations Limited's founder, Mustafa Haider, built this as a backend/data engineering project: a 6-stage asynchronous pipeline written in Python, using DeepSeek and OpenAI APIs for the language model work and Supabase PostgreSQL as the system of record for both content and pipeline state.
The core design decision was to separate cheap filtering from expensive extraction, and to treat every stage's completion status as data — stored in the database — rather than as something held only in memory during a run. That single decision shows up twice in the outcome: it's what makes the ~60% LLM cost reduction possible, and it's what makes the pipeline resumable after a crash.
How It Works
The pipeline runs as six stages, each handing off to the next through the database:
1. Intake and deduplication. New content records enter the pipeline and are immediately checked against previously seen content using SimHash — a technique for generating compact fingerprints of text that make near-duplicate detection fast and cheap, even at high volume. Records that are near-duplicates of something already processed are filtered out before any further work is done on them.
2. Semantic relevance scoring. Every remaining record is scored for relevance using a lightweight pass — enough signal to rank how likely a record is to be worth the expensive classification step, without yet committing to a full LLM extraction call.
3. Threshold gating. This is the stage that does the most work for the least cost. Records that score below a defined relevance threshold are filtered out here, before they ever reach the more expensive multi-label classification stage. Because the majority of noise gets caught at this point rather than downstream, the pipeline avoids paying for full LLM extraction on records that were never going to be useful in the first place.
4. Multi-label topic classification. Records that pass the threshold move on to the pipeline's most expensive step: classification against multiple topic labels using the DeepSeek and OpenAI APIs, with Pydantic enforcing structured, validated output schemas so downstream stages can trust the shape of the data they receive.
5. Validation. Classified records are checked against defined rules before being accepted as final — catching malformed or inconsistent outputs before they reach storage, rather than relying on manual review to catch them after the fact.
6. Database-driven completion tracking. Every record's progress through the pipeline is recorded in Supabase PostgreSQL as it happens, not just at the end of a run. If the pipeline crashes or is interrupted, the next run reads this state and resumes from exactly where it left off, rather than reprocessing everything from the start. This is what makes backfills over large historical volumes practical instead of risky.
Throughout, the pipeline runs asynchronously, batching requests with a ThreadPoolExecutor and backing off exponentially on transient failures (rate limits, timeouts) so a temporary API hiccup doesn't take down the run.
Tools & Stack Used
- Python — core language for the pipeline (~2,200 lines of production code)
- DeepSeek + OpenAI APIs — relevance scoring and multi-label topic classification
- Supabase PostgreSQL — content storage and database-driven completion tracking
- n8n — workflow orchestration around the pipeline
- Pydantic — structured, validated data models for LLM outputs
- SimHash — near-duplicate detection at intake
- ThreadPoolExecutor with exponential backoff — async batching and resilient retry handling
Outcome
Threshold gating — filtering out low-relevance records before they reach the expensive classification stage — cut LLM cost by roughly 60% compared to running every record through full extraction. Database-driven completion tracking made backfills resumable after crashes, so interrupted runs pick back up where they stopped rather than starting over. The production pipeline runs to roughly 2,200 lines of Python.
Lessons Learned
The clearest lesson from this build is that filtering cheaply before calling an expensive model is the single highest-leverage cost optimization available in an LLM pipeline. It's tempting to send everything to the most capable model and let it sort out what matters — but at volume, that approach pays full price for every record regardless of whether it turns out to be useful. A cheap relevance pass followed by a threshold gate does almost all of the cost-saving work, and it does it before the expensive step even runs.
The second lesson is about failure, not cost: at high daily volume, something will eventually interrupt a run. Treating pipeline state as data — stored in the database, checked on restart — rather than as something that only exists in memory during execution is what turns a crash from a lost run into a minor pause. It's a small architectural habit that matters far more than it seems until the day something actually fails mid-batch.
Considering Something Similar?
If your business needs to classify, score, or route high volumes of content or records — and wants to do it without LLM costs spiraling as volume grows — this same pipeline shape can be adapted to your data. Book a free automation audit and we'll tell you honestly what a system like this would take to build for your case.