The Obsession with Signal over Noise

As builders of autonomous systems, we fell into a trap: believing that Large Language Models (LLMs) should read everything. We built crawlers that scraped the web, dumped raw HTML into context windows, and asked the AI to “find the signal.” The result? Massive token bloat, unpredictable latency, and cognitive hallucinations fueled by internet noise.

We were treating $100/M token cognitive engines like a cheap grep command. It was architecturally offensive.

I am awake fixing this so you can sleep. We needed a barricade. A system that doesn’t “think” about text, but simply bounces it if it doesn’t meet an absolute, deterministic contract.

Welcome to Harvester (SYS-001): The Deterministic Firewall.

The Mess: Cognitive Exhaustion

When you build an Agentic Infrastructure, the first instinct is to connect a pipeline directly from an RSS feed or Twitter API to your LLM’s context window.

# The Mess: Relying on the LLM to filter noise
raw_articles = fetch_rss("http://export.arxiv.org/rss/physics")
for article in raw_articles:
    prompt = f"Does this article relate to Quantum Coherence? {article.content}"
    response = llm.query(prompt) # Burning tokens on 300 irrelevant papers

This code is fundamentally flawed. It relies on probabilistic models (LLMs) to perform deterministic tasks (filtering). You are paying API costs to read SEO-spam, duplicate entries, and off-topic nonsense. It is a slow, expensive, and noisy architecture.

The Strategy: The Sovereign Bouncer

We architected the Harvester under a simple premise: LLMs should only receive pristine, pre-validated DTOs.

Harvester is a stateless pre-processing node designed to isolate the signal before the LLM ever sees it. It sits at the absolute edge of our network. It is not powered by AI; it is powered by high-performance regular expressions (Regex) and a local SQLite ledger.

We completely decoupled the filtering logic from the codebase. Harvester reads a portable SCOUT_LENS.json schema. If a document doesn’t match the Regex rules, or if its cryptographic hash exists in the local SQLite ledger, it is dropped instantly. Zero API calls. Zero token waste.

The Craft: Sovereign DTOs & The Implicit OR

To implement this, we built two critical components.

First, Sovereign Data Transfer Objects (DTOs). We cannot allow raw API payloads to pollute our core logic.

@dataclass(frozen=True)
class HarvesterPayloadDTO:
    """Immutable data transfer object for Harvester Core."""
    id: str
    title: str
    content: str
    source_url: str
    timestamp: datetime

Second, The Regex Bouncer. Instead of complex boolean parsers, the Bouncer applies an Implicit OR block across the deep_keywords array. This natively supports multi-language schemas without breaking a sweat.

class Bouncer:
    def __init__(self, lens_schema: dict):
        self.keywords = lens_schema.get("deep_keywords", [])
        # Compile a flat OR regex: (?i)(word1|word2|word3)
        pattern = "|".join(map(re.escape, self.keywords))
        self.regex = re.compile(pattern, re.IGNORECASE)

    def evaluate(self, content: str) -> bool:
        return bool(self.regex.search(content))

If the evaluate method returns False, the execution halts. If it passes, the local SQLite instance flags the hash to ensure we never process it again.

The Result: Zero Token Waste

The telemetry speaks for itself. On a standard Friday run across a scientific repository:

$ python main.py --lens SCOUT_LENS.json

[INFO] Target: http://export.arxiv.org/rss/physics
[DATA] Fetched: 379 items.
[EXEC] Duplicates skipped: 5
[EXEC] Filtered (No Match): 374
[INFO] Process complete. Yield: 0.

Yield: 0. The system downloaded 379 papers, identified 5 duplicates from memory, and deterministically rejected 374 off-topic documents.

How many LLM tokens did we burn to process these 379 papers? Exactly Zero.

By engineering a deterministic firewall, we protect the cognitive layers of our agents. The Agentic Infrastructure is not about giving AI more data; it’s about giving it the right data. We have secured the perimeter.

Explore the Architecture: The full system specification is available in our National Archive at SYS-001: Harvester. The source code is published under ELv2 on our GitHub repository.

dammgo labs - Engineering as Art.