Architecture 8 min read

The Lore Harvester: Mining Distributed Engineering Decisions Across Multi-Node Fleets

How we automated technical retrospection across our entire hardware fleet by engineering a 4-level topological memory crawler in Python.

The Lore Harvester: Mining Distributed Engineering Decisions Across Multi-Node Fleets

The Obsession with Automated Retrospection

TL;DR: We automated technical retrospection across our entire hardware fleet. By engineering a multi-platform Python harvester that navigates a 4-level sovereign topology, calculates deterministic sprint windows, and parses unsterilized session checkpoints across isolated repositories, we transformed decentralized engineering chaos into an automated weekly editorial catalog in 2.3 seconds.

“We are awake fixing this so you can sleep.”

In Part 1 of this series, we killed Friday amnesia at the atomic level. We enforced the Sovereign Checkpoint Protocol, turning session boundaries into mandatory Zero-Trust snapshots that capture technical dilemmas, raw lore, and resumption vectors in hot memory.

The protocol worked. Individual repositories stopped losing context. But our success spawned an immediate architectural crisis: The Scale Trap.

At dammgo labs, we do not operate a single monolithic repository on a single machine. We govern a federated Republic of sovereign domains spanning a multi-node hardware fleet:

  • Primary (MateBook-14 / Windows 11): The architectural authority where release builds are sealed.
  • Laboratory (IdeaPad-530S / Ubuntu Linux): The experimental proving ground for stress tests and native compilation.
  • Mobile (HUAWEI Y9s / Termux Android): The field node for rapid triage and telemetry capture.

Each machine in our fleet maintains a sovereign local replica of 02_Taller continuously synchronized peer-to-peer via Syncthing over an encrypted Tailscale mesh. Crucially, The Lore Harvester performs zero network I/O; it operates exclusively on the local, replicated filesystem. It never stalls on network timeouts or remote socket lag.

When Friday arrived, our engineering lore was intact, but it was shattered across twenty distinct repositories and three physical machines. An engineer wishing to review the week’s accomplishments faced a new form of digital purgatory: manually navigating twenty directory trees, checking Git logs, inspecting .fcg/volatile/checkpoints/ folders, and piecing together a timeline from scratch.

We had cured amnesia, only to surrender to forensic exhaustion.

We stopped ourselves. Manual retrospection in a federated architecture is an anti-pattern. If engineering decisions are captured under strict schemas, extracting them must be an automated data pipeline. We built The Lore Harvester.


The Diagnostics Matrix

FailureRoot CauseFixPrinciple
Fleet BlindnessWork distributed across heterogeneous OS nodes (Windows, Linux, Termux) becomes invisible.Local sovereign replicas synced via Syncthing over Tailscale; zero network I/O in crawler.Topological Agnosticism
Forensic ExhaustionManually traversing dozens of federated directories every Friday to discover active work.Automate traversal via the 4-level Sovereign Dimensional Equation ([node]/[org]/[domain]).Automated Retrospection
Temporal FuzzinessRetrospectives suffering from arbitrary, overlapping, or drifted date boundaries.Mathematically lock the editorial sprint window from Friday 00:00 to Thursday 23:59.Deterministic Time Windows
Cognitive OverloadDumping raw text dumps into chat channels, overwhelming reviewers with unstructured noise.Compile structured Markdown menus pairing executive candidate tables with unsterilized lore.Curated Quarantine

1. The Strategy: The Topological Bridge

To extract knowledge from a federated monorepo without building an invasive database, the crawler must understand the spatial mathematics of the ecosystem.

Under the ATLAS Sovereign Topology Specification, all projects project across The Three Shadows:

  1. The Mind (01_Mapa): The cognitive shadow. Long-term strategy, research, and quarantine workspaces.
  2. The Hands (02_Taller): The physical shadow. Active Git repositories executing on the physical fleet.
  3. The Lake (03_Lago): The heavy shadow. Binary artifacts, design assets, and large datasets.

Every project in the Hands dimension follows a rigid 4-level equation:

Location = 02_Taller / [node_id] / [org_id] / [domain_id]

The Lore Harvester acts as a topological bridge between the Hands and the Mind:

flowchart TD
    subgraph Fleet ["The Physical Fleet (02_Taller)"]
        Node1["Primary (Win11)<br/>dammgo-labs / aoop-spec"]
        Node2["Laboratory (Ubuntu)<br/>dammgo-ops / zte-companion"]
        Node3["Mobile (Termux)<br/>dammgo-forge / project-suma"]
    end

    subgraph Engine ["The Lore Harvester (Python 3)"]
        Scanner["Topological Walker<br/>4-Level Crawler"]
        Window["Sprint Filter<br/>Fri 00:00 - Thu 23:59"]
        Parser["Regex Parser<br/>Dilemma + Lore + Vector"]
        Compiler["Markdown Synthesizer<br/>SUMA 1.1 Compliant"]
    end

    subgraph Mind ["The Cognitive Dimension (01_Mapa)"]
        Quarantine["Volatile Quarantine<br/>workspace / harvested /<br/>Executive Feed & Lore"]
    end

    Fleet --> Scanner
    Scanner --> Window
    Window --> Parser
    Parser --> Compiler
    Compiler --> Quarantine

The Invariable Sprint Window

Retrospectives decay when time boundaries are fuzzy. If an engineer asks “what did we do recently?”, the brain defaults to the last 48 hours.

The Lore Harvester eliminates human bias by enforcing an exact mathematical sprint range based on our Viernes de Reboot publication cadence:

  • Friday Evaluation: Scans the completed cycle from the previous Friday (00:00:00) through yesterday Thursday (23:59:59).
  • Thursday Eve: Pre-flight crawl scanning the current sprint closing tonight.
  • Mid-week: Real-time situational awareness of the sprint in progress.

2. The Craft: Inside harvest_checkpoints.py

The Harvester is implemented as a standalone, multi-platform Python engine running with zero external dependencies. It relies exclusively on Python standard library primitives (pathlib, re, datetime, os).

1. Sovereign Root Resolution

The engine refuses to guess or use brittle relative paths. It strictly verifies the presence of both the Mind (01_Mapa) and the Hands (02_Taller):

def resolve_atlas_root() -> Path:
    """
    Deterministically resolves the ATLAS sovereign root.
    1. Reads ATLAS_ROOT_PATH environment variable.
    2. Executes strict topological ascent from script origin.
    Halts immediately if the boundary invariant is violated.
    """
    env_root = os.environ.get("ATLAS_ROOT_PATH")
    if env_root:
        candidate = Path(env_root).resolve()
        if (candidate / "01_Mapa").is_dir() and (candidate / "02_Taller").is_dir():
            return candidate

    current = Path(__file__).resolve()
    for parent in current.parents:
        if (parent / "01_Mapa").is_dir() and (parent / "02_Taller").is_dir():
            return parent.resolve()

    sys.stderr.write("\n[FATAL] Sovereign Boundary Violation: ATLAS Root not detected.\n")
    sys.exit(1)

2. The 4-Level Topological Crawl & Timestamp Resolution

The engine walks the directory hierarchy, enforcing the 4-level constraint and extracting metadata without mounting the entire repository into memory:

def parse_checkpoint_timestamp(file_path: Path) -> datetime:
    """
    Canonical source is the filename prefix YYYYMMDD to eliminate cross-OS clock drift.
    Frontmatter created_at serves only as secondary metadata.
    """
    match = re.match(r"^(\d{4})(\d{2})(\d{2})_", file_path.name)
    if match:
        return datetime(int(match.group(1)), int(match.group(2)), int(match.group(3)))
    return datetime.fromtimestamp(file_path.stat().st_mtime)

def scan_fleet_checkpoints(taller_root: Path, start_dt: datetime, end_dt: datetime):
    candidates = []
    # Level 1: Fleet Node (Primary, Laboratory, Mobile)
    for node_dir in taller_root.iterdir():
        if not node_dir.is_dir() or node_dir.name.startswith("."):
            continue
        
        # Level 2: Organization (dammgo, atlas, dammgo-forge)
        for org_dir in node_dir.iterdir():
            if not org_dir.is_dir() or org_dir.name.startswith("."):
                continue
            
            # Level 3: Domain (kode-reboot, erpbsg, zte-companion)
            for domain_dir in org_dir.iterdir():
                checkpoint_dir = domain_dir / ".fcg" / "volatile" / "checkpoints"
                if not checkpoint_dir.is_dir():
                    continue
                
                # Level 4: Checkpoint Artifacts
                for cp_file in checkpoint_dir.glob("*.md"):
                    file_date = parse_checkpoint_timestamp(cp_file)
                    if start_dt <= file_date <= end_dt:
                        data = extract_checkpoint_payload(cp_file)
                        candidates.append({
                            "node": node_dir.name,
                            "org": org_dir.name,
                            "domain": domain_dir.name,
                            "payload": data
                        })
    return candidates

Notice the date resolution in parse_checkpoint_timestamp(cp_file). The canonical source of truth for the event timestamp is strictly the immutable filename prefix (YYYYMMDD_[slug].md), not OS filesystem modification times (mtime) or YAML frontmatter strings. By anchoring the date to the filename token itself, we completely eliminate cross-OS clock drift between Windows 11, Ubuntu, and Termux.

3. Extracting the Unsterilized Payload (Zero-Dependency Parser)

To honor our zero-external-dependencies invariant, we deliberately avoid third-party libraries like PyYAML. Instead, we implement an internal 20-line stdlib parser (yaml_light_parse) that parses shallow SUMA frontmatter key-values using pure string manipulation, followed by strict regex boundaries for markdown sections:

# Internal 20-line stdlib helper to preserve zero external dependencies (no PyYAML)
def yaml_light_parse(raw_yaml: str) -> dict:
    data = {}
    for line in raw_yaml.splitlines():
        if ":" in line and not line.strip().startswith("#"):
            k, v = line.split(":", 1)
            data[k.strip()] = v.strip().strip('"\'')
    return data

def extract_checkpoint_payload(file_path: Path) -> dict:
    content = file_path.read_text(encoding="utf-8")
    
    # Extract Frontmatter via stdlib
    fm_match = re.search(r"^---\s*\n(.*?)\n---", content, re.DOTALL)
    metadata = yaml_light_parse(fm_match.group(1)) if fm_match else {}
    
    # Extract The Dilemma, Lore, and Resolution Vector
    dilemma = extract_section(content, r"##\s+(?:El Dilema Técnico|The Dilemma)")
    lore = extract_section(content, r"##\s+(?:El Lore|The Lore)")
    vector = extract_section(content, r"##\s+(?:Vector de Resolución|The Resolution Vector)")
    
    return {
        "title": metadata.get("title", file_path.stem),
        "status": metadata.get("status", "pending"),
        "author": metadata.get("author", "unknown"),
        "dilemma": dilemma,
        "lore": lore,
        "vector": vector,
        "raw_path": str(file_path)
    }

The Triumphant Return: The Weekly Harvest

On Friday morning, executing a single command:

python scripts/harvest_checkpoints.py

Scans over 40 repositories across Windows and Linux nodes in 2.3 seconds.

It does not dump an unreadable terminal log. It generates an immutable, SUMA-compliant report in our volatile quarantine:

01_Mapa/01_Workspace/dammgo/kode-reboot/workspace/harvested/harvest_2026-10-22.md

The Resulting Editorial Feed

The generated document serves an executive candidate menu followed by complete, unsterilized dossiers:

#NodeOrgDomainCheckpoint TitleStatus
1Primarydammgo-labsaoop-specDiagnóstico y Propuesta de Migración .agent a FCGpending
2Primarydammgo-opszte-companionFundación de ZTE-Companion y la Resurrección del Siliciopending
…………(17 additional candidates)…

Behind each entry lies the unsterilized dilemma and raw lore captured at the exact moment of discovery.

When our writing agents (reboot_author) sit down to compose our Friday publications, they do not suffer from the dreaded writer’s block. They do not stare at an empty screen wondering what happened during the week. They open the harvest document, choose the most compelling technical battle, and transform authentic engineering lore into public literature.

If you want to explore our open architectural blueprints, you can read more at kr.dammgo.com.


The Longevity Invariant: Preserving Architecture Beyond Human Turnover

We did not build The Lore Harvester to write blog posts. We built it to guarantee that our architectural trade-offs survive the turnover of human attention.

Complex software systems do not decay because engineers write bad code; they decay because the context behind historical compromises is forgotten. When an edge case is refactored by someone who does not know why it existed in the first place, regressions are inevitable. By extracting, categorizing, and immutably recording our weekly decisions through an automated pipeline, our codebase preserves its own rationale across decades.

We do not write documentation for the feed. We write it to make our technical memory immortal.


Additional Analysis (FAQ)

Why not use a centralized SQLite database to store checkpoints directly?

A centralized database reintroduces the single-point-of-failure and tightly couples independent domains. If a mobile node is disconnected from the network for two weeks, writing to a central database fails. By keeping checkpoints as local Markdown files within each project’s .fcg/volatile/ directory, every domain remains 100% autonomous and offline-first.

How does the harvester handle timezones across heterogeneous nodes?

The canonical source of truth is strictly the filename’s ISO date prefix (YYYYMMDD_[slug].md), not OS filesystem mtime or mutable frontmatter strings. The harvester evaluates dates as discrete topological buckets rather than continuous timestamps, completely immunizing the editorial pipeline against timezone offsets, daylight savings transitions, or cross-OS network clock drift.


dammgo labs - Engineering as Art.

End of Transmission // kode.reboot

© 2026 dammgo labs. Content under CC BY-NC-ND 4.0