"""source_registry — corpus -> the tool that already owns it. Wiring, not new ingestion code.

Operator, 2026-07-28: *"it exists for sure: we have already used these tools to feed the corpus
parts from these bookshelves, so it's nothing new to write - just find, wire-in and push a full
chain for a new tool. The complementary watcher will be waiting for events from new man
arrivals, e.t.c. - like n8n manual upload handlers for a documents and chat logs."*

So this module is a LOOKUP over existing tooling. Every fetcher below is a thin adapter onto a
store or script that already populated the corpus; none of it re-implements ingestion.

ONE INGESTION SURFACE, THREE TRIGGERS — not three mechanisms that resemble each other
(operator, 2026-07-28: *"it's not a patterns to mirror - they are the parts of one whole
thing"*). Everything that puts a document into the pipeline arrives through this registry and
leaves through the corpus's own feeder into `common.submit`:

    MANUAL UPLOAD    n8n upload handlers (documents, chat logs) ─┐
    AUTO ARRIVAL     a package is built and emits its manuals   ─┤─> resolve(meta)
    RECOVERY         a stored artifact is damaged               ─┘      -> handler.fetch(pointer)
                                                                        -> the corpus's feeder
                                                                        -> common.submit
Recovery is NOT a special path. It is the same path with two switches thrown:

    dedup-except add <content_hash>   both dedup gates PASS, so the feed is not a silent no-op
    submit() REUSES the existing doc_id  regenerated artifacts land under the SAME storage
                                         prefix and the ArangoDB node keeps its links and
                                         relations — a REPAIR, not a replacement
    dedup-except rm                   re-arm

Which is why `auto_arrival` is a field on every entry below rather than a separate document:
the arrival watcher and the recovery fetcher are reading the same registry about the same
corpus, and a corpus whose arrival is still a manual sweep is a gap in the SAME surface, not a
missing feature of a different one.

── TWO THINGS THE SURVEY ESTABLISHED, both load-bearing ──────────────────────────────────────

1. `metadata.corpus` IS NOT A SUFFICIENT DISPATCH KEY. It is written only by
   feed_docstore_to_pipeline.py (SRC_TYPE=argv[1]), so 81,923 tasks carry no `corpus` at all and
   `provider=web` occurs both with and without it. Hence `resolve()` below walks
   corpus -> provider -> source -> source_origin -> UNKNOWN. Dispatching on `corpus` alone would
   silently miss ~40% of the pipeline.

2. RECOVERY IS USUALLY A SELECT, NOT A RE-HARVEST. For the large corpora the original body is
   still in the docstore, which is the canonical SoT — ~120,800 of ~203k tasks are recoverable
   with one query, no network, BYTE-IDENTICAL. Byte-identical matters more than convenience:
   the same bytes give the same content_hash, so the dedup exception and the doc_id reuse line
   up and the result is a repair rather than a fork. Re-running a harvester is TIER 2, the
   fallback for when the docstore row is gone too.

   (It is also the practical path: only confhelp-harvest.py has a per-item selector
   (`ONLY=<pkg>`); the man harvesters are whole-sweep only — verified by grepping every getenv
   in both — so per-item tier-2 man recovery means replaying their internals.)

── WHAT IS DELIBERATELY NOT RECOVERABLE ──────────────────────────────────────────────────────
`logsmith_bridge` (56 docs) returns None, and that is a finding rather than a gap. Evidenced
three ways: 0 of 15 bridge-fed transcripts still exist on disk; compaction rewrites transcripts
in place, so even a survivor is not the fed bytes; and the DB fallback is materially incomplete
(log_id=7 holds 118 user+assistant rows while its slice 1 alone recorded turns=907 — the hook
captures events, not renderable turns). A replay yields a different content_hash, i.e. a NEW
document, not a repair. Better to say so than to offer a handler that fails quietly.
"""
from __future__ import annotations

import os
import subprocess
from typing import Callable

DOCSTORE_DB = os.getenv("DOCSTORE_DB_NAME", "docstore")


# ── tier 1: the body is still in the docstore (canonical SoT, byte-identical) ────────────────

def _docstore_body(source: str, table: str = "documents") -> str | None:
    """One SELECT. `source` is the docstore_source pointer, e.g. 'man:re2py.1'."""
    sql = (f"SELECT body FROM {table} WHERE source = "
           f"'{source.replace(chr(92), chr(92)*2).replace(chr(39), chr(92) + chr(39))}' LIMIT 1")
    # --raw is LOAD-BEARING. Without it, batch mode escapes newlines/tabs (\n, \t) and the
    # recovered bytes differ from the ingested ones — the content_hash then does not match, and a
    # re-ingest would FORK the document instead of repairing it. The hash match is the proof that
    # this is a repair, so anything that perturbs the bytes defeats the whole mechanism.
    # The client also appends one trailing newline; strip exactly one.
    r = subprocess.run(["mariadb", DOCSTORE_DB, "-N", "-B", "--raw", "-e", sql],
                       capture_output=True, text=True, timeout=30)
    if r.returncode != 0 or not r.stdout:
        return None
    out = r.stdout
    return out[:-1] if out.endswith("\n") else out


def _fetch_man(meta: dict) -> str | None:
    src = meta.get("docstore_source")
    return _docstore_body(src) if src else None


def _fetch_confswitch(meta: dict) -> str | None:
    src = meta.get("docstore_source")
    return _docstore_body(src) if src else None


def _fetch_web(meta: dict) -> str | None:
    """web_documents holds the RENDERED body; the page dump under /var/lib/andrea/web-archive/
    (6,520 files) is tier 2, and the live URL is a last resort — mutable and untrusted, so it
    would not reproduce the content_hash and would fork the document rather than repair it."""
    src = meta.get("docstore_source")
    if src:
        return _docstore_body(src, table="web_documents")
    return None


def _fetch_file(meta: dict) -> str | None:
    """historical_session_log: metadata carries an absolute original_path; 109/109 verified
    present on disk at survey time."""
    p = meta.get("original_path")
    if p and os.path.isfile(p):
        return open(p, encoding="utf-8", errors="replace").read()
    return None


# ── the registry ────────────────────────────────────────────────────────────────────────────
# fetch: callable(meta) -> content | None      tier2: the existing tool to fall back to
# None fetch == not recoverable, and the reason is stated so the checker can report it.

REGISTRY: dict[str, dict] = {
    "man-page": {
        "fetch": _fetch_man,
        "pointer": "docstore_source='man:<name>.<sec>' + package",
        "tier1": "docstore.documents.body (cab 1 / card 1)",
        "tier2": "deb-man-harvest.py / manpage-harvest.py — WHOLE-SWEEP only, no per-item "
                 "selector; per-item means replaying dpkg-deb --fsys-tarfile | man | col -bx",
        "feeds_via": "feed_docstore_to_pipeline.py man-page -> common.submit",
        "auto_arrival": "deb -> openrepo yes; openrepo -> docstore is a MANUAL sweep",
    },
    "configure-switches": {
        "fetch": _fetch_confswitch,
        "pointer": "docstore_source='confhelp:<pkg>' + package",
        "tier1": "docstore.documents.body (cab 1 / card 2)",
        "tier2": "ONLY=<pkg> python3 scripts/confhelp-harvest.py  — the ONE harvester with a "
                 "real per-item selector (confhelp-harvest.py:49,289)",
        "feeds_via": "feed_docstore_to_pipeline.py configure-switches",
        "auto_arrival": "into docstore via the native-build CI emitter; into the pipeline NO",
    },
    "web": {
        "fetch": _fetch_web,
        "pointer": "url / web_doc_guid / archive_key='<sha256>.html'",
        "tier1": "docstore.web_documents (6,523 rows, rendered body)",
        "tier2": "/var/lib/andrea/web-archive/<archive_key> (6,520 dumps present)",
        "feeds_via": "search/corpus_feed.py feed_grind() -> memory-server POST /jobs/dissect",
        "auto_arrival": "YES — every search-worker / site_ripper crawl feeds automatically",
    },
    "historical_session_log": {
        "fetch": _fetch_file,
        "pointer": "original_path (absolute) + file_mtime",
        "tier1": "the host file itself",
        "tier2": None,
        "feeds_via": "scripts/feed-historical-logs.py -> common.submit(source_type=session_log)",
        "auto_arrival": "NO",
    },
    "bulk_backfill": {
        "fetch": None,      # basename-only pointer; needs a resolve step before it can be wired
        "pointer": "original_filename — a BARE BASENAME, not a path",
        "tier1": "/usr/work/infrastructure/logs/ (47 .txt present, count matches 47 docs)",
        "tier2": "scripts/bulk-ingest-logs.py — whole-dir sweep, no per-file selector",
        "feeds_via": "memory-server /jobs/dissect",
        "auto_arrival": "NO",
        "why_none": "the pointer is a basename; a basename->path resolve must be added before "
                    "this can be fetched unambiguously",
    },
    "oed": {
        "fetch": None,
        "pointer": "{'title':'<vol>.pdf'} — no doc-level handle at all",
        "tier1": "/usr/work/corpora/oed/*.pdf (present)",
        "tier2": "re-run OCR for the whole volume (GPU hours)",
        "feeds_via": "dissect -> ocr -> assemble",
        "auto_arrival": "NO",
        "why_none": "metadata carries only a volume filename, so a single damaged doc cannot be "
                    "addressed; oed_pull_ocr_text.py reads stage.ocr_text OUT OF GARAGE — the "
                    "very artifact that may be damaged — and its volume->doc_id map is hardcoded",
    },
    "logsmith_bridge": {
        "fetch": None,
        "pointer": "log_id + slice_seq + msg_id range + tool_session_guuid",
        "tier1": None,
        "tier2": None,
        "feeds_via": "bridge.py::_post_dissect -> memory-server /jobs/dissect",
        "auto_arrival": "YES (PreCompact / cold-close hook)",
        "why_none": "NOT RECOVERABLE. 0 of 15 bridge-fed transcripts still exist; compaction "
                    "rewrites transcripts in place so even a survivor is not the fed bytes; and "
                    "logsmith.session_messages is materially incomplete (log_id=7: 118 "
                    "user+assistant rows vs turns=907 for slice 1 alone). A replay gives a "
                    "different content_hash => a NEW document, not a repair.",
    },
}


def resolve(meta: dict) -> tuple[str | None, dict | None]:
    """(corpus_key, entry). Precedence corpus -> provider -> source -> source_origin.

    NOT `corpus` alone: it is written only by feed_docstore_to_pipeline.py, so 81,923 tasks have
    none, and provider=web appears both with and without it."""
    for field in ("corpus", "provider", "source", "source_origin"):
        v = meta.get(field)
        if v and v in REGISTRY:
            return v, REGISTRY[v]
    return None, None


def fetch(meta: dict) -> tuple[str | None, str]:
    """(content, why). content is None when the corpus is unknown or not recoverable — and `why`
    always says which, because 'no source exists' and 'I could not look one up' are different
    answers and conflating them is how a document gets written off."""
    key, entry = resolve(meta)
    if not entry:
        return None, (f"UNKNOWN corpus — no registry entry for "
                      f"{ {k: meta.get(k) for k in ('corpus','provider','source','source_origin') if meta.get(k)} }")
    fn: Callable | None = entry.get("fetch")
    if fn is None:
        return None, f"{key}: {entry.get('why_none', 'no fetcher wired')}"
    try:
        body = fn(meta)
    except Exception as e:                                # noqa: BLE001
        return None, f"{key}: fetch raised {e!r}"[:200]
    if not body:
        return None, (f"{key}: tier-1 miss ({entry.get('tier1')}) — "
                      f"tier-2 fallback not automated: {entry.get('tier2')}")
    return body, f"{key}: tier-1 hit ({entry.get('tier1')}), {len(body)} bytes"
