FileFirst: The 4-PIN Extensibility Ecosystem

The core file-first-engine is responsible for archive integrity. It should stay boring.

It does not perform OCR. It does not scrape Discogs. It does not run Whisper. It does not call a vision model. Those jobs belong to external tools.

To enrich the archive without letting every script mutate the core state, FileFirst provides four integration pins. External tools interact through these boundaries. If they fail, the archive does not.

PIN 1: Ingest Inbox API

External scripts must never open _manifest.meta.json with write permissions.

Instead, they write .drop.json files into the inbox or a tracked directory. The Gatekeeper validates the drop against a strict schema, merges it atomically, and deletes the drop file.

Example: a local Whisper process transcribes an audio file. The transcript itself is written as a sidecar file. The .drop.json links that result back to the payload:

{
  "target_expected_filename": "interview_raw.wav",
  "enrichment": {
    "ai_transcript_pointer": "interview_raw.whisper.md",
    "ai_detected_language": "en",
    "ai_confidence_score": 0.98
  }
}

The heavy output stays outside the manifest. The manifest records the relationship.

PIN 2: Read Replica API

External dashboards, static site generators, and graph tools should not parse thousands of JSON manifests directly.

They query the ephemeral SQLite graph cache in read-only mode:

SELECT n.path, d.title, d.description
FROM nodes n
JOIN descriptive d ON n.uuid = d.uuid
WHERE d.tags LIKE '%"bauhaus"%';

SQLite runs in WAL mode, so read-only clients can query without blocking the engine’s single-writer queue.

This is how a web UI, D3 graph, reporting script, or digital garden publisher can get fast answers without becoming part of the archive’s state layer.

PIN 3: Event Bus

Polling a large archive is wasteful.

The engine can broadcast lightweight local events when it ingests a payload, moves a file, updates a manifest, or records a reconciliation event.

Example event:

{
  "event_type": "payload_ingested",
  "timestamp": "2026-02-27T09:43:00Z",
  "path": "/Archive/Audio/Interviews/smith_01.wav",
  "mime_type": "audio/x-wav"
}

A listener script sees the event, checks the MIME type, runs Whisper, and writes the result back through PIN 1.

No polling loop. No cron job walking the filesystem. No script with direct write access to core manifests.

PIN 4: Telemetry and observability

The engine exposes Prometheus-style metrics for operational visibility:

• file_first_reconciliations_total
• file_first_quarantine_events_total
• file_first_sqlite_queue_depth
• file_first_manifest_write_latency_ms

A Grafana dashboard can warn when queue depth spikes or manifest writes slow down. That is much better than discovering a storage issue months later.

Structured quarantine

Some errors need human judgment. FileFirst records those cases as structured, recoverable entries under <vault>/_quarantine.

The current reconciliation model is content-addressed:

• xxh3 hash is authoritative
• expected filename is a disambiguating hint
• mtime is not used as an identity tie-breaker
• content-identical duplicates auto-resolve by default
• under strict_reconciliation, genuinely ambiguous cases become quarantine entries

Inspect and clear entries with:

file-first-engine resolve list
file-first-engine resolve show <id>
file-first-engine resolve pick <id> <index>
file-first-engine resolve dismiss <id>

pick applies a plain filesystem move to the selected canonical location. The Tracker then heals the graph on the next scan.

That makes ambiguity visible and recoverable without turning the Obsidian vault into a pile of manual repair notes.

The complete loop

1. A user drops a new audio file into the archive.
2. The engine detects it, hashes it, writes the manifest, and updates SQLite.
3. The event bus broadcasts payload_ingested.
4. A listener starts Whisper.
5. Whisper writes the transcript as a sidecar and emits a .drop.json file.
6. The Gatekeeper validates and merges the metadata.
7. The Read Replica serves updated state to the Weaver, dashboards, or static publishers.
8. Telemetry records the transaction.

The messy, interesting world of external enrichment stays outside the core engine. The archive remains boring, inspectable, and recoverable.

FileFirst is open source under AGPL-3.0. → GitHub

See also: FileFirst, FileFirst Specification, FileFirst Adoption Guide