FileFirst

Study

Working question: what happens if the filesystem is treated as the authority for a digital archive?

The thing is an archive architecture where metadata lives next to files in plain JSON. The process favors readable files, stable paths, and long-term repairability.

The current page explains the architecture. The study note still needs the meta layer: what FileFirst taught about permanence, integration, and the cost of hiding meaning inside applications.

FileFirst is an architecture for managing large digital archives — photos, audio, scans, documents, notes, and the context around them — using the filesystem as the source of truth.

Metadata lives next to the files it describes, in plain JSON. The engine can build a fast SQLite cache and a Markdown shadow vault, but the archive itself does not depend on either one.

The project is open source under AGPL-3.0. → GitHub

The core idea

Every tracked directory contains a _manifest.meta.json file. The manifest records the directory identity, descriptive metadata, relationships, tags, provenance, and payload entries for the assets in that directory.

The manifest travels with the files. Copy a folder to an external drive, rsync it to a server, or migrate it to a new machine: the context comes along.

No export step. No proprietary catalogue. No database backup that can drift away from the binaries.

Thirty years from now, cat, grep, and jq should still make sense of the archive.

Architecture

The engine is written in Go and distributed as a single statically linked binary.

Core components:

• Tracker: detects filesystem changes via inotify for hot paths and scandir for cold scans
• Ingestor: handles debounced ingestion from an airlock inbox
• Gatekeeper: validates external .drop.json metadata against a strict schema
• Reconciliation Engine: heals moved or renamed files using content-addressed matching
• Graph Cache: an ephemeral SQLite read replica in WAL mode
• Weaver: projects a Markdown shadow vault for Obsidian or similar tools
• Mutator: a single-writer MPSC queue for atomic filesystem and SQLite operations
• Scrubber: planned bit-rot detection via low-priority SHA-256 verification

The daemon watches the filesystem, maintains the cache, and projects a lightweight Markdown mirror. The data layer — JSON on disk — remains independent of the engine.

Identity and state tracking

Files are identified by UUID, not by path. A file can be renamed, moved, or restored from backup without its relationships depending on the filename that happened to contain it yesterday.

State tracking now uses a stricter hierarchy:

• UUID: permanent graph anchor
• xxh3 hash: authoritative content signal for moved or changed files
• Expected filename: a disambiguating hint when identical content exists in multiple places
• mtime: retained as a performance hint, not as an identity tie-breaker

That last point matters. Earlier versions treated modification time as a fallback identity signal. That was too brittle: copy tools, unzip operations, and sync software rewrite mtimes all the time. FileFirst now avoids using mtime to decide whether two files are the same thing.

The old 1,000-file limit is now a soft shard threshold

Earlier FileFirst writing described a hard 1,000-file directory limit. That is no longer accurate.

The current approach keeps the user’s directory topology intact. If a directory grows beyond the configurable shard_limit — default 1,000 payload entries — the engine shards its own manifest payload into chunked sidecars:

_manifest.meta.json
_manifest.payload.000.json
_manifest.payload.001.json
_manifest.payload.002.json

On read, those payload shards are merged back into a single in-memory manifest. The threshold is a serialization strategy, not a command to move files around.

The important rule remains: the software must not silently reshape the user’s archive. If physical subfolders are needed, that is a human decision, not an automatic daemon trick.

A .bypass file still exists, but its meaning is narrower: it is an explicit “leave this folder’s metadata read-only” marker, not the required escape hatch for directories above 1,000 files.

The Weaver

Tools like Obsidian should not index a heavy binary archive directly. The Weaver projects a lightweight Markdown shadow vault into a separate path.

One archive directory becomes one Markdown proxy. YAML frontmatter can carry editable metadata and relations. The payload table is generated inside <!-- WEAVER_START --> / <!-- WEAVER_END --> markers and treated as read-only projection.

The shadow vault is useful because it gives a human-facing surface without making the editor responsible for the archive itself.

Extensibility

External tools never write to manifests directly. They interact through strict integration boundaries:

• Ingest Inbox: external tools drop .drop.json files, which the engine validates and merges atomically
• Read Replica: dashboards and scripts query SQLite read-only, not the JSON files directly
• Event Bus: automation reacts to lightweight filesystem events instead of polling the archive
• Telemetry: Prometheus metrics expose queue depth, write latency, reconciliation counts, and quarantine events

This keeps the core engine boring. OCR, transcription, AI tagging, Discogs lookups, and similar enrichment can run outside the engine and feed results back through the inbox.

Current state

The schema is at v3.3. Phase 1 (Core State Engine) and Phase 2 (Weaver UI & Reverse Sync) are complete. Phase 3 (Scrubber Daemon for bit-rot detection) is next.

This is a long-term project. It does not need to ship fast. It needs to ship right.

Further reading

• FileFirst Specification: the full architecture: axioms, protocols, engine design
• FileFirst Adoption Guide: who should and should not use this
• FileFirst Extensibility: the 4-PIN integration layer

See also: Processes, Digital Garden