The File-First Architecture: A Specification for Digital Permanence

Prologue: escaping the app trap

The contemporary approach to digital organization relies heavily on proprietary platforms. Tags in one catalogue. Relations in another database. Context in an application-specific format.

When that software disappears, the files may survive, but the organizational intelligence often does not.

FileFirst takes a different approach: metadata and relational context live next to the assets at the filesystem level. Standard POSIX directories become self-contained semantic nodes. The software can help, but the archive must remain readable without it.

1. Core axioms

1. The Sidecar Mandate: Metadata (_manifest.meta.json) resides in the same directory as the assets it describes. Block-level transfers, rsync, and physical drive migrations move the binaries and their context together.

2. State-independent identity: Asset identity is a UUID, not a path, filename, or timestamp.

3. Decentralized graph topology: Relationships form a DAG on the filesystem. Edges point to immutable IDs, not fragile paths.

4. Machine and human parseable: The format is structured for deterministic automation via tools like jq, but readable in plain text.

5. Local-first authority: The local disk is the source of truth. UI endpoints are projections. The engine must not silently move, rename, or physically auto-shard the user’s files.

6. Separation of concerns: Heavy binary storage and UI indexing are physically isolated. A Markdown editor should never be asked to index the primary archive.

2. Directory manifests

The atomic unit of the archive is the directory.

Every tracked directory contains one _manifest.meta.json file at its root. That manifest carries identity, metadata, relations, and payload state for the directory.

Earlier versions described a hard 1,000-payload cap. That is obsolete.

The shard protocol

Large directories are supported through manifest payload sharding.

Once the number of payload entries exceeds the configured shard_limit — default 1,000 — the engine splits the payload section into chunked sidecars:

_manifest.meta.json
_manifest.payload.000.json
_manifest.payload.001.json
_manifest.payload.002.json

The main manifest keeps the descriptive and relational state plus an index of payload shard files. On read, the schema layer transparently merges the shards back into a single in-memory manifest.

This keeps individual JSON files small without restructuring the user’s archive.

The shard threshold is not a directory cap. It is not a halt condition. It does not create hidden user folders. It only changes how FileFirst serializes its own metadata.

3. State tracking

Every tracked binary is represented with multiple signals:

• UUID: the permanent graph anchor
• State hash: an xxh3 content hash used as the authoritative signal for matching moved files
• Expected filename: a hint used to disambiguate byte-identical duplicates
• Mtime: retained for scan performance, not for identity

The current design deliberately avoids using mtime as a tie-breaker for identity. Modification times are routinely rewritten by copy tools, unzip operations, cloud sync, and network shares. They are useful for deciding whether to skip rehashing; they are not a stable statement of what a file is.

4. Required tooling

The daemon layer is written in Go and distributed as a single statically linked binary: file-first-engine.

Core mechanisms:

• Tracker: filesystem changes through hot inotify paths and cold scandir loops
• Ingestor: debounced ingestion from an airlock inbox
• Gatekeeper: strict validation of external .drop.json files
• Reconciliation Engine: content-addressed healing using hash, filename hints, and structured quarantine when needed
• Ephemeral Graph Cache: SQLite in WAL mode, rebuilt from manifests
• Weaver: Markdown shadow vault projection and asymmetric sync
• Mutator: single-writer MPSC channel for atomic file and cache writes
• Scrubber: planned bit-rot protection

5. Process protocols

Directory rename protocol

If a parent directory containing many files is renamed, the engine does not hash every child file.

It reads the directory UUID from _manifest.meta.json, finds the existing node in SQLite, and executes a cascading path update. The UUID, not the path, carries the identity.

Orphan reconciliation and quarantine

When an individual payload appears in a new place, the engine matches by content hash first. If there are multiple byte-identical candidates, the expected filename is used as a hint.

The default mode auto-resolves content-identical duplicates deterministically so reconciliation can keep moving. Under strict_reconciliation, genuinely ambiguous cases are recorded as structured quarantine entries under <vault>/_quarantine.

Resolution is no longer a free-text note or a manual touch ritual. The CLI exposes:

file-first-engine resolve list
file-first-engine resolve show <id>
file-first-engine resolve pick <id> <index>
file-first-engine resolve dismiss <id>

pick applies a plain filesystem move to the selected canonical location; the Tracker then heals the graph on the next scan.

The Shadow Vault and Weaver

The archive and the Markdown vault are distinct paths.

If the archive contains:

/mnt/Archive/Visuals/Linocuts/2026_Series/

The Weaver can project:

~/Obsidian/Visuals/Linocuts/2026_Series.md

The generated Markdown contains YAML frontmatter plus a protected payload table between Weaver markers. User notes below the generated block are preserved. A 5-second debounce lock prevents the daemon from clobbering active typing.

If a directory is explicitly marked with .bypass, the Weaver generates a read-only projection without YAML frontmatter. That marker now means “do not sync editable metadata here,” not “this folder exceeded the old 1,000-file limit.”

6. System safeguards

• 60-second ingest debounce: avoids ghost UUIDs during rapid file operations.
• Write-only AI rule: external tools may generate .drop.json files, but must not edit manifests directly.
• Dry-run mode: shows planned database and filesystem operations before mass changes.
• Signal suspend: SIGUSR1 freezes tracker event processing during large offline operations; SIGUSR2 resumes.
• Local cache locality: SQLite lives on the daemon host’s local disk, not inside the archive and not inside the Markdown vault.

7. The conviction

The filesystem is one of the most durable data structures we have. FileFirst treats it that way.

The engine can fail. The cache can be rebuilt. The Markdown vault is a projection. The archive remains readable because the metadata sits beside the files in plain JSON.

FileFirst is open source under AGPL-3.0. → GitHub

See also: FileFirst, FileFirst Adoption Guide, FileFirst Extensibility