Use different storage strategies for files, indexes, and records

A learning document has an original file, ownership records, and search results. These representations serve different access patterns. Treating a search index as the only source makes content difficult to recover after rebuilding or deletion.

The synthetic library stores originals under object keys, records ownership and versions in a database, and uses an index for candidates. The code creates an in-memory lexical index without uploads, vectors, or a search service.

Separate originals, records, and indexes

Object storage holds keyed binary content such as PDFs or images. Database records hold document ID, object key, owner, version, state, and verification information. Search indexes derive from content and metadata and should not be the only retained copy.

Knowing a key does not establish access. Obtain the object key from an authorized record rather than accept an arbitrary client path. Versions and content digests can identify which original the index processed.

Define updates for original changes, metadata changes, and deletion. Track indexed versions, acceptable lag, and rebuild sources. Otherwise users can find an old title that opens new content or deleted records that remain candidates.

Coordinate uploads and database commits

An object upload and a database commit are not automatically one transaction. A staged workflow can create pending, permit a scoped upload, verify the object and size, mark ready, then index. Failed commits need orphan cleanup; failed uploads need retryable or terminal record states.

Reconciliation should inspect expired pending records, orphan objects, and broken references with a defined time window. Immediate deletion of an object without a ready record can race an active upload. Cleanup needs scope, retries, and auditability.

R2 object reads, writes, deletion, and listing are strongly consistent. IAM permission propagation is a separate boundary. CDN caches can retain old content, and object storage does not make database changes atomic. These are not all an eventual-consistency property of R2 objects.

Treat download links as bearer credentials

A presigned URL permits its holder to perform a specific operation on an object until expiry. It does not itself establish the holder's real identity. Authorize issuance and avoid leaking signed addresses through public logs or analytics.

Expiry limits later use but cannot withdraw already downloaded content. Strict auditing or immediate policy needs may warrant mediated access. CORS governs browser cross-origin interaction rather than ownership and does not protect an otherwise public object.

Scope upload permissions and verify actual size and content. Extensions and client Content-Type alone do not prove format safety. Text extraction and thumbnail generation process untrusted files rather than trusted code.

Choose data models from operations

Relational models support constraints, relationships, and transactions. Document or JSON structures support flexible fields but still need schema and query planning. Graph models represent nodes and relationships and suit traversal requirements; merely having related data does not require a graph database.

Time-series aggregation needs windows, units, and missing-value rules. Prometheus labels identify series, so unbounded user or request IDs can cause cardinality problems. They are not interchangeable with ordinary log fields.

Vectors support similarity candidates; full-text indexes retrieve lexical matches. Both need authorization, evaluation, and version handling. Similarity is not factual correctness. Assess write, recovery, and query requirements together.

Intersect lexical postings offline

An inverted index maps each term to documents containing it. Agent retry intersects two sets and returns D-1. Converting a document's words to a set avoids redundant insertion of the same ID for repeated words.

documents = {"D-1": "agent state retry", "D-2": "image queue retry", "D-3": "agent tools"}
index = {}
for doc_id, text in documents.items():
    for term in set(text.split()):
        index.setdefault(term, set()).add(doc_id)
def search(query):
    groups = [index.get(term, set()) for term in query.split()]
    return sorted(set.intersection(*groups)) if groups else []
assert search("agent retry") == ["D-1"]
assert search("unknown") == []
print(search("agent retry"))

The output is ['D-1']. Unknown terms and an empty query yield no results. The model splits on spaces, is case-sensitive, and implements AND matching only. It has no stemming, Chinese segmentation, relevance ranking, vectors, or permissions.

SQLite FTS5 offers fuller text indexing, but language-specific tokenization still needs testing. Splitting Chinese paragraphs only on spaces generally does not provide useful word search. Confirm candidates against authoritative state, version, and current access.

Cover permissions and deletion across copies

Private index titles, summaries, and terms can disclose content even without a download. Restrict candidates to accessible scope and verify current permission before presentation according to the engine and application requirements.

Deletion should hide records, revoke available access, update indexes, and process objects with retries. Backups, caches, and signed links have separate retention boundaries. A final read check helps during index lag but does not replace cleanup.

Evaluate intended matches, no-answer cases, misleading similarity, and unauthorized samples. Candidate volume alone is not quality, and accurate retrieval does not establish authorization. Use synthetic evaluation data and reproducible rebuilds.

Define term combination too. AND, OR, and phrase queries produce different candidates. An empty query should not automatically expose every private document. Compare the defined rules on the same synthetic samples.

Exercise: recover the document lifecycle

Define pending-to-ready upload states, indexing, version changes, and deletion. Simulate upload success with failed database commit, a record with failed upload, a lagging index, and revoked permissions. Specify recovery sources and cleanup conditions.

Originals and metadata must be recoverable, old indexed versions detectable, and unauthorized titles and downloads absent. Signed links must stay scoped, deletion must cover objects and indexes, and unknown or empty queries must return explicit no results. Actual storage, database, and search clients need independent integration tests; the model proves only its lexical matching rules.