Protect the save path when adding AI search

An AI feature made saving less reliable

When I added AI search to an internal application, I initially placed embedding generation and search-index updates inside the document save flow.

The first sequence looked reasonable:

User clicks Save
    -> Write the record to PostgreSQL
    -> Call the embedding service
    -> Update Qdrant
    -> Confirm success

During one test, the embedding service returned a rate-limit error. The implementation at that point rolled back the whole operation. The user could not save the source data because an additional search capability had not completed.

That failure made me reconsider the system boundary. The user was trying to save a document, not asking the system to prove that AI search was ready at the same moment.

The invariant to protect

I defined one invariant for the flow:

Source data is saved when the primary transaction commits. Search data must be rebuildable and must never cause the source record to be lost.

PostgreSQL holds the source. Embeddings and Qdrant records are derived data. If the derived path fails, the system must preserve the document, record that indexing is incomplete and provide an observable recovery path.

This is not only a latency optimization. It defines which dependency is allowed to make the user’s primary work fail.

A new transaction boundary

The team and I split the flow and used a transactional outbox to close the gap between saving data and creating background work.

One database transaction
    -> Save source data
    -> Write outbox event
    -> Commit

Background processing
    -> Read outbox event
    -> Create indexing job
    -> Worker generates embedding
    -> Check current document version
    -> Update Qdrant

If the transaction does not commit, neither the source record nor the outbox event is written. If the data commits but enqueueing temporarily fails, the outbox event remains available for another attempt.

Periodic reconciliation provides another control. It compares source records with indexing state and recreates work when a document is missing or does not match.

PostgreSQL, the queue and Qdrant are named to explain the engineering decision. This is a simplified public model, not the internal architecture.

Retries need idempotency and version checks

Retrying is not simply running the same code again. A job can be delivered more than once, and several edits to one document can finish out of order.

The indexing job uses document_id as its idempotency key. Before writing to Qdrant, the worker loads and checks the current document version. A job handling an older version is not allowed to overwrite a newer result.

When a document changes, the system removes the old search data, generates a new embedding and indexes the new content. When a document is deleted, its corresponding Qdrant data is removed.

Deleting the old index and writing the new one is not atomic. If the worker stops after deletion, the source remains safe but the document can temporarily disappear from search. Retry and reconciliation return the current version to the indexing flow.

This is intentional eventual consistency. I accept a period in which search is behind so that saving does not depend on the AI service.

State model

The operating logic distinguishes states instead of collapsing them into one success signal:

saved
  -> index_pending
  -> indexing
  -> indexed

indexing
  -> retrying
  -> failed
  -> indexed

saved confirms only that the source transaction committed. indexed confirms that the current version is present in the search store. A document in retrying or failed still exists in the source and can re-enter recovery.

The interface should not promise complete success when only saving has finished. Wherever search readiness matters to users, the experience needs to distinguish saved, indexing and not yet searchable.

Failure modes tested in practice

The team tested these scenarios against the implemented flow:

Scenario	Source data	Search state	Expected response
Embedding service is rate limited	Saved	Not updated	Retry with increasing delay
Enqueueing temporarily fails	Saved with outbox event	No job yet	Process the outbox again
Worker stops after deleting the old index	Saved	Temporarily missing	Retry or reconciliation rebuilds the current version
The same job runs more than once	Saved	No duplicate version	document_id keeps the operation idempotent
An older job completes after a newer job	New version saved	Risk of stale data	Worker checks the version and rejects the stale write

These tests show that the failure paths have been executed and observed. They do not prove that the system can never drift, and they are not an exactly-once guarantee.

Recovery and observability

A queue does not solve consistency by itself. It creates a place where unfinished work can be observed and managed. The team needs signals for:

• Time from source commit to current-version search availability.
• Queue depth and retry count.
• Jobs stopped after the retry limit.
• Records reconciliation finds missing or at the wrong version.
• Results of resynchronization attempts.

After a bounded number of retries, failed work remains available for an operator to inspect and recover. The worker does not retry forever because that can hide a persistent fault and continue consuming resources.

What is implemented and what remains limited

The current scope includes:

• A transactional outbox between the save transaction and background work.
• Periodic reconciliation between source data and indexing state.
• An idempotency key based on document_id.
• A current-version check before the worker writes to Qdrant.
• Bounded retries and a retained path for work that cannot complete.
• Removal of old search data and reindexing when a document changes.

The design does not provide strong consistency between PostgreSQL and Qdrant. A newly saved or edited document may not appear in search immediately. That delay needs to be measured and observed if search becomes operationally critical.

What you can apply

When adding AI to an existing flow, start with five questions:

1. Which primary action is the user trying to complete?
2. Which data is the source, and which data can be rebuilt?
3. If saving succeeds but enqueueing fails, where does the system retain evidence?
4. Is retry safe when jobs repeat or finish out of order?
5. Who can see and recover a document that has not been indexed?

If the answer is only “move it to a queue,” the design is not complete. A queue becomes useful when the transaction boundary, idempotency, version check, observability and recovery are defined together.

Publication boundary

This note presents the responsibility model and failure modes tested in practice. It does not disclose schemas, endpoints, credentials, deployment topology, internal data or private operating details. Technology names are included only to make each component’s responsibility clear.