Embeddings & Vector Stores

Semantic recall needs two things: an embedder (one model provider that turns text into vectors, chosen for the whole host) and a vector store (where the vectors live). Memory search, document search and document Q&A use them to find text by meaning, not only by keyword.

Neither is required. Without them, memory and document search run on keywords alone. Since v2.1.0, a failing or slow embedder or vector store no longer fails a chat turn, document search, document Q&A or memory.search: each semantic lookup has an 8-second deadline and then falls back to keyword results. The database is the source of truth, and vectors are a cache that Sophon can always rebuild.

What changed in v2.1.0Copy Anchor Link

• A vector store out of the box. Personal, native and Desktop installs now get an embedded SQLite vector store. Before, they had none unless you ran Qdrant.
• A local embedder first. Sophon looks for a local embedding model (for example nomic-embed-text on Ollama) and prefers it over a cloud provider. It can install nomic-embed-text for you with one click.
• Consent before a cloud embedder. A cloud embedder that has not produced vectors yet sends nothing until an admin consents.
• No more vector wipes. Each embedder writes to its own collections, so switching embedders never mixes or deletes vectors.
• A self-healing index. A background reindex fills in what is missing, for example after an outage, a new embedder, or a restore.

You still need an embedder for semantic recall. Until one is ready, the status shows why (no local model found, waiting for consent, or off) and search uses keywords.

What gets embeddedCopy Anchor Link

Embedding calls count against the provider's budget and appear in Usage under the background source embeddings.

The vector storeCopy Anchor Link

The store is chosen once at startup by Sophon:Vectors:Provider (restart required):

Embedded SQLiteCopy Anchor Link

The SQLite store is one file in WAL mode. Vectors are stored normalized, and each search scans the matching points, up to the newest Sophon:Vectors:Sqlite:MaxScanPoints (default 200,000). Sophon:Vectors:Sqlite:MmapMegabytes (default 256) sizes its memory-mapped I/O. Both need a restart.

QdrantCopy Anchor Link

Pro and Enterprise compose files use Qdrant. Qdrant:Endpoint (Qdrant__Endpoint, outside the Sophon section) is the gRPC URL:

• A URL without a port now means 6334. An explicit :6333 (the REST port) logs a warning.
• A value with no scheme is read as http://. Any scheme other than http or https stops the boot, and so does a URL with a user name and password in it.
• Qdrant:ApiKey (Qdrant__ApiKey) is new. It is sent with every call to a secured Qdrant or Qdrant Cloud and is never logged. Using it over plain http:// to a non-local host logs a warning; use https:// off a private network.

Either store has deadlines: 5 seconds for a search, 10 seconds for a write. After a failed search the store is skipped for a cool-down (30 seconds, doubling up to 10 minutes; 5 minutes after an authentication failure). Writes skipped during that window come back with the next reindex.

The embedderCopy Anchor Link

Choosing the embedderCopy Anchor Link

Sophon:Embeddings:Mode picks how the host embedder is chosen. It applies without a restart.

PreferLocal (default true) ranks local candidates first under auto.

The choice is sticky. Sophon never switches embedders by itself. If a better local embedder appears later, it is offered as a suggestion (Use it in the Dashboard, or sophon models embeddings use <providerId>). The choice changes when you change the mode or the selection settings, when its provider is removed or disabled, or, under local, when it is not local. It is stored in <dataDir>/config/embedder-state.json, which is backed up.

Sophon probes local servers in the background for installed embedding models: every 5 minutes while a local candidate is pending, otherwise every 30 minutes, and at once after a provider is added, changed, enabled, disabled or removed, after a change to an embeddings setting, or after sophon ops action reload-runtime. Embedding models do not need to be listed on the provider or have a context window declared.

Supported embeddersCopy Anchor Link

Any other provider type cannot embed. nomic-embed-text has 768 dimensions and a 2,048-token input window. Inputs are cut to the model's input window (about 3 characters per token), and large batches are split per provider.

Set up local-first embeddingsCopy Anchor Link

ollama pull nomic-embed-text
# or
sophon models embeddings install-local --wait

sophon models embeddings auto     # local first (the default)
sophon models embeddings local    # never use a cloud embedder

sophon models embeddings status

Consent for cloud embeddersCopy Anchor Link

A cloud embedder (Sophon Forge, OpenAI, Gemini, Mistral, Together, NVIDIA NIM, Azure OpenAI, or a custom OpenAI-compatible server on a public address) that has not produced vectors yet embeds nothing until an admin of the default tenant consents. Until then the status reads awaitingConsent, and the admin can see how many memories, daily logs and documents it would receive.

Consent is recorded by:

• the Confirm… banner in Settings → Models → Embeddings ("Save and allow", "Allow and build the index");
• saving that provider in the Embeddings section, or PUT /api/models/embeddings/settings naming it;
• sophon models embeddings use <providerId> --yes or sophon models embeddings consent --yes;
• POST /api/models/embeddings/consent.

A cloud embedder's first full index also needs a confirmed start (--yes, --reindex, startReindex=true, or Reindex…). Consent and warm-up send only fixed probe texts, never your data.

Choose a cloud embedder and build its index in one step (a test embed runs first):

sophon models embeddings use openai-1a2b3c4d5e6f7a8b9 --model text-embedding-3-small --yes

Or, when a cloud embedder is already waiting (status awaitingConsent):

sophon models embeddings consent             # shows the counts, then asks
sophon models embeddings consent --reindex --yes

Local-only chatsCopy Anchor Link

A local-only chat never calls a cloud embedder. While the host embedder is a cloud one, its recall is keyword-only, and documents it stores are flagged so a cloud embedder's background reindex skips them (an admin rebuild still re-embeds them). Its memories are not flagged: they are embedded later by the host embedder. To keep them on the machine, choose a local embedder (sophon models embeddings local).

Collections and switching embeddersCopy Anchor Link

Each embedder writes to its own collections, named after a fingerprint of providerType:model:dimensions:recipe, for example memory__1a2b3c4d and documents__1a2b3c4d. The provider id is not part of it, so re-adding the same provider keeps its vectors. Switching embedders never mixes or wipes vectors, and switching back reuses the earlier collections and only catches up what they missed.

Upgrading from v2.0 on Qdrant. v2.0 wrote the bare memory and documents collections. On first start, Sophon finds the v2.0 embedder whose vector size matches and aliases its fingerprinted names to the bare collections. Nothing is deleted or recreated, and the bare collections stay for a downgrade. Tools that read Qdrant directly must use the fingerprinted names.

Earlier embedders' collections stay until an admin deletes them (Delete on an inactive collection, or DELETE /api/models/embeddings/collections/{name}). The current, pending and recorded embedders' collections cannot be deleted.

Reindex and rebuildCopy Anchor Link

Background reindex. About once a minute (the first check 30 seconds after start), Sophon checks for missing vectors. With a local embedder it builds a new embedder's index, refills a deleted or restored vectors.db, embeds documents processed while embeddings were down, and catches up on memory written during an outage. A cloud embedder's first index never starts by itself. Failures back off from 5 minutes up to 60.

sophon models embeddings reindex --wait --yes   # start (and follow) the backlog
sophon models embeddings reindex --cancel       # stop it

Sophon:Embeddings:AutoReindex=false makes every reindex manual. The existing Sophon:Documents:AutoEmbed=false now also keeps the background reindex away from documents.

Admin rebuild. Drops and recreates the current embedder's two collections and re-embeds every user of every tenant in the background. Only one runs at a time, and its progress is kept in memory, so a restart during a rebuild needs another run.

sophon models embeddings rebuild --yes --log-days 30

Without --yes, rebuild asks in an interactive terminal and fails in a script.

Your own vectors. Re-embed your own memories, daily logs and documents:

sophon memory reindex --vectors --log-days 30

The bare sophon memory reindex still rebuilds the memory graph, as before.

Backup and restoreCopy Anchor Link

vectors/vectors.db and its -wal, -shm and -journal files are a cache. Dashboard backups, sophon backup and sophon snapshot leave them out. sophon restore and sophon snapshot-restore delete them (on Windows, stop the Gateway first), and the background reindex rebuilds the store from the restored data. For an embedder that had built its index before, that rebuild runs by itself at that provider's price. Back up Qdrant separately with Qdrant snapshots.

A damaged vectors.db is fixed by stopping the Gateway, deleting vectors/vectors.db* and starting again.

StatusCopy Anchor Link

GET /api/models/embeddings/status (and sophon models embeddings status, the Dashboard card and sophon health) report what actually works:

Any signed-in user can read the status. Error text, counts and candidates are shown only to an admin of the default tenant. Dashboard terminal members may run sophon models embeddings status, which is on the default allowlist.

CLI referenceCopy Anchor Link

All except status and test need an admin of the default tenant. With --json, stdout carries only the Gateway's JSON; notes and errors go to stderr.

REST APICopy Anchor Link

"Admin" means ManageModels and an admin of the default tenant. Gateways that support these routes advertise embeddings.v1 in GET /api/system/info. GET /api/memory/search now also returns semanticAvailable and semanticError.

AUTH='Authorization: Bearer <admin token>'

# choose a provider; the test embed runs first, and 422 embedding_test_failed saves nothing
curl -X PUT "$GW/api/models/embeddings/settings" -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"mode":"provider","providerId":"ollama-7f3c2a9b1d4e5f6a7","model":"nomic-embed-text"}'

# what a waiting cloud embedder would receive, then consent and start its first index
curl -H "$AUTH" "$GW/api/models/embeddings/consent?logDays=30"
curl -X POST -H "$AUTH" "$GW/api/models/embeddings/consent?startReindex=true"

curl -H "$AUTH" "$GW/api/models/embeddings/status"

Saving settings from the Dashboard, the CLI or PUT /api/models/embeddings/settings writes Sophon:Embeddings to <dataDir>/config/appsettings.user.json, which overrides SOPHON__Embeddings__* environment variables from then on.

ConfigurationCopy Anchor Link

All Sophon:Embeddings keys apply without a restart. Sophon:Vectors and the root Qdrant keys need a restart.

After BreakerFailureThreshold failures in a row, the embedder is skipped for the base cool-down, doubling up to the maximum; an invalid key or missing model skips it for the permanent cool-down. Embedding calls are retried with backoff on 429, 408, 5xx and network errors. A retry count of 0 turns retries off; for the timeout and breaker keys, 0 or less means the default.

{
  "Sophon": {
    "Vectors": { "Provider": "auto" },
    "Embeddings": { "Mode": "local", "PreferLocal": true, "AutoReindex": true }
  },
  "Qdrant": { "Endpoint": "http://qdrant:6334", "ApiKey": "" }
}

As environment variables: SOPHON__Vectors__Provider=sqlite, SOPHON__Embeddings__Mode=local, Qdrant__Endpoint=http://qdrant:6334.

TroubleshootingCopy Anchor Link

Known limitsCopy Anchor Link

• Documents are still chunked at about 2,000 characters. With an embedder whose input window is smaller (512-token models such as mxbai-embed-large or andora-embed), the tail of a full chunk is not embedded.
• Reindex progress counts owners and documents, not individual items.
• The admin rebuild's state lives in memory; a restart during a rebuild leaves half-filled collections and the status reads idle.
• The v2.0 Qdrant adoption is covered by automated tests but has not been run against a live Qdrant.
• On several replicas, a delete reaches only the collections that replica wrote or listed in the last 60 seconds.
• The 274 MB figure for nomic-embed-text is approximate.

RelatedCopy Anchor Link

• Routing, Failover & Budgets — chat models, roles and local-only chats.
• Memory — how memories and daily logs are written and searched.
• Backup & Upgrade — what changes in the data directory on upgrade.