DocumentsNEW

Sophon's document pipeline lets you upload files and then talk to them. PDFs, Word docs, spreadsheets, images, Markdown, HTML, and audio all get normalized into searchable text, chunked, embedded (on Pro/Enterprise), and indexed. Agents can summarize, compare, extract, and answer questions — with retrieval-grounded answers that cite the exact passages they came from. You can also save any URL straight into the library, view every format in the in-app viewer, and keep version history when a document's content changes.

Supported formats

Unsupported formats fail fast with a clear message. Extractors are plugins registered with the document pipeline, so support for additional formats can land without changes to the rest of the stack.

Uploading

From the Dashboard

Drag and drop onto Documents (/documents). You'll see a progress bar per file and a status chip (extracting → chunking → embedding → ready).

From chat

Attach a file to any chat message. The attachment upload goes through the same pipeline; small files are indexed by the time the agent sees your message, larger ones finish processing in the background and become referenceable as soon as they're ready.

From a channel

Send a file on Telegram, WhatsApp, Slack, etc. The channel adapter normalizes it into a SophonMessage attachment, and the file is auto-registered as a document — see Channel attachments.

From the CLI

sophon documents upload report.pdf
sophon documents upload *.pdf --tag q3-review
sophon documents list
sophon documents summarize report.pdf
sophon documents delete <id>

Ingest from a URL

Point Sophon at any URL and it lands in the library:

• Agent tool — document.save_url saves whatever the URL points at. HTML pages go through readability extraction and become clean, readable documents; direct file links (PDF, DOCX, images, …) are saved as-is and processed by the normal pipeline.
• REST — POST /api/documents/ingest-url does the same thing programmatically.

Fetches are SSRF-guarded: requests to private, loopback, and link-local addresses are refused, so an agent can't be tricked into pulling content from your internal network.

Channel attachments

Files sent over WhatsApp, Telegram, Slack, and other channels are automatically registered as documents — no separate upload step. The inbound attachment is stored, tagged with its channel source, and processed like any other upload, so you can ask about a PDF minutes after someone texts it to you.

Auto-ingest is on by default and controlled by Sophon:Documents:IngestChannelAttachments.

The pipeline

Upload → file stored, document row created
  │        (document.uploaded fires here)
  ▼
Small file?  ──yes──▶ processed inline
  │ no
  ▼
Durable queue (EF-backed) → background worker
  │
  ▼
Detect type → route to extractor → extract text + metadata
  │   (scanned-PDF heuristic → Tesseract OCR)
  ▼
Chunk (semantic-boundary-aware, overlap)
  │
  ▼
Embed (Pro/Enterprise)                FTS5 index (all tiers)
  │                                   │
  ▼                                   ▼
Qdrant / pgvector / Milvus    SQLite FTS5 / Postgres FTS
  │                                   │
  └──────────────┬────────────────────┘
                 ▼
      Ready (DocumentProcessedEvent)

Processing is asynchronous and durable. Files at or below InlineProcessingMaxBytes are processed inline in the upload request; larger files hand off to an EF-backed queue worker that survives restarts and retries failures. PDFs whose extracted text looks too thin for their page count are flagged as likely scans and run through Tesseract OCR. Chunks are embedded automatically on upload when AutoEmbed is on (the default). Small documents are ready in seconds; large PDFs with OCR can take minutes. You can start asking questions as soon as status flips to ready.

Storage layout

~/.sophon/documents/
├── uploads/                          # Original uploaded files
│   ├── <doc-id>-<original-name>
│   └── versions/                     # Archived prior revisions
│       └── <doc-id>/
│           └── v<n>-<original-name>
├── processed/                        # Extraction working area
└── index/                            # Document search index

The database tracks metadata (title, size, pages, tags, ownership) and the extracted text. The vector store (if present) holds chunk embeddings with the document ID as metadata. The FTS index is in the same DB (SQLite FTS5 on Personal; Postgres tsvector on Pro/Enterprise).

Grounded Q&A with citations

Ask questions in chat:

"What does the Q3 report say about revenue growth?"

The agent retrieves the most relevant passages and answers from them — every claim carries a [n] marker pointing at the exact passage it came from. Scope the question to specific documents:

"Based only on q3-report.pdf and q4-plan.docx, how does the revenue target change?"

The ask endpoints

Two REST endpoints expose the same grounded Q&A directly:

• POST /api/documents/{id}/ask — ask one document a question
• POST /api/documents/ask — ask across your whole library

Small documents (up to AskContextMaxChars) are answered from their whole text; larger ones go through top-k chunk retrieval first. The model is instructed to cite [n] markers, which are parsed into DocumentAnswer.Citations — each DocumentCitation carries docId, fileName, chunkIndex, and a snippet of the passage it points at, so answers are verifiable back to the source.

Honest no-answers

When retrieval finds nothing relevant, the response comes back with Grounded: false and no model call is made. You get an explicit "I don't have that in your documents" instead of a plausible-sounding guess.

Multi-document analysis

Upload several files. Then:

"Compare the three competitor pitch decks. What's their shared messaging?"

The agent retrieves chunks across all three, synthesizes, and answers. For large batches (10+), use the Batch operation button in the Documents page:

• Summarize all — one summary per document
• Extract — pull structured fields (dates, amounts, names) across the batch
• Merge — combine into a single output document

Version history

Updating a document's content no longer means delete-and-reupload:

• PUT /api/documents/{id}/content replaces the content while keeping the same id — the previous revision is archived, the version number bumps, and extraction/chunking/embedding re-run on the new content. Every reference to the document keeps working.
• History is pruned to the most recent MaxVersionsPerDocument revisions (default 5).
• GET /api/documents/{id}/versions lists archived revisions, and each one can be downloaded individually.

Agents can do the same via the document.update tool.

Document tools

Agent-invocable tools in the document.* namespace:

• document.upload — ingest a file from a path or attachment
• document.save_url — save any URL as a document (SSRF-guarded; HTML becomes a readable doc, files are saved as-is)
• document.search — hybrid keyword + semantic search (keyword only on Personal)
• document.retrieve — verbatim top-k passages with source ids for grounding and citing (~24k char output cap)
• document.ask — synthesized answer with citations, over one document or the whole library
• document.read — fetch a document's extracted text
• document.list / document.info — enumerate the library and inspect a document's metadata
• document.summarize — summarize by style (brief / detailed / executive)
• document.update — version-preserving content update (previous revision archived)
• document.delete — remove a document (Medium risk — gates if part of a plan)

document.retrieve and document.ask split the grounding job: use retrieve when the agent should reason over raw source text and cite it itself, ask when a ready-made, citation-backed answer is enough.

In-app viewer

Every document opens in a rich in-app viewer on the Dashboard — no downloading files to see what's in them:

• PDF — paged viewer with the extracted text side-by-side
• Spreadsheets — sheet and grid rendering
• DOCX — formatted document view
• Images — inline preview
• Audio — player with the transcript
• Markdown — rendered, including Mermaid diagrams
• Code — syntax-highlighted

The same viewer powers document frames on the Canvas, so uploaded and generated files open directly where you're working.

Dashboard

The Documents module has:

• Library — all documents with thumbnails, search, sort, filter by type/tag/date/owner
• Detail view — extracted text side-by-side with original, per-chunk preview, metadata, tags
• Q&A pane — chat-like interface scoped to the open document
• Storage manager — disk usage by type, retention policies, cleanup

Configuration

DocumentsOptions, under the Sophon:Documents:* config section:

Limits and gotchas

• Single-file upload limit: 100 MB (MaxUploadBytes). Larger files return HTTP 413.
• OCR quality depends on the source — low-contrast scans produce noisy extracted text.
• Audio transcription requires an embedding/Whisper-compatible provider configured in Settings → Models.
• Deleted documents are not recoverable. The stored file under uploads/ is removed along with the DB + index entries.
• Vector search is disabled on Personal; keyword search still works but returns fewer conceptually related hits.

Security and isolation

• All documents are user-scoped. Cross-user reads are impossible.
• Tenant-scoped in Enterprise (EF Core global filters).
• Uploaded files are virus-scanned via the configured scanner (ClamAV in the reference deployment).
• Vector metadata is filtered server-side, so semantic search can't return another user's chunks.

Where to go next

• Memory — add document-derived facts to memory
• Workflows — wire a file-change trigger into a document pipeline
• Skills — OCR, Web Scrape, and other extraction skills
• PowerPoint Generation — create and edit .pptx decks that land in the library
• Media Understanding — how inline media attachments become text for the agent