Diagnostic Logs

Sophon uses Serilog for structured logging across every component. Logs land in files by default; sinks can be configured to ship to Seq, Datadog, Elasticsearch, Loki, or any structured-logging system.

Log locations

All files rotate daily with 14-day retention by default (configurable).

Log format

JSON-structured, one entry per line:

{
  "Timestamp": "2026-04-22T10:15:23.847Z",
  "Level": "Error",
  "MessageTemplate": "Failed to invoke tool {ToolName}",
  "Properties": {
    "ToolName": "gmail.send_email",
    "UserId": "usr_abc",
    "SessionId": "sess_xyz",
    "RequestId": "req_abc123",
    "TenantId": "tenant_acme"
  },
  "Exception": "System.HttpRequestException: ..."
}

Every entry includes RequestId — use it to correlate across components.

Log levels

Levels cascade: setting a level enables that level and all below it.

Verbose < Debug < Information < Warning < Error < Fatal

Defaults:

Adjust in appsettings.json:

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Sophon.Core.Orchestration": "Debug",
        "Microsoft.AspNetCore.SignalR": "Debug"
      }
    }
  }
}

Restart the Gateway for level changes to take effect. Or at runtime via:

sophon ops set-log-level --source Sophon.Core.Orchestration --level Debug

(runtime changes don't survive restart; they revert to the config file values on next start).

Debugging specific problems

Orchestration / pipeline issues

Set Sophon.Core.Orchestration to Debug. You'll see:

• Every middleware entry / exit
• Feature-collection writes
• Budget decisions
• Provider routing (which provider / model was picked and why)
• Context-window compaction triggers
• Tool filter selections
• Plan generation + execution
• Tool loop detection hits

LLM provider issues

Set Sophon.Core.Models to Debug:

• Every LLM request (messages, tools, params)
• Response (truncated)
• Provider failover decisions
• API-key rotation events
• Rate-limit headers from upstream

Redaction still applies — message content is hashed by default. To see raw content:

{
  "Sophon": {
    "Logging": { "RedactContent": false }
  }
}

Only do this in dev — prod content can contain PII.

SignalR connection issues

Set Microsoft.AspNetCore.SignalR to Debug. You'll see:

• Connect / disconnect per client
• Group joins / leaves
• Invocation attempts + results
• Transport fallback (WebSocket → SSE → long polling)

Workflow execution

Set Sophon.Core.Workflows to Debug. You'll see:

• Trigger matches
• Node execution order
• Expression evaluation
• Error handler branches
• Sub-workflow invocations

Sandbox / skills

Set Sophon.Sandbox to Debug. You'll see:

• Container lifecycle (create / start / stop)
• Resource limits applied
• stdin / stdout byte counts
• Security violations (if any)

The skill's own logs (stdout/stderr from the skill process) are captured at Information regardless.

Node

Node logs are on the Node machine:

~/.sophon-node/logs/sophon-node-YYYYMMDD.log

Increase verbosity in ~/.sophon-node/appsettings.json:

{
  "Serilog": {
    "MinimumLevel": { "Default": "Debug" }
  }
}

Restart the Node.

MCP

Both client and server:

• Sophon.Mcp.Client — external MCP server calls
• Sophon.Mcp.Server — incoming MCP client requests

Querying logs

Logs are JSON — query with jq:

# All errors in the last log file
jq 'select(.Level == "Error")' gateway-20260422.log

# All tool calls by a specific user
jq 'select(.Properties.UserId == "usr_abc" and .MessageTemplate | test("tool"))' gateway-*.log

# Correlate all logs for a request
jq 'select(.Properties.RequestId == "req_abc123")' gateway-*.log

# Count errors by source
jq -r 'select(.Level == "Error") | .Properties.SourceContext' gateway-*.log | sort | uniq -c | sort -rn

Shipping logs externally

Seq (local Sophon-friendly UI)

{
  "Serilog": {
    "WriteTo": [
      { "Name": "File", "Args": { "path": "logs/gateway-.log", "rollingInterval": "Day" } },
      { "Name": "Seq", "Args": { "serverUrl": "http://seq:5341", "apiKey": "{{vault:seq_key}}" } }
    ]
  }
}

Datadog / Elastic / Loki / Splunk

Serilog has sinks for all major logging platforms. Install the corresponding NuGet package and add it to the WriteTo array.

Syslog / rsyslog

{
  "Serilog": {
    "WriteTo": [
      { "Name": "RemoteSyslog", "Args": { "host": "syslog.example.com", "port": 514, "facility": "local0" } }
    ]
  }
}

Diagnostic bundle

For support tickets, capture a full diagnostic bundle:

sophon ops diagnostic-bundle --out bundle.zip

Bundle includes:

• Last 24 hours of logs (all sources)
• Current appsettings.json (secrets redacted)
• Feature flags
• Recent audit entries (last 1000)
• Process stats (CPU, memory, GC counters)
• Database migration history
• License info
• Connected clients (SignalR)
• Installed skills + versions

Attach to your support ticket. See Support.

OpenTelemetry traces

If OTel is configured (see Operations), every request emits a trace with spans for each middleware, tool call, LLM call, and database query. Open your trace backend (Datadog, Honeycomb, Tempo, Jaeger) and search by request ID:

service.name:sophon-gateway request.id:req_abc123

Much easier than pattern-matching through raw logs when investigating complex issues.

Log retention

External sinks (Seq, Datadog) have their own retention — managed at the sink.

Privacy

• Log content is hashed by default for content / arguments / parameters fields.
• Credentials, API tokens, passwords are never logged regardless of level.
• Tenant / user IDs are logged as-is (they're identifiers, not PII).
• If you enable RedactContent: false in dev, raw user messages land in logs — never do this in prod.

Where to go next

• Common Issues — symptom → fix
• Support — escalation
• Operations — live operational dashboards