LLM support - Sure Docs (draft)

LLM usage is optional and must be explicitly configured. You can maintain full control by running local models on your own systems through Ollama, LM Studio, or similar tools. No data is sent to external services unless you configure a cloud provider.

Architecture: two AI pipelines

Sure has two separate AI systems that operate independently. Understanding this matters because they have different configuration requirements. Chat assistant (conversational) — the interactive chat where users ask questions about their finances. It routes through one of two backends:

Builtin (default): uses the LLM provider selected by LLM_PROVIDER (OpenAI-compatible or native Anthropic). Calls Sure’s function tools directly.
External: delegates the entire conversation to a remote AI agent. The agent calls back to Sure via MCP to access financial data. Set ASSISTANT_TYPE=external as a global override, or configure it per family in Settings.
MCP only: disables the in-app assistant entirely — no chat sidebar, no chat API, no bills AI shortcuts, no AI-written insight text. The /mcp endpoint stays active so external agents can still connect. Background auto-categorization also continues. Use this when you bring your own agent over MCP and want to remove the built-in chat UI. Set ASSISTANT_TYPE=mcp_only as a global override, or choose “None in app (connect your own agent via MCP)” in Settings → Self-Hosting.

Auto-categorization (background) — a two-step pipeline that classifies uncategorized transactions on every sync:

Naive Bayes pass — trains a multinomial Bayes model on the family’s existing categorized transactions and classifies new ones locally. No provider or API key is needed. Only transactions that clear a 70% confidence threshold are categorized; the rest fall through to step 2. The Bayes pass requires at least 20 categorized transactions across at least 2 categories before it activates. If Bayes categorizes every transaction in a batch, the LLM step is skipped entirely.
LLM pass — sends the remaining uncategorized transactions to the provider selected by LLM_PROVIDER. This step is skipped when Bayes covers the full batch.

Merchant detection (background) — uses the preferred configured LLM provider (OpenAI-compatible or native Anthropic). Selecting Jev for categorization does not route merchant detection to Jev.

Setting	Chat assistant	Auto-categorization	Merchant detection
`ASSISTANT_TYPE=builtin` (default)	Uses the LLM provider	Bayes first, then LLM provider or Jev	Uses the LLM provider
`ASSISTANT_TYPE=external`	Uses external agent	Bayes first, then LLM provider or Jev	Uses the LLM provider
`ASSISTANT_TYPE=mcp_only`	Disabled (in-app chat off)	Bayes first, then LLM provider or Jev	Uses the configured LLM provider

If you use an external agent for chat, auto-categorization and merchant detection still use the configured LLM provider. Set OPENAI_ACCESS_TOKEN (for the OpenAI-compatible provider) or ANTHROPIC_ACCESS_TOKEN / ANTHROPIC_API_KEY (for the native Anthropic provider), or configure Jev as the categorization provider. Families with at least 20 categorized transactions across 2 or more categories will see Bayes reduce or eliminate LLM calls for categorization.

Where AI jobs run (Sidekiq lanes)

AI work runs in the Sidekiq worker in its own capsule (“lane”), so a slow model can’t hold up syncs, balance recalculation, and scheduled jobs. Both lanes are defined in config/sidekiq.yml:

Lane	Queues	Threads	Jobs
default	`scheduled`, `high_priority`, `medium_priority`, `low_priority`, `default`	`RAILS_MAX_THREADS - RAILS_AI_MAX_THREADS` (3 - 1 = 2 by default)	syncs, balances, rules, imports, scheduled jobs
ai	`ai_high_priority`, then `ai_low_priority` (strict order)	`RAILS_AI_MAX_THREADS` (default `1`)	chat (`ai_high_priority`); auto-categorization, merchant detection and enhancement, PDF processing, insights (`ai_low_priority`)

Key behaviors:

RAILS_MAX_THREADS is the worker’s total thread count and also sizes its database pool. It must be greater than RAILS_AI_MAX_THREADS (at least 2 by default). A lane with 0 threads stops the worker from starting. Puma reads the same variable, so if you lower it for the web service to save memory, set the worker’s value separately.
Strict order: a waiting chat runs next, but it doesn’t interrupt the AI job already running. Auto-categorization, merchant-detection, and merchant-enhancement work is batched using LLM_MAX_ITEMS_PER_CALL (default 25). An OpenAI-compatible provider can split a batch into several calls when it does not fit the context window. PDF import, insight, and chat jobs can make several calls. With one AI thread, a PDF import or other ai_low_priority job waits behind earlier ones, and continuous chat use pauses background AI work.
Busy notice: when a chat would wait behind another AI job (a chat has waited over 30 seconds, or, with one AI thread, a background AI job has waited over 30 seconds), its “Thinking” bubble says the system is busy until the chat’s job starts. AI_RESPONSE_TIMEOUT includes that wait.
One AI thread suits a model that serves one request at a time, such as a local model on CPU. A chat turn’s calls run back to back, keeping the model’s prompt cache warm. If your provider serves requests in parallel (hosted APIs or a local server with several slots), raise RAILS_AI_MAX_THREADS and RAILS_MAX_THREADS by the same amount so the default lane keeps its threads.
Load config/sidekiq.yml to use the configured AI capsule. bundle exec sidekiq discovers this file by default. A -q option overrides the default capsule’s queues but does not remove the AI capsule from a loaded configuration. If a custom worker command loads a different configuration without the AI capsule, ensure that some worker polls ai_high_priority and ai_low_priority. Otherwise System health and the site-wide banner report “No worker is processing the AI queues”, and the chat page warns that responses may not be delivered. Use -C config/sidekiq.yml to select Sure’s configuration explicitly.
System health: AI backlog doesn’t count towards “queue backed up” (a large merchant-detection run can queue for hours), but “Oldest job age” still includes it. Check the queue table or /sidekiq to see which queue is waiting.
Large batches: Sure doesn’t limit LLM_MAX_ITEMS_PER_CALL. OpenAI-compatible providers split a batch that doesn’t fit the context window. With the native Anthropic provider, raise ANTHROPIC_MAX_TOKENS (default 4096) along with LLM_MAX_ITEMS_PER_CALL, or replies are cut off.
Upgrading: AI jobs moved from high_priority (chat), medium_priority, and scheduled to the AI queues. Jobs already queued under the old names still run once, in the default lane.

Quickstart

The fastest way to enable AI features is to use OpenAI:

Get an API key from OpenAI.
Set the environment variable: OPENAI_ACCESS_TOKEN=sk-proj-...your-key-here...
Restart Sure (both web and worker services). The AI assistant becomes available once you enable it in the UI.

Sure uses gpt-4.1 as the default model.

Supported providers

Sure supports two first-class LLM providers:

OpenAI (or any OpenAI-compatible endpoint, including Ollama, LM Studio, OpenRouter, Groq, Together AI, etc.)
Anthropic (native Claude support, including custom Anthropic-compatible endpoints such as Amazon Bedrock or Google Vertex)

Set the active provider with the LLM_PROVIDER environment variable (or the LLM Provider setting in the UI). It defaults to openai. The matching access token must be configured for the selected provider. When set, LLM_PROVIDER takes precedence over any provider choice saved in the UI.

`LLM_PROVIDER`	Required credential
`openai` (default)	`OPENAI_ACCESS_TOKEN`
`anthropic`	`ANTHROPIC_ACCESS_TOKEN` (or `ANTHROPIC_API_KEY`)

Chat routing honors LLM_PROVIDER first: the provider selected by that setting is tried before any fallback. If the selected provider does not support the model in use, the other configured provider is tried as a fallback. This means setting LLM_PROVIDER=anthropic reliably routes chat to Anthropic even when a custom OpenAI-compatible endpoint (OPENAI_URI_BASE) is also configured.

Configuration via settings

Configure AI settings through the web interface:

Go to Settings → Self-Hosting
Scroll to the AI Provider section

The form contains the following fields, grouped by function: Provider & credentials

LLM Provider — openai or anthropic
Access Token — API key for the selected provider
URI Base — Custom endpoint URL (OpenAI-compatible providers only; leave blank for OpenAI)
Model — Model name (required for custom endpoints)

Behavior

JSON Mode — Output mode for structured responses: auto, strict, json_object, or none
Chat Response Timeout — Whole-turn watchdog in seconds (see Chat response timeout)

Token budget

Context Window — Total tokens the model accepts
Max Response Tokens — Tokens reserved for the model’s reply
Max Items Per Batch — Upper bound for auto-categorize and merchant detection batches

Choose your LLM Provider (openai or anthropic)
Configure the matching credentials and options:

OpenAI / OpenAI-compatible provider fields:

Field	Description
Access Token	API key for the provider (maps to `OPENAI_ACCESS_TOKEN`)
URI Base	Custom endpoint URL — leave blank for OpenAI (maps to `OPENAI_URI_BASE`)
Model	Model name — required for custom endpoints (maps to `OPENAI_MODEL`)
JSON Mode	JSON output mode for OpenAI-compatible categorization and merchant-detection calls, plus text-based PDF and bank-statement extraction: Auto, Strict, JSON Object, or None (maps to `LLM_JSON_MODE`)
Context Window	Total tokens the model accepts (maps to `LLM_CONTEXT_WINDOW`, default `2048`)
Max Response Tokens	Tokens reserved for the model’s reply (maps to `LLM_MAX_RESPONSE_TOKENS`, default `512`)
Max Items Per Batch	Upper bound for auto-categorize / merchant detection batches (maps to `LLM_MAX_ITEMS_PER_CALL`, default `25`)
Chat Response Timeout	Whole-turn watchdog in seconds (maps to `AI_RESPONSE_TIMEOUT`, default `90`)

Anthropic provider fields:

Field	Description
Access Token	API key for Anthropic (maps to `ANTHROPIC_ACCESS_TOKEN`)
Base URL	Custom endpoint URL for Bedrock / Vertex proxies (maps to `ANTHROPIC_BASE_URL`)
Model	Model name — required when Base URL is set (maps to `ANTHROPIC_MODEL`, default `claude-sonnet-4-6`)

Environment variables take precedence over UI settings when set to a non-blank value. A blank environment variable (e.g. OPENAI_MODEL=) is treated as unset and the UI setting is used instead. If an environment variable is set to a non-blank value, the corresponding field is disabled in the UI.

Choosing a provider

Sure ships with two first-class LLM providers:

OpenAI (and OpenAI-compatible) — default. Works with OpenAI directly, Ollama, LM Studio, OpenRouter, Groq, and any other OpenAI-compatible endpoint.
Anthropic — native Claude support via the Messages API, including auto-categorization, merchant detection, and provider-merchant enhancement.

Select the active provider with the LLM_PROVIDER environment variable:

Variable	Description	Default
`LLM_PROVIDER`	Active provider. Set to `openai` or `anthropic`. When set, this value takes precedence over any provider choice saved in the UI.	`openai`

Chat tries the provider selected by LLM_PROVIDER first when it supports the chat model, then tries the other configured provider. A custom OpenAI-compatible endpoint does not automatically override a compatible Anthropic selection. Check the chat model and provider credentials if a request uses the fallback provider.

OpenAI compatible API

Sure supports any OpenAI-compatible API endpoint, giving you flexibility to use:

OpenAI - Direct access to GPT models
Ollama - Run models locally on your hardware
LM Studio - Local model hosting with a GUI
Google Gemini - Use Gemini models directly with a Gemini API key
OpenRouter - Access to multiple providers (Anthropic, Google, etc.)
Other providers - Groq, Together AI, Anyscale, Replicate, and more

Sure normalizes tool-call payloads to the strict OpenAI wire format before sending them. Tool result messages omit the deprecated name field that some endpoints reject, and function arguments are always serialized as a JSON-encoded string rather than a raw object. Blank or missing arguments are sent as {}. These adjustments fix 400 errors on endpoints such as GLM/Z.ai that enforce the schema strictly.

OpenAI

OPENAI_ACCESS_TOKEN=sk-proj-...
# No other configuration needed

Recommended models:

gpt-4.1 - Default, best balance of speed and quality
gpt-5 - Highest quality
gpt-6-sol - GPT-6 Sol, vision and PDF capable
gpt-6.1-sol - GPT-6.1 Sol, vision and PDF capable
gpt-4o-mini - Cheaper, good quality

gpt-6-sol and gpt-6.1-sol are vision-capable and use the max_completion_tokens request parameter for PDF processing, matching the behavior of o1 and o3.

Ollama (local)

# Dummy token (Ollama doesn't need authentication)
OPENAI_ACCESS_TOKEN=ollama-local

# Ollama API endpoint
OPENAI_URI_BASE=http://localhost:11434/v1

# Model you pulled
OPENAI_MODEL=llama3.1:13b

Install and run Ollama:

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start Ollama
ollama serve

# Pull a model
ollama pull llama3.1:13b

LM Studio (local)

Download from lmstudio.ai
Download a model through the UI
Start the local server
Configure Sure:

OPENAI_ACCESS_TOKEN=lmstudio-local
OPENAI_URI_BASE=http://localhost:1234/v1
OPENAI_MODEL=your-model-name

LM Studio and some other OpenAI-compatible runtimes reject response_format: { type: "json_object" } with an error like 'response_format.type' must be 'json_schema' or 'text'. If bank statement imports or other structured AI calls fail with this message, set LLM_JSON_MODE=none to omit the response_format parameter entirely.

Google Gemini

You can connect to Gemini directly via Google AI Studio’s OpenAI-compatible endpoint, or via OpenRouter. Recommended Gemini models:

gemini-3.8-flash - Best all-around choice; fast, strong reasoning, and reliable tool calling for chat and finance tools
gemini-3.5-flash-lite - Ultra-low cost and fast; ideal for background categorization and merchant detection
gemini-3.1-pro-preview - Best for complex financial analysis, portfolio queries, and multi-year trends
gemini-2.5-flash - Previous-generation option for users with existing access
gemini-2.5-pro - Previous-generation reasoning model for users with existing access

Google limits direct Gemini 2.5 API access to users who actively used those models previously. For new projects, use gemini-3.5-flash-lite or gemini-3.8-flash; see Google’s model availability guidance.

Direct via Google AI Studio

Google AI Studio provides direct API access to Gemini models with a free tier and an OpenAI-compatible endpoint:

OPENAI_ACCESS_TOKEN=your-google-ai-studio-api-key
OPENAI_URI_BASE=https://generativelanguage.googleapis.com/v1beta/openai
OPENAI_MODEL=gemini-3.8-flash

Via OpenRouter

Access multiple providers through a single API:

OPENAI_ACCESS_TOKEN=your-openrouter-api-key
OPENAI_URI_BASE=https://openrouter.ai/api/v1
OPENAI_MODEL=google/gemini-3.8-flash

Recommended models via OpenRouter:

google/gemini-3.8-flash - Fast and capable
anthropic/claude-sonnet-4.5 - Excellent reasoning
anthropic/claude-haiku-4.5 - Fast and cost-effective

Cost estimates on the LLM usage screen work for OpenRouter models. Sure strips the provider/ prefix from model names (for example, google/gemini-3.8-flash is priced as gemini-3.8-flash) before looking up the rate. Models not in Sure’s pricing table show no estimated cost.

Anthropic

Sure can talk to Anthropic’s Messages API directly, with full chat (text + streaming + tool use) parity with the OpenAI provider.

LLM_PROVIDER=anthropic
ANTHROPIC_ACCESS_TOKEN=sk-ant-...
# Optional — defaults to claude-sonnet-4-6
ANTHROPIC_MODEL=claude-sonnet-4-6

Recommended models:

claude-sonnet-4-6 - Default, best balance of speed and quality
claude-haiku-4-5 - Fast and cost-effective

Issue #4125 reports that Claude Opus 5.5, Sonnet 5.5, and Fable 5.1 reject the forced tool choices used by categorization, merchant detection and enhancement, PDF processing, bank-statement extraction, and bill setup suggestions. The report says chat is unaffected. Until compatibility is resolved, use a model known to support forced tool choice for those features.

Set LLM_CONTEXT_WINDOW to your model’s context size; the setting also applies to native Anthropic chat, even though its UI field is in the OpenAI panel. Values below 4096 collapse account and category rosters to counts. Larger values allow lists, but families with more than 25 visible accessible accounts or more than 60 categories still receive counts and use tools to retrieve the full lists.

Custom Anthropic-compatible endpoints

Set ANTHROPIC_BASE_URL to route requests through Amazon Bedrock, Google Vertex, or any other Anthropic-compatible proxy. ANTHROPIC_MODEL is required when a custom base URL is set, because proxy providers use their own model-ID conventions (for example, Bedrock IDs like anthropic.claude-sonnet-4-5-20250929-v1:0).

LLM_PROVIDER=anthropic
ANTHROPIC_ACCESS_TOKEN=your-proxy-token
ANTHROPIC_BASE_URL=https://your-proxy.example.com
ANTHROPIC_MODEL=anthropic.claude-sonnet-4-5-20250929-v1:0

Anthropic environment variables

Variable	Description	Default
`ANTHROPIC_ACCESS_TOKEN` (or `ANTHROPIC_API_KEY`)	API key for Anthropic (or a compatible proxy)	—
`ANTHROPIC_MODEL`	Model name. Required when `ANTHROPIC_BASE_URL` is set	`claude-sonnet-4-6`
`ANTHROPIC_BASE_URL`	Custom endpoint URL (Bedrock, Vertex, or other proxy). Leave blank for `api.anthropic.com`	—
`ANTHROPIC_REQUEST_TIMEOUT`	HTTP timeout in seconds	`600`
`ANTHROPIC_MAX_TOKENS`	Maximum tokens returned by the model on each chat turn	`4096`

Batch operations

Auto-categorization, merchant detection, and provider-merchant enhancement are implemented on Anthropic via forced tool calls: a single output tool whose input_schema mirrors the desired result, invoked with tool_choice: { type: "tool", disable_parallel_tool_use: true }. This guarantees schema-valid output without JSON-mode fallbacks. Batch size is controlled by LLM_MAX_ITEMS_PER_CALL (default 25), the same variable used by the OpenAI-compatible provider. Raise ANTHROPIC_MAX_TOKENS alongside it — the Messages API requires a reply-token limit and a large batch can exceed the default 4096 — or replies will be cut off.

Token budget

Sure applies a token budget to every outbound LLM call — chat history, auto-categorization, merchant detection, and PDF processing. The defaults are conservative (2048-token context window) so small-context local models like Ollama work out of the box. If you use a cloud provider or a larger-context local model, raise these values.

Configure via settings UI

Go to Settings → Self-Hosting
Scroll to the AI Provider section
Under Token Budget, configure:
- Context Window — total tokens the model accepts (default: 2048)
- Max Response Tokens — tokens reserved for the model’s reply (default: 512)
- Max Items Per Batch — upper bound for auto-categorize and merchant detection batches (default: 25)

Configure via environment variables

Environment variables take precedence over the settings UI.

Variable	Description	Default
`LLM_CONTEXT_WINDOW`	Total tokens the model will accept	`2048`
`LLM_MAX_RESPONSE_TOKENS`	Tokens reserved for the model’s reply. When set, this value is sent to the provider as `max_tokens` (Chat Completions) or `max_output_tokens` (Responses API). Leave unset to use the provider’s default uncapped behavior.	`512`
`LLM_MAX_HISTORY_TOKENS`	Maximum tokens for conversation history. Derived automatically if unset (`context_window - max_response_tokens - system_prompt_reserve`)	Derived
`LLM_SYSTEM_PROMPT_RESERVE`	Tokens reserved for the system prompt	`256`
`LLM_MAX_ITEMS_PER_CALL`	Upper bound on auto-categorize / merchant detection batch size	`25`

LLM_MAX_ITEMS_PER_CALL applies to all providers (both OpenAI-compatible and Anthropic). OpenAI-compatible providers split a batch that doesn’t fit the context window; with the native Anthropic provider, raise ANTHROPIC_MAX_TOKENS alongside LLM_MAX_ITEMS_PER_CALL or replies may be cut off.

Recommended values

Setup	Context window	Max response tokens	Max items per batch
Ollama / small local models	`2048` (default)	`512` (default)	`25` (default)
Cloud OpenAI (gpt-4.1, gpt-5)	`16384` or higher	`4096`	`50`
Cloud OpenAI (gpt-6-sol, gpt-6.1-sol)	`131072` or higher	`8192`	`50`
Large-context local models	`8192`	`2048`	`50`

Responses API routing

Sure automatically routes chat requests to the OpenAI Responses API when using the official OpenAI endpoint, and falls back to the Chat Completions API for custom providers. You can override this behavior with the OPENAI_SUPPORTS_RESPONSES_ENDPOINT environment variable.

Variable	Description	Default
`OPENAI_SUPPORTS_RESPONSES_ENDPOINT`	Set to `true` to force the Responses API on a custom provider, or `false` to force the Chat Completions API on all providers	Auto-detected

AI cache management

Sure caches AI-generated results (like auto-categorization and merchant detection) to avoid redundant API calls and costs.

What is the AI cache?

When AI rules process transactions, Sure stores:

Enrichment records - Which attributes were set by AI (category, merchant, etc.)
Attribute locks - Prevents rules from re-processing already-handled transactions

This caching means:

Transactions won’t be sent to the LLM repeatedly
Your API costs are minimized
Processing is faster on subsequent rule runs

When to reset the AI cache

You might want to reset the cache when:

Switching LLM models - Different models may produce better categorizations
Improving prompts - After system updates with better prompts
Fixing miscategorizations - When AI made systematic errors
Testing - During development or evaluation of AI features

Resetting the AI cache will cause all transactions to be re-processed by AI rules on the next run. This will incur API costs if using a cloud provider.

How to reset the AI cache

Via UI (recommended):

Go to Settings → Rules
Click the menu button (three dots)
Select Reset AI cache
Confirm the action

The cache is cleared asynchronously in the background. You can monitor the reset’s progress and any errors in Settings → Debug by filtering for the ai_cache_reset category. Each run logs when the job is enqueued, when it starts, how many AI cache entries were removed (broken down by scope), and any records that could not be cleared. Automatic reset: The AI cache is automatically cleared for all users when the OpenAI model setting is changed. This ensures that the new model processes transactions fresh.

What happens when cache is reset

AI-locked attributes are unlocked - Transactions can be re-enriched
AI enrichment records are deleted - The history of AI changes is cleared
User edits are preserved - If you manually changed a category after AI set it, your change is kept

Observability with Langfuse

Sure can send traces of every LLM interaction to Langfuse for debugging and usage analytics. Langfuse is open-source and can be self-hosted or used as a cloud service.

Setup

Create a Langfuse project (cloud or self-hosted).
Copy the public key and secret key from the project’s settings.
Set the environment variables:

LANGFUSE_PUBLIC_KEY=your_public_key
LANGFUSE_SECRET_KEY=your_secret_key
# Optional: only needed when self-hosting Langfuse or using a non-default domain
LANGFUSE_HOST=https://your-langfuse-domain.com

In Docker deployments, add these variables to compose.yml and the .env file. Sure reads them on boot and enables tracing automatically. When the keys are absent, Sure runs normally without Langfuse.

What gets traced

Trace	Description
`chat_response`	Conversational AI assistant turns
`auto_categorize`	Background transaction categorization
`auto_detect_merchants`	Background merchant detection

Each trace records the prompt, model, response, and token usage when available.

Viewing traces

After starting Sure with the variables set, open your Langfuse dashboard to see traces and generations grouped under openai.* traces.

Evaluation system

Test and compare different LLMs for your specific use case. The eval system helps you benchmark models for transaction categorization, merchant detection, and chat assistant functionality. See the evaluation framework documentation for details on:

Running evaluations
Comparing models
Creating custom datasets
Langfuse integration for tracking experiments

Vector store (document search)

Sure’s AI assistant can search documents uploaded to a family’s vault. Documents are indexed in a vector store, and the assistant retrieves relevant passages when answering questions.

Supported backends

Backend	Status	Best for	Requirements
OpenAI (default)	Ready	Cloud deployments, zero setup	`OPENAI_ACCESS_TOKEN`
Pgvector	Ready	Self-hosted, full data privacy	PostgreSQL with `pgvector` extension + embedding model
Qdrant	Scaffolded — not yet functional	Self-hosted, dedicated vector DB	Running Qdrant instance

OpenAI (default)

No extra configuration is needed. If you have OPENAI_ACCESS_TOKEN set for the AI assistant, document search works automatically.

Pgvector (self-hosted)

Use PostgreSQL’s pgvector extension for fully local document search. All data stays on your infrastructure. Requirements:

Use the pgvector/pgvector:pg16-trixie Docker image instead of postgres:16
An embedding model served via an OpenAI-compatible /v1/embeddings endpoint (for example, Ollama with mxbai-embed-large)
Run migrations with VECTOR_STORE_PROVIDER=pgvector set to create the vector_store_chunks table

VECTOR_STORE_PROVIDER=pgvector

EMBEDDING_MODEL=mxbai-embed-large           # Default: mxbai-embed-large
EMBEDDING_DIMENSIONS=1024                   # Must match your embedding model (default: 1024)
EMBEDDING_URI_BASE=http://ollama:11434/v1   # Falls back to OPENAI_URI_BASE if not set
EMBEDDING_ACCESS_TOKEN=                     # Optional; falls back to OPENAI_ACCESS_TOKEN

The LLM and embedding endpoints are independent. A common fully local setup uses an OpenAI-compatible chat model via OPENAI_URI_BASE, pgvector for storage, and a separate embedding model via EMBEDDING_URI_BASE. Sure enables the vector extension automatically when it first provisions the chunks table, provided the database user has permission. If the AI status page reports the extension is available but not enabled, connect as the PostgreSQL superuser and run:

CREATE EXTENSION vector;

If you use Ollama, pull the embedding model before starting Sure:

docker compose exec ollama ollama pull mxbai-embed-large

Do not change EMBEDDING_MODEL for an existing pgvector index without rebuilding it. Vectors from different models are not comparable even when they have the same dimensions. Remove existing documents from Sure, drop the chunks table, update the embedding settings, restart Sure, and re-upload the source documents.

Verifying the configuration

Super-admins can open System health → AI at /admin/system_health?tab=ai to run live checks. The tab verifies:

The configured LLM model is accessible
The vector store responds correctly
The embedding endpoint returns vectors with the expected dimensions

These checks run from the web process. To verify that worker processes can also reach the configured provider, click Verify worker configuration on the same tab. See AI status tab and Verifying worker configuration in the self-hosting guide for details on the checks, worker-specific caveats, and how to tune probe timeouts.

Additional environment variables

These optional variables fine-tune provider selection and the behavior of the OpenAI-compatible provider. See Anthropic environment variables for the Anthropic-specific options.

Variable	Description	Default
`LLM_PROVIDER`	Active LLM provider: `openai` or `anthropic`. Takes precedence over any provider choice saved in the UI.	`openai`
`OPENAI_ACCESS_TOKEN`	API key for the provider	—
`OPENAI_URI_BASE`	Custom endpoint URL (leave blank for OpenAI)	—
`OPENAI_MODEL`	Model name (required for custom endpoints). A blank value is treated as unset and falls through to the UI setting, then `gpt-4.1`	`gpt-4.1`
`OPENAI_REQUEST_TIMEOUT`	HTTP timeout in seconds for each individual HTTP call to the model	`60`
`AI_RESPONSE_TIMEOUT`	Whole-turn watchdog timeout in seconds. See Chat response timeout for sizing guidance	`90`
`ASSISTANT_MAX_TOOL_CALL_ITERATIONS`	Maximum number of tool-call rounds per chat turn	`8`
`OPENAI_SUPPORTS_PDF_PROCESSING`	Set to `false` for endpoints without vision support	`true`
`OPENAI_SUPPORTS_RESPONSES_ENDPOINT`	Override Responses API vs Chat Completions routing	Auto-detected
`LLM_JSON_MODE`	JSON output mode for OpenAI-compatible categorization and merchant-detection calls, plus text-based PDF and bank-statement extraction: `auto`, `strict`, `json_object`, or `none`	—

Native OpenAI Responses calls and PDF vision processing do not use LLM_JSON_MODE. For text extraction without a JSON schema, strict falls back to json_object. See Token budget for LLM_CONTEXT_WINDOW, LLM_MAX_RESPONSE_TOKENS, and related variables.

Chat response timeout

Sure uses two separate timeouts for AI chat turns:

OPENAI_REQUEST_TIMEOUT — bounds each individual HTTP call to the model. The default is 60 seconds.
AI_RESPONSE_TIMEOUT — a whole-turn watchdog that starts when the message is queued and covers everything: queue wait, every model call, and all tool executions in between. The default is 90 seconds, which is sized for typical cloud latency.

Why the default 90 seconds is often too short for local models

Custom OpenAI-compatible providers (Ollama, LM Studio, etc.) route through a synchronous code path, so nothing renders until the full reply is generated. On top of that, a tool-using turn runs multiple model calls in sequence. The total time the watchdog must cover is:

(1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS) × OPENAI_REQUEST_TIMEOUT
  + tool execution time + queue wait

With the defaults (ASSISTANT_MAX_TOOL_CALL_ITERATIONS=8, OPENAI_REQUEST_TIMEOUT=60), a worst-case turn takes up to 540 seconds of model time — far beyond the 90-second watchdog. The chat will report “assistant not available” even though the model is still generating a reply.

Recommended approach

The cheapest lever is lowering ASSISTANT_MAX_TOOL_CALL_ITERATIONS rather than setting a very large timeout. Fewer iterations means a shorter worst-case bound, and long tool chains fail with a clear limit error instead of a silent timeout. For a local Ollama setup with slow generation:

OPENAI_REQUEST_TIMEOUT=300
ASSISTANT_MAX_TOOL_CALL_ITERATIONS=3
# (1 + 3) × 300 = 1200s model time + ~300s headroom = 1500
AI_RESPONSE_TIMEOUT=1500

In compose.yml, pass ASSISTANT_MAX_TOOL_CALL_ITERATIONS alongside the other AI variables so the setting actually reaches the web and worker containers:

x-rails-env: &rails_env
  OPENAI_REQUEST_TIMEOUT: "300"
  ASSISTANT_MAX_TOOL_CALL_ITERATIONS: ""   # leave empty to use the app default (8)
  AI_RESPONSE_TIMEOUT: "1200"

Leave the value empty to let the app’s own default govern, or set it explicitly to cap tool-call rounds. For cloud providers (OpenAI, Anthropic) the defaults are usually fine. If you see timeouts, raise AI_RESPONSE_TIMEOUT using the formula above.

AI_RESPONSE_TIMEOUT must be sized as a sum, not just set above OPENAI_REQUEST_TIMEOUT. The watchdog covers the entire turn — queue wait, every model call, and all tool executions — so the bound is:

(1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS) × OPENAI_REQUEST_TIMEOUT
  + tool execution time + queue wait

Setting AI_RESPONSE_TIMEOUT equal to or below OPENAI_REQUEST_TIMEOUT means any tool-using turn will always time out. The 90-second default is sized for typical cloud latency; use the formula above once per-call latency approaches the timeout.

Configure via settings UI

AI_RESPONSE_TIMEOUT is also configurable from the web interface:

Go to Settings → Self-Hosting
Scroll to the AI Provider section
Set Chat Response Timeout to your desired value in seconds

The environment variable takes precedence over the UI setting. The minimum accepted value is 30 seconds.

Docker compose example

Basic Ollama setup

services:
  sure:
    environment:
      - OPENAI_ACCESS_TOKEN=ollama-local
      - OPENAI_URI_BASE=http://ollama:11434/v1
      - OPENAI_MODEL=llama3.1:13b
      # Optional: raise token budget for larger-context models
      # - LLM_CONTEXT_WINDOW=8192
      # - LLM_MAX_RESPONSE_TOKENS=2048
      # - LLM_MAX_ITEMS_PER_CALL=50
    depends_on:
      - ollama

  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    # Uncomment if you have an NVIDIA GPU
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: 1
    #           capabilities: [gpu]

volumes:
  ollama_data:

Advanced AI setup with OpenClaw

For advanced AI features including code execution and tool use, you can use the local-ai profile with OpenClaw:

services:
  sure:
    profiles:
      - local-ai
    environment:
      - OPENAI_ACCESS_TOKEN=ollama-local
      - OPENAI_URI_BASE=http://openclaw:8080/v1
      - OPENAI_MODEL=llama3.1:13b
    depends_on:
      - openclaw
      - ollama

  openclaw:
    image: openclaw/openclaw:latest
    profiles:
      - local-ai
    ports:
      - "8080:8080"
    environment:
      - OLLAMA_HOST=http://ollama:11434
    depends_on:
      - ollama

  ollama:
    image: ollama/ollama:latest
    profiles:
      - local-ai
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

volumes:
  ollama_data:

To use the local AI setup:

# Start with local AI profile
docker compose --profile local-ai up

# Or include it with other profiles
docker compose --profile local-ai --profile external-assistant up

The local-ai profile includes:

Ollama - Local LLM inference
OpenClaw - Gateway providing enhanced AI capabilities and tool use
Automatic routing between Sure and the AI stack

Custom AI prompts

Family admins can override the built-in prompts from Settings → AI Prompts — no fork or redeploy required. Overrides are stored per family, so one family’s edits never affect another on the same deployment. Five prompts are editable:

Prompt	What it controls
Chat system prompt	The static instructions the assistant follows in every conversation
Categorizer (OpenAI)	Instructions for the background transaction categorization job
Categorizer (Anthropic)	Same, when using an Anthropic provider
Merchant detector (OpenAI)	Instructions for the background merchant detection job
Merchant detector (Anthropic)	Same, when using an Anthropic provider

Each field opens with the current default text for your deployment, or the saved override if one exists. Clearing a field resets it to the default. The Reset to default button asks for confirmation before restoring the original text. A live character counter and status label sit below each field. The limit is 20,000 characters per prompt. When using a custom OpenAI-compatible endpoint (OPENAI_URI_BASE set), the categorizer and merchant detector fields show the terse variant those models receive. A single override replaces both the terse and detailed variants, so editing the field on a custom endpoint affects local models only.

How the chat prompt works

The builtin assistant’s system prompt has two parts: a static half (STATIC_INSTRUCTIONS) that is byte-identical on every request so providers can cache it, and a volatile ## Session context block that holds the current date, currency, account roster, and categories. Only the static half is editable here. What you can adjust: tone and personality, response format, domain-specific rules and constraints. Overriding the chat prompt gives up prefix caching for that family. The static half is normally byte-stable across all families sharing an API key; a family that overrides it gets its own prefix, which is not shared with others. The first request after each save also pays the full uncached price.

Output format warnings for categorizer and merchant prompts

The categorizer and merchant parsers expect a specific JSON structure ({"categorizations": [...]} or {"merchants": [...]}). Sure also asks for those keys in a per-request message that your override cannot replace, so rewording the example is generally safe. Parsing breaks when an override contradicts the output format — for example, asking for reasoning before the answer, using a different wrapper key, or requesting YAML instead of JSON. Smaller local models tend to follow the system prompt over the per-request instruction. For that reason, the editor warns when a custom OpenAI override removes the example JSON block, even though the request-level fallback usually covers it.

External AI assistant

Instead of using the built-in LLM, you can delegate chat to an external AI agent. The agent receives the conversation, can call back to Sure’s financial data via MCP, and streams a response back. This is useful when:

You have a custom AI agent with domain knowledge, memory, or personality.
You want to use a non-OpenAI-compatible model (the agent handles translation).
You want to keep LLM credentials and logic entirely outside Sure.

The external assistant handles chat only. Auto-categorization and merchant detection still use the configured LLM provider (OPENAI_ACCESS_TOKEN for OpenAI-compatible providers, or Jev if you have selected it as the categorization provider).

How it works

A user sends a message in the Sure chat UI.
Sure forwards the conversation to your agent’s API endpoint (OpenAI chat completions format).
Your agent processes it using whatever LLM, tools, or context it needs.
Your agent can call Sure’s /mcp endpoint for financial data and actions (accounts, transactions, balance sheet, budgets, file search, statement import, goals).
Your agent streams the response back to Sure via Server-Sent Events (SSE).

The agent’s API must be OpenAI chat completions compatible: accept POST with a messages array and return SSE with delta.content chunks.

Configuration

Settings UI:

Go to Settings → Self-Hosting.
Set Assistant type to “External (remote agent)”.
Enter the Endpoint URL and API Token from your agent provider.
Sure loads the available agents from the endpoint’s /v1/models response and shows them in the Agent dropdown. Select the agent that should handle conversations.

Environment variables:

ASSISTANT_TYPE=external                          # Global override (or set per-family in UI)
EXTERNAL_ASSISTANT_URL=https://your-agent/v1/chat/completions
EXTERNAL_ASSISTANT_TOKEN=your-api-token
EXTERNAL_ASSISTANT_MODEL=openclaw/main           # Agent identifier from /v1/models; defaults to openclaw/main
EXTERNAL_ASSISTANT_SESSION_KEY=agent:main:main   # Optional, for session persistence
EXTERNAL_ASSISTANT_ALLOWED_EMAILS=user@example.com  # Optional, comma-separated allowlist

When environment variables are set, the corresponding UI fields are disabled and the env values take precedence. Set ASSISTANT_TYPE=external to route all users to the external agent globally. Without it, routing falls back to each family’s assistant_type setting in the UI (default: builtin).

EXTERNAL_ASSISTANT_AGENT_ID is a legacy variable kept for backwards compatibility. Use EXTERNAL_ASSISTANT_MODEL for new installs.

MCP callback endpoint

Sure exposes a Model Context Protocol (MCP) endpoint at /mcp so your external agent can call back and query financial data. Authentication: Bearer token via Authorization header. Sure supports both OAuth bearer tokens (via its discovery and registration endpoints) and static bearer tokens configured with MCP_API_TOKEN and MCP_USER_EMAIL. Static-token environment variables:

MCP_API_TOKEN=your-secret-token    # Bearer token the agent sends to authenticate
MCP_USER_EMAIL=user@example.com    # Email of the Sure user the agent acts as

See MCP server for external AI assistants for the full list of available tools and protocol details.

Access control

Use EXTERNAL_ASSISTANT_ALLOWED_EMAILS to restrict which users can access the external assistant. When set, only users whose email is in the comma-separated list see the AI chat. When blank, all users have access.

Docker Compose example

x-rails-env: &rails_env
  ASSISTANT_TYPE: external
  EXTERNAL_ASSISTANT_URL: https://your-agent/v1/chat/completions
  EXTERNAL_ASSISTANT_TOKEN: your-api-token
  MCP_API_TOKEN: your-mcp-token
  MCP_USER_EMAIL: user@example.com

You can also configure the assistant via the Settings UI after startup; the MCP environment variables are still required for agent callbacks.

​Architecture: two AI pipelines

​Where AI jobs run (Sidekiq lanes)

​Quickstart

​Supported providers

​Configuration via settings

​Choosing a provider

​OpenAI compatible API

​OpenAI

​Ollama (local)

​LM Studio (local)

​Google Gemini

​Direct via Google AI Studio

​Via OpenRouter

​Anthropic

​Custom Anthropic-compatible endpoints

​Anthropic environment variables

​Batch operations

​Token budget

​Configure via settings UI

​Configure via environment variables

​Recommended values

​Responses API routing

​AI cache management

​What is the AI cache?

​When to reset the AI cache

​How to reset the AI cache

​What happens when cache is reset

​Observability with Langfuse

​Setup

​What gets traced

​Viewing traces

​Evaluation system

​Vector store (document search)

​Supported backends

​OpenAI (default)

​Pgvector (self-hosted)

​Verifying the configuration

​Additional environment variables

​Chat response timeout

​Why the default 90 seconds is often too short for local models

​Recommended approach

​Configure via settings UI

​Docker compose example

​Basic Ollama setup

​Advanced AI setup with OpenClaw

​Custom AI prompts

​How the chat prompt works

​Output format warnings for categorizer and merchant prompts

​External AI assistant

​How it works

​Configuration

​MCP callback endpoint

​Access control

​Docker Compose example

Architecture: two AI pipelines

Where AI jobs run (Sidekiq lanes)

Quickstart

Supported providers

Configuration via settings

Choosing a provider

OpenAI compatible API

OpenAI

Ollama (local)

LM Studio (local)

Google Gemini

Direct via Google AI Studio

Via OpenRouter

Anthropic

Custom Anthropic-compatible endpoints

Anthropic environment variables

Batch operations

Token budget

Configure via settings UI

Configure via environment variables

Recommended values

Responses API routing

AI cache management

What is the AI cache?

When to reset the AI cache

How to reset the AI cache

What happens when cache is reset

Observability with Langfuse

Setup

What gets traced

Viewing traces

Evaluation system

Vector store (document search)

Supported backends

OpenAI (default)

Pgvector (self-hosted)

Verifying the configuration

Additional environment variables

Chat response timeout

Why the default 90 seconds is often too short for local models

Recommended approach

Configure via settings UI

Docker compose example

Basic Ollama setup

Advanced AI setup with OpenClaw

Custom AI prompts

How the chat prompt works

Output format warnings for categorizer and merchant prompts

External AI assistant

How it works

Configuration

MCP callback endpoint

Access control

Docker Compose example