Skip to content
View as markdown Ask ChatGPT Ask Claude

RAG Runtime#

The Retrieval-Augmented Generation (RAG) runtime provides additional context to large language models (LLMs) by indexing relevant information and retrieving it during AI interactions.

The RAG runtime is implemented as a system-managed service that:

  • Processes configured content and generates embeddings.
  • Stores content chunks and their embeddings in the system ClickHouse database.
  • Retrieves the most relevant content when requested by an MCP Server.
  • Returns retrieved content that the LLM can use to generate a context-aware response.

Runtime Components#

The RAG runtime uses the following system components:

  • RAG service - indexes content, generates embeddings, performs similarity searches, and returns relevant context to requesting MCP Servers.
  • Ollama - provides the embedding model used by the RAG service and, with that model, generates vector embeddings.
  • ClickHouse - stores the chunked content and embeddings used by the RAG service.

Runtime Availability#

The system administrator must deploy and configure the RAG service before AI Agents can use Retrieval-Augmented Generation capabilities. The runtime components are system-managed and are not configured from the AI administration pages.

Runtime Behavior#

When content is indexed, the RAG service splits it into smaller chunks, generates embeddings using Ollama, and stores both the chunks and their embeddings in ClickHouse.

The RAG service automatically updates indexed content when relevant deployment or operation changes occur. These updates run asynchronously and do not require configuration in the AI administration pages.

During an AI interaction, an MCP Server can request relevant context from the RAG service. The RAG service performs a similarity search against the indexed content, retrieves the most relevant chunks, and returns them to the MCP Server. The MCP Server then includes this context in the prompt sent to the LLM, enabling more accurate and context-aware responses.

For information about the available AI Agents and MCP Servers, see LibreChat Integration.