Architecture
What it is
Learn2RAG is a self-hosted RAG (retrieval-augmented generation) system. An administrator uses a web UI (the “configurator”) to register language models, data sources (local folders, websites, Drupal, MediaWiki, Jira, SharePoint) and pipelines. Each pipeline then runs as a small group of local processes: an importer that indexes documents into a Qdrant vector database, and a chat UI where end users ask questions and get answers grounded in those documents. No Docker is needed. Everything runs as plain processes started from one executable.
User documentation: https://docs.learn2rag.de/
Big picture
There are two diagrams, because two different things happen: first processes get started, then data flows through them.
1. Who starts what
An arrow means “starts (spawns) this process”. Nothing here carries data.
flowchart TB
Admin([Administrator]) -- uses browser --> UI["Admin UI<br/>learn2rag.ui, Flask :9000"]
UI -- saves models, sources, pipelines as YAML --> Data[("instance/data/*.yml")]
UI -- starts at launch --> Ollama["Ollama :11434"]
UI -- "Start pipeline: renders a compose template" --> Compose["learn2rag.compose<br/>process orchestrator, PIDs in compose.db"]
Compose -- starts --> Qdrant[("Qdrant<br/>vector database")]
Compose -- "starts (import / continuous)" --> Importer["learn2rag.importer"]
Compose -- "starts (pipeline / continuous)" --> UserUI["Chat UI<br/>learn2rag.userui :9001+"]
Compose -- "starts (optimization)" --> Opt["learn2rag.optimization"]
Which processes run depends on the template you choose: pipeline starts Qdrant and the chat UI, import starts Qdrant and the importer once, continuous starts Qdrant, the importer (every N hours) and the chat UI, and optimization starts Qdrant and the optimizer.
2. How data flows
An arrow means “data moves this way”. Read it top to bottom: the importer fills Qdrant (A), a user’s question reads from it (B), and optimization tunes the settings both use (C).
flowchart TB
subgraph A["A. Import: runs once or on a schedule"]
Sources[("Sources: folders, web, Drupal,<br/>MediaWiki, Jira, SharePoint")]
Sources -- raw documents --> Loaders["importer loaders"]
Loaders -- documents --> Ingest["pipeline.ingestion<br/>split into chunks + embed to vectors"]
end
Ingest -- "chunks + vectors (upsert)" --> Qdrant[("Qdrant<br/>stored chunks + vectors")]
subgraph B["B. Chat: every user question"]
User([End user]) -- question --> UserUI["Chat UI /chat"]
UserUI -- "POST /api/v1/chat/completions" --> Search["1. Search<br/>rewrite question, embed it"]
Search -- query vector --> Qdrant
Qdrant -- most similar chunks --> Rerank["2. Rerank + authorization filter<br/>drop chunks this user may not see"]
Rerank -- top chunks --> Gen["3. Generate<br/>prompt = question + chunks"]
Gen -- prompt --> LLM["LLM<br/>Ollama or OpenAI-compatible API"]
LLM -- answer text --> Gen
Gen -- "answer (streamed)" --> UserUI
UserUI -- answer --> User
end
subgraph C["C. Optimization: optional"]
Train[("training.csv<br/>questions + expected answers")] --> Opt["optimization<br/>runs Search + Generate, scores answers"]
end
Opt -- best parameters --> OptCfg[("opt_config.json<br/>chunk size, embedding model, top-k, prompt...")]
OptCfg -. read by .-> Ingest
OptCfg -. read by .-> Search
After an import, the data does not go anywhere by itself. It stays in Qdrant until a user asks a question. Search then reads it back out, and only the few best-matching chunks are sent to the LLM along with the question.
Main modules
All Python code lives under learn2rag/.
| Module | Responsibility |
|---|---|
__main__.py | Launcher. Every process starts as configurator <module> [args]. It reads config.yml from the CWD, sets up logging, imports the module, builds that module’s arguments and calls module.main(...). With --schedule-interval it calls main repeatedly on a timer (APScheduler). Continuous import uses this. Defaults to learn2rag.ui. |
ui/ | Admin UI (Flask, server-rendered Jinja + htmx, German/English via Babel). CRUD for models, sources and pipelines. Starting a pipeline renders a compose template and hands it to compose. On startup it also launches Ollama. |
ui/templates/compose/ | Process templates (Jinja-rendered YAML). pipelines/pipeline.yml (chat only), import.yml (one-off import), continuous.yml (scheduled import + chat), optimization.yml. components/ollama*.yml start Ollama or pull a model. |
data/ | Tiny persistence layer: each model, source or pipeline is one YAML file under <instance>/data/<category>/<uuid>.yml. |
compose/ | Process orchestrator, a minimal docker-compose for local processes. It writes the files listed in a template, starts services (command, env, working dir), runs curl healthchecks and tracks PIDs and status in SQLite (compose.db). Handles POSIX process groups and Windows. |
importer/ | Ingestion job. It reads importer_config.json, runs one loader per source (loaders/*.py) and passes the documents to pipeline.ingestion. In --delta mode it keeps import_state.json (hashes and timestamps) and only re-indexes changed documents. |
pipeline/ | RAG core. It covers chunking and ingestion, embeddings (sentence-transformers / BGE-M3: dense, sparse, ColBERT), the Qdrant client, query rewriting, search with reranking, per-source authorization filters (Drupal, SharePoint), generation with LangChain chat models (llm.py), and provenance and profiling logs (prov.py). operators/ composes these into async steps. BasicPipeline = SearchOperator → GenerationOperator. api.py exposes it over HTTP. |
userui/ | End-user chat app (FastAPI + uvicorn). It mounts pipeline.api under /api, serves the llama.cpp web UI as static files under /chat, and provides OAuth login (auth/) so the pipeline can filter results by what each user may see. |
optimization/ | Hyperparameter tuning of opt_config.json (retrieval, prompt and contrastive strategies, SMAC/ConfigSpace, bert-score) against a Q&A training CSV that the admin uploads. |
bootstrap/ | Shared Bootstrap CSS, images and base templates for both UIs. |
ollama_tool.py, download_models.py | Helpers to pull Ollama models and pre-download embedding and reranker models. |
evaluation/ | Offline evaluation scripts. Not used at runtime. |
Outside learn2rag/:
services/: Qdrant, Ollama and llama.cpp binaries plus their install scripts.services/llama.cppis a submodule; its built web UI is what end users chat in.drupal/,mediawiki/: docker-compose test instances of those sources.ansible/,systemd/: server deployment.pyapp/,package-*,build-python: build the singleconfiguratorbinary for release bundles.
How the pieces connect
1. Admin configures and starts a pipeline
- The admin adds a model (an Ollama model to pull, or a URL, token and model name for an OpenAI-compatible API), one or more sources, and a pipeline that ties a model to sources and a storage directory. Each is saved as YAML through
learn2rag.data. - On “start”,
start_pipeline()inui/__init__.py:- builds
importer_config.jsoncontent from the selected sources (mappinglocal→DirectoryLoader,web→HTMLLoader, and so on), - picks free ports (
find_free_ports, preferring 9001–9010), - generates a random Qdrant API key,
- renders the chosen compose template with this context and calls
compose.Project.create(...).start().
- builds
composewrites the config files into the pipeline’s storage directory and spawns each service. Every service command calls back into the same executable, for exampleconfigurator learn2rag.userui --port 9001 ....
2. Import (indexing)
configurator learn2rag.importer --config importer_config.json --delta. The loaders fetch documents from the sources, then pipeline.ingestion.index chunks them, embeds them and upserts them into the Qdrant collection. A continuous pipeline runs the same command with --schedule-interval PT<n>M (every 12 h by default).
3. Query (chat)
- The browser loads the llama.cpp UI from
/chat. That UI calls the relative pathsv1/modelsandv1/chat/completions. A middleware (PrefixRewriteMiddleware) rewrites these from/chat/...to/api/.... pipeline/api.pyreceives an OpenAI-style chat request and runsBasicPipeline:- Search: optionally rewrite the question using the chat history, embed it, query Qdrant, rerank, then drop documents the user may not access (
authorization.pywith the per-source filters, using the OAuth tokens stored in the user’s session). - Generate: build a prompt with the retrieved context and call the configured LLM.
- Search: optionally rewrite the question using the chat history, embed it, query Qdrant, rerank, then drop documents the user may not access (
- The answer is returned as JSON or streamed as server-sent events. Profiling and provenance lines go to
<storage>/logs/profiling.log.
APIs and external services
Exposed HTTP endpoints
| Process | Endpoint | Purpose |
|---|---|---|
Admin UI (:9000) | /, /models, /sources, /pipelines, /pipelines/<name>, /pipelines/<name>/logs/<file>, /pipelines/<name>/training, /ps, /shutdown | HTML pages and form POSTs |
Chat UI (:9001+) | GET / | Landing page with OAuth login buttons |
/chat/* | llama.cpp web UI (static), plus /chat/props | |
GET /api/v1/models | Returns a single model, Learn2RAG | |
POST /api/v1/chat/completions | OpenAI-compatible chat, streaming or not | |
POST /api/v1/stream | Streaming variant | |
/auth/oauth/{login,callback,logout} | OAuth login against a source (Drupal, SharePoint) |
Outgoing calls
| Called service | From | How it is configured |
|---|---|---|
| LLM (Ollama or OpenAI-compatible) | pipeline/llm.py via LangChain ChatOllama / ChatOpenAI | LLM_API_TYPE, LLM_API_URL, LLM_API_TOKEN, LLM_API_MODEL, LLM_API_PROXY |
| Ollama management (list, pull) | ui/__init__.py via the ollama client | OLLAMA.port in config.yml (default 11434) |
| Qdrant | pipeline/qdrant.py | QDRANT_LOCATION (URL) or QDRANT_PATH (embedded, on disk), plus QDRANT__SERVICE__API_KEY |
| Data sources | importer/loaders/* | Per-source settings (URLs, credentials, OAuth client) in importer_config.json. Jira can also use JIRA_USERNAME / JIRA_API_TOKEN. |
| Authorization checks (Drupal, SharePoint / MS Graph) | pipeline/authorization_*.py | Uses the end user’s OAuth token from the session |
| Hugging Face models (embeddings, rerankers) | pipeline/embeddings.py, search.py, download_models.py | Model names in opt_config.json |
Authentication
- Admin UI: optional HTTP basic auth via
SIMPLE_AUTH: {username, password}inconfig.yml. By default it binds to 127.0.0.1 only. - Chat UI: optional basic auth (
CHAT.username/password, falling back toSIMPLE_AUTH), passed as--basic-username/--basic-password. OAuth applications are discovered automatically from sources that haveoauth_client_idand a client secret. After login, the token is stored in the session and used to filter search results per user. - TLS for both UIs:
TLS: {CERTFILE, KEYFILE}inconfig.yml.
Configuration and environment variables
Admin-level configuration (YAML, layered):
learn2rag/ui/config.yml: built-in defaults such asSUGGESTED_MODELS. Users do not edit it.config.ymlin the working directory (next to the executable): user overrides, merged on top. Main keys:UI(host,port),CHAT(host,username,password,forwarded-allow-ips),SIMPLE_AUTH,TLS,OLLAMA.port,PREFERRED_PORTS,logging.debug,flask.instance_path.
Admin UI state: the Flask instance directory, by default the platform data dir Learn2RAG/instance (e.g. ~/.local/share/Learn2RAG/instance):
data/{models,sources,pipelines}/*.yml: everything the admin created.storage/<pipeline>/: default per-pipeline storage (Qdrant data, generated configs, logs).ollama/: Ollama’s working directory.
Process state is kept in compose.db (SQLite, in the CWD unless COMPOSE_DB is set).
Per-pipeline files (generated into the storage directory by the compose template):
importer_config.json: the list of loaders and source settings.basic_user_config.json: collection name and LLM. Read viaPIPELINE_USER_CONFIG.opt_config.json: tunable retrieval and generation parameters (chunking, embedding model, top-k, reranker, prompt…), copied frompipeline/opt_config.jsonand overwritten by optimization. Read viaPIPELINE_OPT_CONFIG. Seepipeline/CONFIG_PARAMS.md.qdrant_config.yml,logging_config.yml,logs/{status,error,profiling,debug}.log.
Environment variables set for pipeline processes (see ui/templates/compose/pipelines/*.yml): LEARN2RAG_PATH, STORAGE_PATH, PIPELINE_USER_CONFIG, PIPELINE_OPT_CONFIG, IMPORTER_CONFIG, QDRANT_LOCATION, QDRANT_PATH, QDRANT__SERVICE__API_KEY, LLM_API_TYPE, LLM_API_URL, LLM_API_TOKEN, LLM_API_MODEL. pipeline/config.py loads the three JSON configs at import time from these variables, falling back to the copies in learn2rag/pipeline/.