Hybrid semantic search MCP server for research papers, code, and notes. Ingests documents into a local SQLite database with FTS5 full-text search and sqlite-vec vector similarity, then exposes them as MCP tools for AI assistants.

Part of a four-layer cognitive architecture research project. Companion to memory-engine (Rust crate, Memory layer).

• Persistent knowledge base — ingest PDFs, markdown, code, and web pages into a single searchable index that persists across sessions
• Hybrid search — combines BM25 keyword search (FTS5) with cosine vector similarity (sqlite-vec), merged via Reciprocal Rank Fusion, with stage-2 reranking
• Paper management — register papers with metadata, track relationships (extends, contradicts, replicates), export/sync BibTeX
• Structured extraction — LLM-powered extraction of methods, datasets, metrics, and entities from paper text; cross-paper comparison on shared datasets
• Map-reduce for long documents — structured extraction handles documents of any size by splitting into chunks, extracting per-chunk, then merging with entity resolution
• Multi-space embeddings — maintain multiple embedding spaces simultaneously, compare retrieval quality, promote/deprecate spaces without data loss
• Dual chunking — 8K + 32K chunk strategies for fine-grained and broad-context retrieval
• Auto-relationship discovery — similarity-based detection of cross-paper connections
• Folder-level semantic embeddings — context boosting from folder summaries
• Keyword intent extraction — pre-filter search queries for better precision
• Ingestion session tracking — co-occurrence signals for retrieval ranking
• Configurable LLM backend — use Ollama natively or any OpenAI-compatible endpoint (e.g. LM Studio, vLLM)
• Embedding model flexibility — swap embedding models and re-embed all chunks without data loss

• No cloud sync — the index is a local SQLite database
• No automatic paper discovery — you ingest documents manually or via URL
• Structured extraction depends on an LLM (Ollama or OpenAI-compatible) — quality varies with model capability

┌─────────────────────────────────────────────┐
│              MCP Client (Claude, etc.)       │
└──────────────────┬──────────────────────────┘
                   │ MCP protocol
┌──────────────────▼──────────────────────────┐
│              FastMCP Server (46 tools)        │
├──────────────────────────────────────────────┤
│  ingestion.py  │ search.py    │ papers.py    │
│  embeddings.py │ extraction.py│ operations.py│
├──────────────────────────────────────────────┤
│  embeddings.py (pluggable: Ollama, OpenAI)   │
├──────────────────────────────────────────────┤
│  SQLite + FTS5 + sqlite-vec                  │
└──────────────────────────────────────────────┘

• Python 3.12+
• uv package manager
• Ollama running locally (or on a Windows host for WSL2)
  • ollama pull bge-m3 for embeddings
  • ollama pull qwen3.5:27b for structured extraction (optional, or use any OpenAI-compatible endpoint)

git clone https://github.com/dutiona/knowledge-base.git
cd knowledge-base
uv sync

uv run knowledge-base

Or register as an MCP server in your client's config (e.g. Claude Code settings.json):

{
  "mcpServers": {
    "knowledge-base": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/path/to/knowledge-base",
        "knowledge-base"
      ]
    }
  }
}

uv run pytest tests/ -q

Phase 2 complete (338+ tests, ~11K lines Python, 34 source files). Phase 2.5 stabilization (bug fixes + module decomposition) done.

The index is stored at ~/.local/share/knowledge-base/knowledge.db by default. Override the location with (highest precedence first) the --db-path CLI flag, the KNOWLEDGE_BASE_DB environment variable, or fall back to the default — the MCP server reads KNOWLEDGE_BASE_DB. Key tables:

• chunks — document content with content-hash deduplication
• chunks_fts — FTS5 full-text index (auto-synced via triggers)
• chunks_vec — sqlite-vec vector index
• papers / papers_fts — paper metadata with full-text search
• relationships — inter-paper relationships
• conclusions — research conclusions with supersession chains
• methods / datasets / metrics — structured extraction results
• entities / entity_mentions — resolved entities and surface forms
• embed_spaces — multiple embedding space configurations
• chunk_sessions — N:M ingestion session tracking
• folder_summaries / folder_summaries_vec — folder-level semantic embeddings
• prediction_errors — stale result detection
• jobs — background job tracking
• config — key-value store (embedding model, LLM settings, schema version)

The database carries a schema_version (a config row). On open, the code validates the version and refuses to operate on a DB that is newer than the running build or behind it (telling you to migrate). Two operator commands — both offline, embedding-free, and safe to use in a CI/release gate:

# Report live vs current schema version. Exit code 0 = match, non-zero = mismatch.
knowledge-base-ingest --db <path> schema

# Dry run: list pending migrations without touching the DB (non-zero unless current).
knowledge-base-ingest --db <path> migrate --check

# Apply pending migrations. A fresh DB is initialized to the current version;
# an existing DB is BACKED UP first (see below), then migrated.
knowledge-base-ingest --db <path> migrate

• Stop the MCP server before migrate. The server caches its connection and would keep serving the old schema (and may contend for the write lock) until restarted. schema / migrate --check are read-only and safe against a running server.
• Backups. migrate writes a timestamped copy to <db dir>/backups/ (or --backup-dir) via VACUUM INTO before mutating. If a migration fails, the backup is restored automatically and the command aborts non-zero. Backups are not auto-pruned — clean them up yourself.

• Single-user, single-process (SQLite WAL mode, no concurrent writers)
• AST-aware chunking only for Python files; other code uses fixed-size chunks
• Embedding model swap re-embeds all chunks sequentially (can be slow for large indexes)