A high-performance semantic code search engine designed for the MediaWiki ecosystem. Built on the Qwen 0.6B neural retrieval model, optimized for large-scale codebases like MediaWiki Core, Extensions, and WMF Operations. Metadata is managed via indexed SQLite for sub-second responses and a low-memory footprint (Toolforge compatible).

As featured on Wikimedia Diff (v1.0.0).

• 📂 Global MediaWiki Indexing: Covers Core, Extensions, Skins, Libraries, Services, and more (2,500+ unique repos).
• 🧠 Single-Stage Neural Retrieval: Uses Qwen3-Embedding-0.6B with FAISS IndexIVFPQ for lightning-fast results (approx. 0.3s).
• 🌳 Granular Structural Filtering: High-precision extraction and filtering of Functions, Types, Template Functions, and Template Types across 10 languages.
• 🏗️ Split-Build Architecture: Optimized for asymmetric hardware—run heavy extraction on a laptop and neural vectorization on a GPU.
• 🌍 Massive Localization Footprint: Fully localized UI supporting 17 languages.
• 🎨 Codex UI: A clean, accessible frontend built with Wikimedia's Codex Design System for a native look and feel.
• 🔍 Advanced Multi-select Filtering: Granular control over results by repository group, programming language, and entry type.

mediawiki-code2code-search/
├── frontend/                  # Codex-based Static Frontend
│   ├── css/style.css          # Stylesheets using the Codex Design System
│   ├── js/main.js             # Main frontend application logic
│   └── i18n/                  # Localization JSONs supporting 17 languages
├── backend/                   # FAISS Index, SQLite & Vector DB Management
│   ├── generate_embeddings.py # Computes neural embeddings from raw snippets (saves embeddings.npy)
│   ├── build_index.py         # Trains and builds the FAISS search index from saved embeddings
│   ├── migrate_to_sqlite.py   # RAM optimization script (JSON metadata -> SQLite)
│   ├── snippets.db            # SQLite metadata store for fast lookups
│   └── mediawiki.index        # Compiled FAISS vector index
├── preprocessing/             # Global-Scale Indexing Pipeline (Phases 1-4)
│   ├── list_repos.py          # Discovers and lists 2,500+ MediaWiki repositories
│   ├── download_repos.py      # Handles shallow clones of target repositories
│   ├── extract_entities.py    # Structural parsing & AST entity extraction
│   ├── archive_to_swh.py      # Software Heritage archiving pipeline scripts
│   └── resolve_swh_hashes.py  # Resolves local Git hashes to SWH SHA1 IDs
├── tests/                     # Parser & API Verification Suite
│   ├── test_api.py            # Backend API endpoint tests
│   ├── test_*_parser.py       # Syntax extraction validations for 10+ languages
│   └── example.*              # Target language snippets parsed during testing
├── scripts/                   # Internal utilities & metadata migration helpers
├── manuscript/                # Academic paper & System documentation (LaTeX)
│   ├── main.tex               # Manuscript source file documenting architecture
│   └── main.pdf               # Compiled system documentation/paper
├── app.py                     # Root FastAPI web application entry point
├── download_models.py         # Script to pre-download model weights locally
├── requirements.txt           # Python backend dependencies
└── CITATION.cff               # CITATION file for academic/repository reference

The indexing pipeline is designed for a mass-scale, distributed build.

To run the search engine immediately without running the entire indexing pipeline (Phases 1-6) from scratch, you can download our pre-computed database and FAISS index from the Zenodo Dataset:

1. Download snippets.db and mediawiki.index.
2. Place both files inside the backend/ directory of the project.

For the frozen software source code release of the engine, see GitHub Release v2.0.0.

Create and activate a virtual environment (optional but recommended), install dependencies, and pre-download the neural models:

python -m venv venv
# Windows:
.\venv\Scripts\activate
# Linux/macOS:
source venv/bin/activate

pip install -r requirements.txt
python download_models.py

The frontend is built with vanilla JavaScript and the Codex Design System. It consists of static HTML, CSS, and JS files located in the frontend/ directory. These files are served directly by the FastAPI backend.

There is no compilation step required for the frontend.

First, discover the ecosystem and mirror it for processing:

cd preprocessing
python list_repos.py      # Fetches 2,500+ repo URLs
python download_repos.py  # Shallow clones (approx. 8GB disk space)

Ensure all repositories are archived in Software Heritage for on-demand retrieval.

python archive_individual_to_swh.py

Perform high-precision structural parsing on your local machine. This captures functions/types with qualified names (e.g., Class::Method) and handles complex language features.

python extract_structural_entities.py

Resolve Git-compatible hashes to standard SHA1. You can do this either locally (fast) or via the Software Heritage API (official):

• Option A: Local Resolution (Recommended)

  python resolve_swh_hashes_local.py

• Option B: API-based Resolution

  python resolve_swh_hashes.py

Move raw_snippets.json to a GPU-equipped environment to compute neural vectors and build the FAISS index.

cd backend
python generate_embeddings.py  # Computes and saves embeddings to embeddings.npy
python build_index.py          # Trains and builds FAISS index from embeddings.npy

Before deploying, convert the production metadata to SQLite to stay within 6GiB RAM limits:

cd backend
python migrate_to_sqlite.py

Once the index and database are ready, start the FastAPI backend from the root directory:

# From the project root
uvicorn app:app --host 0.0.0.0 --port 8000

The server will be available at http://localhost:8000. You can access the automatic API documentation at http://localhost:8000/docs.

Follow these steps to deploy the application on Wikimedia Toolforge.

Since the model weights and indexes are large, they should be uploaded from your local machine to the Toolforge project data directory:

# From the project root
scp -rp "./models" supnabla@login.toolforge.org:/data/project/code2codesearch/
scp -rp "./backend/mediawiki.index" supnabla@login.toolforge.org:/data/project/code2codesearch/backend/
scp -rp "./backend/snippets.db" supnabla@login.toolforge.org:/data/project/code2codesearch/backend/

Log into Toolforge and set the necessary permissions:

ssh supnabla@login.toolforge.org

chmod -R a+rX /data/project/code2codesearch/models/
chmod a+r /data/project/code2codesearch/backend/snippets.db
chmod a+r /data/project/code2codesearch/backend/mediawiki.index

Now you are ready to deploy the webservice:

# Switch to the code2codesearch project
become code2codesearch

# Stop and clean existing build
toolforge webservice buildservice stop --mount=all
toolforge build clean -y

# Start build from repository
toolforge build start https://github.com/ftosoni/mediawiki-code2code-search

# Start webservice with 6GiB RAM
toolforge webservice buildservice start --mount=all -m 6Gi

# Monitor logs
toolforge webservice logs -f

Steps 1 and 2 are one-off. Step 3 is automated: every push to main — including the daily i18n sync from translatewiki.org — rebuilds the image and restarts the webservice, but only after the lint and test jobs pass (see .github/workflows/python-ci.yml).

Two repository secrets are required (Settings → Secrets and variables → Actions):

The workflow can also be run by hand from the Actions tab (Run workflow).

Apache 2.0 License. Created for advanced code-to-code retrieval within the Wikimedia developer ecosystem.