Add Airflow 2.x plugin mode support for MCP server#186
Merged
kaxil merged 3 commits intomainastronomer/agents:mainfrom Apr 16, 2026
add-airflow-2-plugin-modeastronomer/agents:add-airflow-2-plugin-modeCopy head branch name to clipboard
Merged
Add Airflow 2.x plugin mode support for MCP server#186kaxil merged 3 commits intomainastronomer/agents:mainfrom add-airflow-2-plugin-modeastronomer/agents:add-airflow-2-plugin-modeCopy head branch name to clipboard
kaxil merged 3 commits intomainastronomer/agents:mainfrom
add-airflow-2-plugin-modeastronomer/agents:add-airflow-2-plugin-modeCopy head branch name to clipboard
Conversation
The MCP server previously supported Airflow 2 only in standalone mode. This adds plugin mode support so the server can run inside Airflow 2's webserver process, exposing MCP endpoints at /mcp/v1/ alongside the existing Airflow 3 plugin. Airflow 2 uses Flask (WSGI) instead of FastAPI (ASGI), so the plugin registers a Flask blueprint and bridges requests to FastMCP's ASGI app via a persistent asyncio event loop running in a daemon thread. The lifespan is started once (lazy init on first request, to survive gunicorn's fork) and each request submits to the shared loop. Per-request auth is stored in a module-level dict rather than ContextVars (which don't propagate across the thread boundary). Gunicorn sync workers handle one request at a time per worker, so this is safe. Both bearer tokens (Astro) and basic auth (local) are forwarded to internal API calls. The blueprint is exempt from CSRF since MCP clients use token-based auth rather than session cookies. On Astro, webserver.base_url is populated by the runtime after plugin import, so the internal API URL is constructed lazily in _ensure_ready() rather than at module load time. This ensures the deployment path prefix (e.g. /d99lgbz8) is included when calling localhost. Tested end-to-end on: - Local Docker (Airflow 2.11.0, basic auth) - 75 DAGs listed - Astro stage (Airflow 2.11.2, bearer token) - version + DAG tool calls
Updates the plugin mode section to cover both AF2 and AF3: - Note that the plugin supports Airflow 2.4+ as well as 3.x - Explain the two integration mechanisms (fastapi_apps for AF3, flask_blueprints for AF2 with an ASGI→WSGI bridge) - Document the lazy base_url resolution for Astro AF2 deployments - Add Airflow 2.x-specific notes (CSRF exemption, lazy lifespan, per-request auth forwarding for both bearer and basic auth) - Update auth header guidance to mention AF2 basic auth option
There was a problem hiding this comment.
Pull request overview
Adds Airflow 2.x “plugin mode” support for the MCP server by embedding it into the Airflow 2 webserver via a Flask blueprint, while keeping the existing Airflow 3 FastAPI plugin integration.
Changes:
- Implement Airflow 2.x Flask blueprint integration that forwards requests into the existing ASGI MCP app via a background asyncio loop.
- Add a new
plugin-v2optional extra and update package/docs messaging to cover Airflow 2/3 plugin support. - Expand plugin-related unit tests.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
astro-airflow-mcp/src/astro_airflow_mcp/plugin.py |
Adds AF2 Flask blueprint + ASGI bridge, auth forwarding, and updated plugin metadata. |
astro-airflow-mcp/tests/test_plugin.py |
Adds tests for flask_blueprints and new auth ContextVar symbols. |
astro-airflow-mcp/pyproject.toml |
Updates description and introduces plugin-v2 extra. |
astro-airflow-mcp/README.md |
Updates plugin-mode documentation to describe AF2 (Flask) and AF3 (FastAPI). |
astro-airflow-mcp/uv.lock |
Lockfile update to expose plugin-v2 extra metadata. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Detect Airflow major version via airflow.__version__ instead of relying on FastAPI importability. FastAPI can be present transitively in AF2 environments, which previously would have blocked the Flask blueprint from registering. Both plugin branches now explicitly gate on the detected major version. - Replace module-level _v2_state dict with per-request ContextVars for auth forwarding. The handler passes the request's bearer/basic auth into the coroutine, which sets them on ContextVars scoped to that task's context. This prevents credentials from leaking to concurrent requests or subsequent requests without auth headers, and removes the dead _request_basic_auth ContextVar warning Copilot flagged. - Verify ASGI lifespan startup actually completed before marking the loop ready. Previously we set ready=True even on timeout or lifespan.startup.failed, leaving the plugin in a broken state. Now we raise if startup doesn't complete within 10s or reports failure. - Preserve response header list instead of collapsing to dict, so multi-value headers like Set-Cookie survive the ASGI→Flask hop. - Add test for ContextVar isolation across request contexts. - Update README to match actual behavior: version-based detection (not FastAPI-import detection) and FASTMCP_STATELESS_HTTP only applies to standalone mode (plugin mode always uses stateless).
tayloramurphy
approved these changes
Apr 16, 2026
Contributor
tayloramurphy
left a comment
There was a problem hiding this comment.
I'm approving for docs only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the MCP server's plugin mode to work with Airflow 2.x, matching the existing Airflow 3 plugin support. The MCP server can now run embedded in an AF2 webserver process with endpoints at
/mcp/v1/, rather than only as a standalone sidecar.Design rationale
Why a Flask blueprint? AF2's webserver is Flask/WSGI, not FastAPI/ASGI like AF3. Plugins register via
flask_blueprintsinstead offastapi_apps. The blueprint lives in the sameplugin.pyas the AF3 integration and only activates when FastAPI isn't importable.Why an ASGI bridge? FastMCP is ASGI-native. Rather than reimplement the MCP Streamable HTTP protocol in Flask, the plugin runs one asyncio event loop in a daemon thread and submits each Flask request to it via
run_coroutine_threadsafe. The FastMCP lifespan is started once on the shared loop so the task group stays initialized.Why lazy init (not at plugin load)? Gunicorn forks worker processes and threads don't survive the fork. The event loop + lifespan start on the first request in each worker, guarded by a threading lock.
Why a module-level dict for auth (not ContextVars)? ContextVars don't propagate across the thread boundary from the gunicorn worker into the background asyncio loop. A plain dict works because gunicorn sync workers handle one request at a time per worker. Both bearer tokens (Astro) and basic auth (local) are captured from the incoming request and read by the adapter when it makes internal calls.
Why lazy
_get_plugin_url()? On Astro,webserver.base_urlis populated by the runtime after plugin import. The deployment path prefix (e.g./d99lgbz8) is required for internal calls to localhost, so the URL is constructed on first request rather than at module load.Why exempt the blueprint from CSRF? MCP clients use bearer/basic auth, not session cookies. Flask-WTF's CSRF check would reject every POST otherwise. Handled in
@bp.record_once.Usage
Install into an Airflow 2.x environment using the
plugin-v2extra. The plugin auto-registers. Connect from an MCP client athttps://<airflow>/mcp/v1/with anAuthorizationheader matching the webserver's auth backend (basic auth locally, bearer token on Astro).Tested
get_airflow_versionandlist_dagsreturned 75 DAGs with auth correctly forwardedexample_astronauts)Gotchas