Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Add Airflow 2.x plugin mode support for MCP server#186

Merged
kaxil merged 3 commits intomainastronomer/agents:mainfrom
add-airflow-2-plugin-modeastronomer/agents:add-airflow-2-plugin-modeCopy head branch name to clipboard
Apr 16, 2026
Merged

Add Airflow 2.x plugin mode support for MCP server#186
kaxil merged 3 commits intomainastronomer/agents:mainfrom
add-airflow-2-plugin-modeastronomer/agents:add-airflow-2-plugin-modeCopy head branch name to clipboard

Conversation

@kaxil
Copy link
Copy Markdown
Contributor

@kaxil kaxil commented Apr 16, 2026

Summary

Extends the MCP server's plugin mode to work with Airflow 2.x, matching the existing Airflow 3 plugin support. The MCP server can now run embedded in an AF2 webserver process with endpoints at /mcp/v1/, rather than only as a standalone sidecar.

Design rationale

Why a Flask blueprint? AF2's webserver is Flask/WSGI, not FastAPI/ASGI like AF3. Plugins register via flask_blueprints instead of fastapi_apps. The blueprint lives in the same plugin.py as the AF3 integration and only activates when FastAPI isn't importable.

Why an ASGI bridge? FastMCP is ASGI-native. Rather than reimplement the MCP Streamable HTTP protocol in Flask, the plugin runs one asyncio event loop in a daemon thread and submits each Flask request to it via run_coroutine_threadsafe. The FastMCP lifespan is started once on the shared loop so the task group stays initialized.

Why lazy init (not at plugin load)? Gunicorn forks worker processes and threads don't survive the fork. The event loop + lifespan start on the first request in each worker, guarded by a threading lock.

Why a module-level dict for auth (not ContextVars)? ContextVars don't propagate across the thread boundary from the gunicorn worker into the background asyncio loop. A plain dict works because gunicorn sync workers handle one request at a time per worker. Both bearer tokens (Astro) and basic auth (local) are captured from the incoming request and read by the adapter when it makes internal calls.

Why lazy _get_plugin_url()? On Astro, webserver.base_url is populated by the runtime after plugin import. The deployment path prefix (e.g. /d99lgbz8) is required for internal calls to localhost, so the URL is constructed on first request rather than at module load.

Why exempt the blueprint from CSRF? MCP clients use bearer/basic auth, not session cookies. Flask-WTF's CSRF check would reject every POST otherwise. Handled in @bp.record_once.

Usage

Install into an Airflow 2.x environment using the plugin-v2 extra. The plugin auto-registers. Connect from an MCP client at https://<airflow>/mcp/v1/ with an Authorization header matching the webserver's auth backend (basic auth locally, bearer token on Astro).

Tested

  • Local Docker (Airflow 2.11.0, basic auth): get_airflow_version and list_dags returned 75 DAGs with auth correctly forwarded
  • Astro stage (Airflow 2.11.2, deployment JWT): 20/20 requests succeeded, tool calls return real data (example_astronauts)
  • Unit tests: 437 pass (10 plugin tests, up from 7)

Gotchas

  • On Astro dev deployments, the plugin is loaded per-worker. During rolling restarts there's a brief window where some requests may hit workers that haven't loaded the plugin yet; they'll see 404s until the rollout settles.
  • The FastMCP lifespan warms on first request, which adds ~100ms of latency to the first MCP call per worker process.

kaxil added 2 commits April 16, 2026 20:33
The MCP server previously supported Airflow 2 only in standalone mode.
This adds plugin mode support so the server can run inside Airflow 2's
webserver process, exposing MCP endpoints at /mcp/v1/ alongside the
existing Airflow 3 plugin.

Airflow 2 uses Flask (WSGI) instead of FastAPI (ASGI), so the plugin
registers a Flask blueprint and bridges requests to FastMCP's ASGI app
via a persistent asyncio event loop running in a daemon thread. The
lifespan is started once (lazy init on first request, to survive
gunicorn's fork) and each request submits to the shared loop.

Per-request auth is stored in a module-level dict rather than
ContextVars (which don't propagate across the thread boundary). Gunicorn
sync workers handle one request at a time per worker, so this is safe.
Both bearer tokens (Astro) and basic auth (local) are forwarded to
internal API calls.

The blueprint is exempt from CSRF since MCP clients use token-based auth
rather than session cookies.

On Astro, webserver.base_url is populated by the runtime after plugin
import, so the internal API URL is constructed lazily in _ensure_ready()
rather than at module load time. This ensures the deployment path
prefix (e.g. /d99lgbz8) is included when calling localhost.

Tested end-to-end on:
- Local Docker (Airflow 2.11.0, basic auth) - 75 DAGs listed
- Astro stage (Airflow 2.11.2, bearer token) - version + DAG tool calls
Updates the plugin mode section to cover both AF2 and AF3:
- Note that the plugin supports Airflow 2.4+ as well as 3.x
- Explain the two integration mechanisms (fastapi_apps for AF3,
  flask_blueprints for AF2 with an ASGI→WSGI bridge)
- Document the lazy base_url resolution for Astro AF2 deployments
- Add Airflow 2.x-specific notes (CSRF exemption, lazy lifespan,
  per-request auth forwarding for both bearer and basic auth)
- Update auth header guidance to mention AF2 basic auth option
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Airflow 2.x “plugin mode” support for the MCP server by embedding it into the Airflow 2 webserver via a Flask blueprint, while keeping the existing Airflow 3 FastAPI plugin integration.

Changes:

  • Implement Airflow 2.x Flask blueprint integration that forwards requests into the existing ASGI MCP app via a background asyncio loop.
  • Add a new plugin-v2 optional extra and update package/docs messaging to cover Airflow 2/3 plugin support.
  • Expand plugin-related unit tests.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
astro-airflow-mcp/src/astro_airflow_mcp/plugin.py Adds AF2 Flask blueprint + ASGI bridge, auth forwarding, and updated plugin metadata.
astro-airflow-mcp/tests/test_plugin.py Adds tests for flask_blueprints and new auth ContextVar symbols.
astro-airflow-mcp/pyproject.toml Updates description and introduces plugin-v2 extra.
astro-airflow-mcp/README.md Updates plugin-mode documentation to describe AF2 (Flask) and AF3 (FastAPI).
astro-airflow-mcp/uv.lock Lockfile update to expose plugin-v2 extra metadata.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread astro-airflow-mcp/README.md Outdated
Comment thread astro-airflow-mcp/src/astro_airflow_mcp/plugin.py
Comment thread astro-airflow-mcp/src/astro_airflow_mcp/plugin.py Outdated
Comment thread astro-airflow-mcp/src/astro_airflow_mcp/plugin.py
Comment thread astro-airflow-mcp/src/astro_airflow_mcp/plugin.py
Comment thread astro-airflow-mcp/src/astro_airflow_mcp/plugin.py Outdated
Comment thread astro-airflow-mcp/tests/test_plugin.py
Comment thread astro-airflow-mcp/README.md Outdated
Comment thread astro-airflow-mcp/src/astro_airflow_mcp/plugin.py
Comment thread astro-airflow-mcp/src/astro_airflow_mcp/plugin.py Outdated
- Detect Airflow major version via airflow.__version__ instead of
  relying on FastAPI importability. FastAPI can be present transitively
  in AF2 environments, which previously would have blocked the Flask
  blueprint from registering. Both plugin branches now explicitly gate
  on the detected major version.

- Replace module-level _v2_state dict with per-request ContextVars for
  auth forwarding. The handler passes the request's bearer/basic auth
  into the coroutine, which sets them on ContextVars scoped to that
  task's context. This prevents credentials from leaking to concurrent
  requests or subsequent requests without auth headers, and removes the
  dead _request_basic_auth ContextVar warning Copilot flagged.

- Verify ASGI lifespan startup actually completed before marking the
  loop ready. Previously we set ready=True even on timeout or
  lifespan.startup.failed, leaving the plugin in a broken state. Now
  we raise if startup doesn't complete within 10s or reports failure.

- Preserve response header list instead of collapsing to dict, so
  multi-value headers like Set-Cookie survive the ASGI→Flask hop.

- Add test for ContextVar isolation across request contexts.

- Update README to match actual behavior: version-based detection
  (not FastAPI-import detection) and FASTMCP_STATELESS_HTTP only
  applies to standalone mode (plugin mode always uses stateless).
Copy link
Copy Markdown
Contributor

@tayloramurphy tayloramurphy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm approving for docs only.

Copy link
Copy Markdown
Member

@schnie schnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@kaxil kaxil merged commit 5935c43 into main Apr 16, 2026
10 checks passed
@kaxil kaxil deleted the add-airflow-2-plugin-mode branch April 16, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.