Pull Request: AI Agent Framework for Galaxy

Summary

This PR introduces an AI-powered multi-agent framework for Galaxy. The focus is on establishing the framework and APIs - the agents themselves are functional but will continue to evolve, and of course we want to add more. This provides the foundation for intelligent assistance in error analysis, tool recommendations, custom tool creation, and training material discovery. This is NOT meant to be merged as-is -- we want to iterate on the framework and basic agents with a wider audience and this will likely be trimmed down prior to merge. The GTN agent works well and is useful, but I don't love the deployment mechanism and we want to use chatgxy-materials embeddings (likely) anyway, which can happen in a separate branch.

Key Points

• Framework-first approach: The core value here is the infrastructure - agent base classes, registry, service layer, API endpoints, and configuration system. Individual agents will improve over time.
• Dual API surface: Both conversational (/api/chat) and direct programmatic access (/api/ai/agents/*) endpoints
• Flexible configuration: Per-agent model/temperature settings with sensible defaults
• Model-agnostic: Works with OpenAI, Anthropic, Google, and local models (vLLM, Ollama)

What's Included

Framework Components

• lib/galaxy/agents/base.py - Base agent classes with pydantic-ai integration
• lib/galaxy/agents/registry.py - Dynamic agent registration and lookup
• lib/galaxy/managers/agents.py - Centralized AgentService for execution
• lib/galaxy/schema/agents.py - Pydantic API schemas

API Endpoints

• GET /api/ai/agents - List available agents
• POST /api/ai/agents/query - Unified query with auto-routing
• POST /api/ai/agents/error-analysis - Direct error analysis
• POST /api/ai/agents/tool-recommendation - Direct tool recommendations
• POST /api/ai/agents/custom-tool - Direct custom tool creation
• Enhanced /api/chat for ChatGXY

Initial Agents (5)

1. Router - Analyzes user queries and routes them to the appropriate specialist agent. Uses intent classification to determine whether someone needs help debugging, finding tools, learning, etc.
2. Error Analysis - Examines job failures, stderr output, and tool context to diagnose what went wrong and suggest fixes. Can recommend alternative tools if the current one isn't suitable.
3. Tool Recommendation - Helps users find the right Galaxy tool for their analysis task. Searches the toolbox and provides guidance on parameters and usage.
4. Custom Tool - Generates Galaxy tool XML/YAML definitions from natural language descriptions. Useful for wrapping command-line tools or scripts.
5. GTN Training Helper - Searches the Galaxy Training Network using full-text search to find relevant learning materials for any topic.

UI Integration

• ChatGXY as full-page activity in activity bar
• GalaxyWizard integration for error analysis
• Action suggestions with interactive buttons

Configuration

# Minimal - uses defaults (this is the pre-existing config options)
ai_api_key: ${OPENAI_API_KEY}
ai_model: gpt-4o-mini

# Or new per-agent configuration falling back to 'default'
inference_services:
  default:
    model: "llama-4-scout"
    api_key: "sk-local-test-master-key"
    api_base_url: "http://localhost:4000/v1/"
    temperature: 0.7
    max_tokens: 2000
  custom_tool:
    model: "anthropic:claude-sonnet-4-5"
    api_key: "sk-ant-api03-SUPERSECRETKEY"

What's NOT Included (Future Work)

• Dataset analyzer agent (Junhao has a branch based on this one using this framework to implement it)
• MCP (Model Context Protocol) integration - separated into galaxy-mcp branch
• Workflow generation from natural language
• Per-user API key management
• Additional specialized agents

Testing

• To make this easy, we plan to deploy on test and will allow folks to primarily exercise it that way.

• There is a start of a comprehensive pytest suite with mocked and live LLM modes

• mypy and lint checks passing

• Tested with claude (sonnet and haiku) and local LLMs via LiteLLM

# Run mocked tests
pytest lib/galaxy_test/api/test_agents.py -k Mocked -v

# Run live LLM tests
GALAXY_TEST_ENABLE_LIVE_LLM=1 pytest lib/galaxy_test/api/test_agents.py -k LiveLLM -v

Notes

• The GTN search database (gtn_search.db) needs to be built at deployment time - not included in repo (well, it is now, as John noticed, but it won't be)
• Agents gracefully handle models without structured output support