A production-ready Reinforcement Learning from AI Feedback (RLAIF) system for stock and crypto trading that combines foundation models, multi-agent Claude LLM analysis, deep reinforcement learning, and an autonomous strategy experimentation loop inspired by autoresearch-mlx.

This system implements the frontier of AI-driven trading by integrating:

• Foundation Models: TimesFM 2.5 / TTM for time series prediction
• Multi-Agent LLM: Claude-powered specialized analysts (fundamental, sentiment, technical, risk)
• RAG System: FAISS-based retrieval for financial documents
• Deep RL: TD3/SAC ensemble for tactical execution
• RLAIF Loop: Market-outcome-based feedback to fine-tune LLM analysis
• AutoTrader: Autonomous strategy experimentation loop that generates, backtests, and hot-swaps Python signal functions
• Multi-Broker: Alpaca, Tradier, Binance (24/7 crypto), and paper trading
• Production Deployment: RunPod Serverless with comprehensive monitoring

Target Performance (based on Trading-R1 benchmarks):

• 8%+ cumulative returns
• 2.5-3.0 Sharpe ratio
• 65-70% hit rate
• <20% maximum drawdown

See ARCHITECTURE.md for detailed system design.

Data Ingestion → Feature Engineering → Foundation Models
                                             ↓
                        Multi-Agent LLM Analysis (Claude + RAG)
                                             ↓
                          RL Execution (TD3/SAC Ensemble)
                                             ↓
                            Market Outcomes
                                             ↓
                    RLAIF Feedback Loop → Improved LLM

MarketSentinel ──→ ThesisGenerator (Claude) ──→ generate_signals() code
                                                        ↓
                                              Sandboxed Backtest (5 min)
                                                        ↓
                                              Composite Metric Score
                                                        ↓
                                            Keep / Discard decision
                                                        ↓ (if kept)
                                              Hot-swap live strategy
                                                        ↓
                                              RLAIF preference pairs
                                                        ↓
                                              Reward model training
                                                        ↓
                                              LOOP FOREVER

• Python 3.11+
• macOS or Linux
• Anthropic API key for cloud LLM features (optional for smoke tests)
• Alpaca API key for live market data and brokerage features (optional for smoke tests)

1. Clone the repository

git clone https://github.com/yourusername/rlaif-trading.git
cd rlaif-trading

1. Run the setup script

./setup.sh --minimal

For a broader install including cloud LLM and monitoring dependencies:

./setup.sh --full

For Apple Silicon local MLX support too:

./setup.sh --full --apple-local

1. Activate the virtual environment

source .venv/bin/activate

1. Fill in .env if you want real APIs

cp .env.example .env  # only needed if setup script did not create it
$EDITOR .env

Check system status:

python -m src.cli status

Run pipeline smoke tests:

python -m pytest tests/test_pipeline.py -q -o addopts=''

Run the options quickstart demo:

python scripts/quickstart.py --symbol SPY

Run the example pipeline script:

python scripts/example_pipeline.py

Start the API server:

uvicorn deployment.api.main:app --host 0.0.0.0 --port 8000

Run a backtest through the CLI:

python -m src.cli backtest --start 2023-01-01 --end 2024-01-01 --symbols AAPL,MSFT

Start the autonomous strategy experimentation loop:

python -m src.cli autotrader start                # NEVER STOP loop
python -m src.cli autotrader start --mode hourly  # Once per hour
python -m src.cli autotrader run-once             # Single experiment
python -m src.cli autotrader status               # Dashboard in terminal
python -m src.cli autotrader history              # Experiment log
python -m src.cli autotrader dashboard            # Web UI at :8501

rlaif-trading/
├── ARCHITECTURE.md          # Detailed system architecture
├── README.md                # This file
├── pyproject.toml          # Project dependencies and configuration
├── .env.example            # Environment variable template
│
├── configs/                # Configuration files
│   ├── config.yaml        # Main configuration
│   └── universe.yaml      # Asset universe definitions
│
├── src/                   # Source code
│   ├── data/             # Data ingestion and processing
│   │   ├── ingestion/   # Data sources (Alpaca, news, SEC)
│   │   └── processing/  # Data cleaning and preparation
│   │
│   ├── features/         # Feature engineering
│   │   ├── technical.py # Technical indicators
│   │   ├── sentiment.py # Sentiment analysis (FinBERT)
│   │   ├── fundamental.py # Financial statement parsing
│   │   └── store.py     # Feature store (Feast)
│   │
│   ├── models/           # ML models
│   │   ├── foundation/  # TimesFM/TTM wrappers
│   │   └── rl/         # TD3/SAC agents
│   │
│   ├── agents/           # Multi-agent LLM system
│   │   ├── base_agent.py
│   │   ├── fundamental_analyst.py
│   │   ├── sentiment_analyst.py
│   │   ├── technical_analyst.py
│   │   ├── risk_analyst.py
│   │   ├── manager_agent.py
│   │   ├── rag_system.py
│   │   └── claude_client.py
│   │
│   ├── autotrader/       # Autonomous strategy experimentation (autoresearch pattern)
│   │   ├── orchestrator.py         # NEVER STOP loop
│   │   ├── thesis_generator.py     # Claude-powered signal code generation
│   │   ├── experiment_runner.py    # Sandboxed backtest execution
│   │   ├── sentinel.py             # Market event detection
│   │   ├── strategy_spec.py        # Serializable strategy with Python code
│   │   ├── strategy_swapper.py     # YOLO hot-swap
│   │   ├── safety.py               # AST validation, rate limits, kill switch
│   │   ├── metrics.py              # Composite score (Sharpe/return/DD/hit)
│   │   ├── experiment_log.py       # TSV results log (autoresearch pattern)
│   │   ├── rlaif_bridge.py         # Feeds experiments into RLAIF reward model
│   │   └── dashboard.py            # FastAPI real-time web dashboard
│   │
│   ├── rlaif/            # RLAIF feedback loop
│   │   ├── preference_generator.py
│   │   ├── reward_model.py
│   │   └── ppo_trainer.py
│   │
│   ├── environments/     # Trading environment
│   │   └── trading_env.py
│   │
│   ├── backtesting/      # Backtesting framework
│   │   ├── walk_forward.py
│   │   ├── metrics.py
│   │   └── risk_controls.py
│   │
│   └── deployment/       # Production deployment
│       ├── docker/      # Dockerfiles
│       ├── api/         # FastAPI application
│       └── monitoring/  # Monitoring configuration
│
├── scripts/              # Utility scripts
│   ├── download_data.py
│   ├── train.py
│   ├── backtest.py
│   └── run_rlaif.py
│
├── tests/               # Unit and integration tests
│
├── historical_data/     # Downloaded market data (gitignored)
├── logs/               # Application logs (gitignored)
└── models/             # Model checkpoints (gitignored)
    └── checkpoints/

The system is highly configurable via configs/config.yaml. Key sections:

data:
  assets: [AAPL, MSFT, GOOGL]
  sources:
    market_data:
      provider: alpaca
      bar_interval: 1Min

llm:
  model: claude-3-5-sonnet-20241022
  agents:
    fundamental_analyst:
      enabled: true
    sentiment_analyst:
      enabled: true
    # ... more agents

rl:
  algorithm: ensemble
  ensemble:
    agents:
      - type: td3
        weight: 0.3
      - type: sac
        weight: 0.2

rlaif:
  enabled: true
  iterations: 5
  preferences:
    pairs_per_iteration: 1000

See configs/config.yaml for all options.

Fine-tuned TimesFM 2.5 or TTM for financial time series:

• 25-50% improvement over baselines
• Uncertainty quantification
• Multi-timeframe predictions

Specialized Claude agents with RAG:

• FundamentalAnalyst: Financial statements, growth metrics
• SentimentAnalyst: News, social media, earnings calls
• TechnicalAnalyst: Charts, indicators, patterns
• RiskAnalyst: Volatility, correlations, position sizing
• Manager: Synthesis via structured debate

Ensemble of TD3/SAC agents:

• Augmented state space (50-80 dimensions)
• Multi-objective rewards (return, Sharpe, drawdown, turnover)
• Prioritized experience replay with n-step returns
• Risk controls (position limits, turbulence thresholds)

Learn from actual market outcomes:

1. Collect trading episodes with LLM reasoning
2. Generate preference pairs based on P&L
3. Train reward model on outcomes
4. Fine-tune Claude via PPO/DPO
5. Improved analysis quality → Better predictions

Inspired by autoresearch-mlx, the AutoTrader applies the "edit code, run, measure, keep/discard, repeat" pattern to trading strategies:

• Market Sentinel: Detects regime changes, vol spikes, drawdown breaches, and P&L deviations
• Thesis Generator: Claude writes complete Python generate_signals() functions
• Sandboxed Runner: Backtests in a subprocess with AST-validated code (blocks os, subprocess, eval)
• Composite Metric: Weighted Sharpe (35%) + return (30%) + drawdown (20%) + hit rate (15%)
• YOLO Hot-Swap: Immediately deploys winning strategies to live trading
• RLAIF Bridge: Experiment outcomes feed the reward model for a double feedback loop
• Safety: Rate limits (12 experiments/hr, 6 swaps/day), kill switch, crash tracking
• Dashboard: Real-time web UI with score evolution chart and experiment history

rlaif autotrader start    # Runs ~8-10 experiments/hour, forever

• Walk-forward analysis (no data snooping)
• Purged cross-validation (no label leakage)
• Realistic transaction costs (0.1-0.5% per trade)
• Multiple market regimes (bull, bear, volatile)
• Comprehensive metrics (Sharpe, Sortino, Calmar, CVaR)

• Docker containers (<5GB target)
• RunPod Serverless (T4/A100 GPUs)
• FastAPI with authentication
• Comprehensive monitoring (Evidently AI, Grafana)
• Drift detection and alerting

from src.train import RLAIFTrainer

trainer = RLAIFTrainer(
    config_path="configs/config.yaml",
    assets=["AAPL", "MSFT", "GOOGL"],
    start_date="2020-01-01",
    end_date="2024-01-01"
)

# Train foundation model
trainer.train_foundation_model()

# Train RL agents
trainer.train_rl_agents()

# Run RLAIF loop
trainer.run_rlaif_loop(iterations=5)

from src.backtesting import WalkForwardBacktest

backtest = WalkForwardBacktest(
    config_path="configs/config.yaml",
    train_window_months=24,
    test_window_months=6
)

results = backtest.run(
    start_date="2020-01-01",
    end_date="2024-01-01"
)

print(f"Sharpe Ratio: {results.metrics['sharpe']:.2f}")
print(f"Max Drawdown: {results.metrics['max_drawdown']:.2%}")

import requests

response = requests.post(
    "http://localhost:8000/predict",
    json={
        "asset": "AAPL",
        "horizon": "1D",
        "features": {...}
    },
    headers={"Authorization": "Bearer YOUR_TOKEN"}
)

prediction = response.json()
print(f"Signal: {prediction['signal']}")
print(f"Confidence: {prediction['confidence']}")
print(f"Reasoning: {prediction['reasoning']}")

pytest tests/ -v --cov=src

# Format code
black src/ tests/

# Lint code
ruff check src/ tests/

# Type checking
mypy src/

python -m cProfile -o output.prof scripts/train.py

docker build -t rlaif-trading:latest -f deployment/docker/Dockerfile .

# Configure RunPod API key
export RUNPOD_API_KEY=your_key_here

# Deploy
python scripts/deploy_runpod.py --gpu T4 --min-workers 2 --max-workers 5

# Start Grafana
docker-compose -f deployment/monitoring/docker-compose.yml up -d

# Access dashboard at http://localhost:3000

Based on backtesting from 2020-2024:

Results may vary based on assets, timeframe, and configuration.

• Project structure
• Data ingestion pipeline
• Basic feature engineering
• Baseline RL agent

• TimesFM integration
• TTM integration
• Fine-tuning pipeline
• Prediction API

• Claude API wrapper
• Multi-agent system
• RAG implementation
• Chain-of-Thought prompts

• TD3 implementation
• SAC implementation
• Ensemble coordination
• Multi-objective rewards

• Preference generation
• Reward model
• PPO fine-tuning
• Iterative refinement

• Docker optimization
• RunPod deployment
• Monitoring setup
• Paper trading validation

• Autonomous strategy experimentation loop (autoresearch pattern)
• Claude-powered signal function code generation
• Sandboxed backtest execution with AST validation
• YOLO hot-swap with audit trail
• RLAIF bridge (double feedback loop)
• Binance broker integration (24/7 crypto)
• Real-time web dashboard

• Automated retraining
• A/B testing
• Performance tracking
• Strategy refinement

Contributions are welcome! Please read CONTRIBUTING.md for guidelines.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

This project is licensed under the MIT License - see LICENSE for details.

This project builds on cutting-edge research:

• Trading-R1: Reverse reasoning distillation for RLAIF
• TimesFM: Google's foundation model for time series
• FinRL: Production-ready RL framework for finance
• Claude: Anthropic's LLM for multi-agent analysis

If you use this project in your research, please cite:

@software{rlaif_trading_2025,
  title = {RLAIF Trading Pipeline: Reinforcement Learning from AI Feedback for Stock Prediction},
  author = {RLAIF Trading Team},
  year = {2025},
  url = {https://github.com/yourusername/rlaif-trading}
}

This software is for educational and research purposes only.

• Past performance does not guarantee future results
• Trading involves substantial risk of loss
• Never invest more than you can afford to lose
• Consult a licensed financial advisor before making investment decisions
• The authors are not responsible for any financial losses

• Documentation: See docs/ folder
• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Email: your-email@example.com

• ARCHITECTURE.md - System architecture
• Research Document - Comprehensive research overview
• Configuration Guide - Detailed configuration options
• API Documentation - API reference
• Deployment Guide - Production deployment

Built with Claude Code | Powered by Anthropic Claude API | MIT License