INFO 7375 — Prompt Engineering & Generative AI | Final Project
Aravind Balaji | Northeastern University, College of Engineering

A single prompt produces a book report. CodeSentinel conducts a peer review.

CodeSentinel is a LangGraph-based multi-agent system where specialized AI agents debate each other's findings, challenge false positives, and refuse to approve output until it meets a rigorous quality bar.

START → Security Sentinel → Code Quality Auditor → Evaluator/Guardian → END
              ↑____________________________|__________________________|
                               (on rejection, route back with precise feedback)

Security Sentinel — RAG-grounded vulnerability detection using OWASP Top 10 2025 + CWE taxonomy
Code Quality Auditor — Style, readability, error handling, and maintainability review
Evaluator/Guardian — False positive detection, completeness checking, cross-agent contradiction resolution

The inter-agent feedback loop is the key differentiator: the Evaluator can reject findings and route them back with specific feedback before anything reaches the user. Maximum 3 retries (circuit breaker).

git clone https://github.com/yourusername/codesentinel.git
cd codesentinel
pip install -r requirements.txt

cp .env.example .env
# Edit .env and add: ANTHROPIC_API_KEY=your-key-here
export ANTHROPIC_API_KEY=your-key-here

python -m rag.ingest

This loads OWASP Top 10 2025, CWE taxonomy, vulnerability patterns, and style guides into ChromaDB. Run once; re-run to refresh.

streamlit run app/streamlit_app.py

# Run full benchmark (multi-agent vs single-prompt baseline)
python -m eval.run_benchmark

# Run only multi-agent on all samples
python -m eval.run_benchmark --mode multi_agent

# Run single sample
python -m eval.run_benchmark --sample TOY-001

codesentinel/
├── app/
│   └── streamlit_app.py          # Streamlit UI (paste code or upload files)
├── graph/
│   ├── state.py                  # Shared LangGraph state (TypedDict)
│   ├── build_graph.py            # LangGraph wiring + conditional routing
│   ├── agents/
│   │   ├── security_sentinel.py  # OWASP/CWE vulnerability detection
│   │   ├── code_quality_auditor.py # Style and maintainability review
│   │   └── evaluator_guardian.py # False positive detection + feedback loop
│   └── prompts/
│       ├── security.md           # Security Sentinel system prompt
│       ├── quality.md            # Code Quality Auditor system prompt
│       └── evaluator.md          # Evaluator/Guardian system prompt
├── rag/
│   ├── ingest.py                 # RAG ingestion pipeline
│   ├── retriever.py              # ChromaDB retrieval with citations
│   └── data/
│       ├── owasp_top10_2025.txt  # OWASP knowledge base
│       ├── cwe_subset.csv        # CWE taxonomy subset
│       ├── patterns.md           # Language-specific vulnerability patterns
│       └── style_guides.md       # PEP 8, Google Style, Airbnb JS
├── eval/
│   ├── baseline_single_prompt.py # Single-prompt baseline system
│   ├── run_benchmark.py          # Evaluation framework (multi-agent vs baseline)
│   └── datasets/
│       └── toy_suite.json        # 10 ground-truth labeled samples
├── utils/
│   └── report_schema.py          # Final Markdown report builder
├── requirements.txt
├── .env.example
└── README.md

Why local embeddings? Decouples the RAG layer from the LLM provider, reduces cost, and enables reproducible offline development.

Every finding from the Security Sentinel is required to have:

• finding_id — SEC-001, SEC-002, etc.
• category — e.g., Injection
• cwe_id — e.g., CWE-89
• owasp_ref — e.g., A03:2025 — Injection
• severity — CRITICAL / HIGH / MEDIUM / LOW / INFO
• confidence — 0.0–1.0
• evidence — exact lines flagged
• fix — concrete, implementable remediation
• rag_source — which OWASP/CWE document informed this finding

Opaque "trust me" outputs are rejected by the Evaluator.

If the multi-agent architecture does not outperform a single prompt, this project does not hide that result.

The evaluation framework compares:

• Multi-agent CodeSentinel vs single-prompt baseline
• Metrics: True Positive Rate, False Positive Rate, CWE accuracy
• Ground truth: 10-sample labeled toy suite + OWASP Benchmark Project compatibility

An honest negative result is more valuable than a fabricated success. This is computational skepticism in practice.

The three-agent MVL (Security Sentinel + Code Quality Auditor + Evaluator) solves the core problem. The Performance Analyst and Synthesizer are planned future extensions.

Built for INFO 7375 — Prompt Engineering & Generative AI, Northeastern University, Spring 2026