A production-ready reference architecture for automated IT support ticket classification using Databricks Unity Catalog AI Functions, Vector Search, LangChain, and LangGraph.

• 5-Tab Progressive Architecture: From simple classification to sophisticated AI agents
• Dual AI Agent Approaches: Sequential orchestration + Adaptive LangGraph ReAct agent
• Unity Catalog AI Functions: Serverless AI with ai_classify, ai_extract, ai_gen
• Vector Search Integration: Semantic search over knowledge base documents
• Genie API Integration: Natural language querying of historical tickets
• Multi-Environment Support: Dev (fast iteration), Staging, Production
• Cost Optimized: Estimated <$0.002 per ticket at scale

The dashboard provides five progressively sophisticated approaches:

Tab 1: 🚀 Quick Classify          → Single UC function call (fastest, ~1s)
Tab 2: 📋 6-Phase Classification  → Traditional pipeline (educational)
Tab 3: 📊 Batch Processing        → High-volume CSV processing
Tab 4: 🤖 AI Agent Assistant      → Sequential multi-agent orchestration
Tab 5: 🧠 LangGraph ReAct Agent   → Adaptive intelligent agent (state-of-the-art)

4-Agent Sequential System for comprehensive ticket intelligence:

1. Agent 1: Classification - UC Function: ai_classify(ticket_text)

   • Returns: category, priority, assigned_team

2. Agent 2: Metadata Extraction - UC Function: ai_extract(ticket_text)

   • Returns: JSON with priority_score, urgency_level, affected_systems

3. Agent 3: Knowledge Search - Vector Search over knowledge base

   • Top 3 relevant documents using BGE embeddings

4. Agent 4: Historical Tickets - Genie Conversation API

   • Natural language query for similar resolved tickets
   • Shows resolution details, root causes, and resolution times

When to use: Guaranteed comprehensive analysis for every ticket, compliance-heavy scenarios.

Intelligent Tool Selection based on ticket complexity:

• Uses LangChain + LangGraph's ReAct (Reasoning + Acting) pattern
• Simple ticket (P3 password reset): Uses 2 tools → $0.0005, ~1-2s
• Complex issue (P1 database down): Uses all 4 tools → $0.0018, ~4-5s
• Cost savings: 40-60% on simple tickets while maintaining quality

When to use: High-volume environments where cost and speed optimization matter.

• Databricks Runtime: 16.4 LTS (Spark 3.5.2)
• Unity Catalog: AI Functions + Vector Search
• Genie API: Natural language SQL generation & execution
• Agent Framework: LangChain + LangGraph
• LLM: Claude Sonnet 4 (via Databricks Foundation Model API)
• Embedding Model: databricks-bge-large-en (free)
• Vector Search: Delta Sync with TRIGGERED mode
• Dashboard: Streamlit (local + Databricks Apps)
• Deployment: Databricks Asset Bundles (DAB)

• Databricks workspace (Azure, AWS, or GCP)
• Unity Catalog enabled
• Databricks CLI configured (~/.databrickscfg)
• Existing cluster for dev OR ability to create job clusters

1. Clone the repository

git clone https://github.com/bigdatavik/databricks-ai-ticket-vectorsearch.git
cd databricks-ai-ticket-vectorsearch

1. Configure Databricks CLI

# Check existing configuration
cat ~/.databrickscfg

# Should have a profile with:
# - host
# - token

# Or configure new profile
databricks configure --profile DEFAULT_azure

1. Update databricks.yml

# Edit databricks.yml - Update cluster ID for dev:
existing_cluster_id: YOUR_CLUSTER_ID

# Validate configuration
databricks bundle validate

# Deploy bundle (notebooks, app code, configs)
databricks bundle deploy

# Run infrastructure setup (creates catalog, tables, functions, vector search)
databricks bundle run setup_infrastructure

# App will auto-deploy as part of the bundle

Access your app at: https://[your-app-name].[workspace-id].azuredatabricksapps.com

Use databricks.staging_prod.yml for production deployments:

# Deploy to staging
databricks bundle deploy -t staging

# Run infrastructure
databricks bundle run setup_infrastructure -t staging

# Or deploy to prod
databricks bundle deploy -t prod
databricks bundle run setup_infrastructure -t prod

.
├── README.md                         # This file
├── databricks.yml                    # Dev config (interactive cluster)
├── databricks.staging_prod.yml       # Staging/Prod config (job clusters)
├── .gitignore                        # Git ignore rules
│
├── dashboard/                        # Streamlit application
│   ├── app_databricks.py            # Production app (Databricks Apps)
│   ├── app.yaml                     # Databricks App configuration
│   ├── requirements.txt             # Python dependencies
│   └── local_dev/                   # Local development setup
│       ├── app_simple.py            # Simplified local version
│       ├── README.md                # Local dev instructions
│       └── run_local.py             # Local runner script
│
├── notebooks/                        # Infrastructure setup notebooks
│   ├── 00_cleanup_full_mode.py      # Cleanup for full deployments
│   ├── 00_setup_catalog_schema.py   # Create catalog & schema
│   ├── 00_validate_environment.py   # Environment validation
│   ├── 01_deploy_uc_function_ai_classify.py
│   ├── 02_deploy_uc_function_ai_extract.py
│   ├── 03_deploy_uc_function_ai_gen.py
│   ├── 04_deploy_uc_function_quick_classify.py
│   ├── 06_prepare_sample_tickets.py
│   ├── 08_grant_app_permissions.py  # Grant service principal permissions
│   ├── 09_grant_genie_permissions.py # Grant Genie space access
│   ├── 10_upload_knowledge_docs.py  # Upload KB files to volume
│   ├── 13_reload_kb_with_proper_chunking.py  # Process KB with chunking
│   └── 14_recreate_vector_search_index.py
│
└── knowledge_base/                   # Knowledge base documents
    ├── IT_infrastructure_runbook.txt
    ├── application_support_guide.txt
    ├── security_incident_playbook.txt
    ├── user_access_policies.txt
    ├── ticket_classification_rules.txt
    ├── cloud_resources_guide.txt
    ├── email_system_troubleshooting.txt
    ├── database_admin_guide.txt
    ├── network_troubleshooting_guide.txt
    ├── monitoring_and_alerting_guide.txt
    ├── slack_collaboration_guide.txt
    └── storage_backup_guide.txt

Dev (databricks.yml):

• Uses existing interactive cluster
• Fast startup for rapid iteration
• Configure cluster ID in databricks.yml

Staging/Prod (databricks.staging_prod.yml):

• Job clusters (autoscaling)
• Runtime: 16.4 LTS
• Spot instances with fallback
• Photon enabled

Full Mode (dev default):

• Drops and recreates everything (except shared vector endpoint)
• Clean slate for testing major changes

Incremental Mode (staging/prod default):

• Updates only what changed
• Faster, safer for production

• Endpoint: one-env-shared-endpoint-2 (shared, never deleted)
• Sync Mode: TRIGGERED (manual, cost-effective)
• Embedding Model: databricks-bge-large-en (free)
• Index Type: Delta Sync

Basic ticket classification

Returns:

STRUCT<
  category STRING,
  priority STRING, 
  assigned_team STRING
>

Example:

SELECT ai_classify('My laptop screen is flickering')
-- Returns: {category: "Hardware", priority: "Medium", assigned_team: "Desktop Support"}

Extract structured metadata

Returns:

STRUCT<
  priority_score FLOAT,
  urgency_level STRING,
  affected_systems ARRAY<STRING>,
  assigned_team STRING
>

Generate context-aware summaries

Returns: STRING (summary with recommendations)

All-in-one classification (combines all phases)

Returns: Complete classification with all metadata

• Real-Time Classification: Instant ticket categorization
• 5 Progressive Tabs: Choose complexity level based on needs
• Vector Search Display: Top 3 relevant KB documents with similarity scores
• Sample Tickets: Pre-loaded test cases for quick testing
• Performance Metrics: Processing time, cost per ticket, phase breakdown
• AI Agent Reasoning: View LangGraph agent's decision-making process

The deployment automatically grants permissions to the app's service principal:

• USE CATALOG on target catalog
• USE SCHEMA on support_ai schema
• SELECT on all tables
• READ VOLUME on knowledge_docs
• EXECUTE on all UC functions
• Genie space access (if configured)

1. TRIGGERED Sync - Vector Search sync on-demand (vs CONTINUOUS)
2. Shared Endpoint - Reuse vector search endpoint across projects
3. Free Embeddings - databricks-bge-large-en (no cost)
4. Job Clusters - Autoscale + spot instances for staging/prod
5. Adaptive Agent - LangGraph agent uses fewer tools for simple tickets

Problem: Bundle validation errors

# Solution: Check databricks.yml syntax
databricks bundle validate

Problem: Cluster not found

# Solution: Update cluster ID in databricks.yml
existing_cluster_id: YOUR_CLUSTER_ID

Problem: App permissions not working

• Cause: App must be deployed before granting permissions
• Fix: Infrastructure job includes permission granting as final steps

Problem: 403 errors

• Cause: Service principal missing SELECT permission on index
• Fix: Permissions granted in 08_grant_app_permissions.py

Problem: Index not syncing

• Cause: Index not ONLINE yet
• Fix: Notebooks wait for ONLINE status before syncing

1. bind_tools() Pattern (Critical for reliability):

from langchain_community.chat_models import ChatDatabricks
from langgraph.prebuilt import create_react_agent

# Explicitly bind tools to LLM for consistent JSON format
llm_with_tools = ChatDatabricks(endpoint="claude-sonnet-4").bind_tools(tools)
agent = create_react_agent(llm_with_tools, tools)

2. Tool Input Schemas (Pydantic):

from pydantic import BaseModel, Field

class ClassifyInput(BaseModel):
    ticket_text: str = Field(description="The support ticket text to classify")

3. ReAct Loop:

1. Think: Analyze ticket complexity
2. Act: Call necessary tools
3. Observe: Review tool outputs
4. Decide: Determine if more tools needed
5. Respond: Provide final analysis

This is a production-ready reference architecture demonstrating:

• ✅ Complete end-to-end AI system on Databricks
• ✅ Five progressive approaches (simple → sophisticated)
• ✅ Modern AI agent patterns (LangChain + LangGraph)
• ✅ Cost-optimized serverless architecture
• ✅ Multi-environment deployment (dev/staging/prod)

Adapt this for:

• Customer service routing
• Email classification
• Document processing
• Incident management
• Any classification/routing workflow

Feel free to:

• Customize UC functions for your domain
• Add more knowledge base documents
• Extend the classification workflow
• Improve the dashboard UI

This system is production-ready with:

• ✅ Automated deployment via Databricks Asset Bundles
• ✅ Multi-environment support (dev/staging/prod)
• ✅ Cost optimization (estimated <$0.002/ticket)
• ✅ High accuracy (95%+ tested)
• ✅ Fast processing (<3 seconds)
• ✅ Secure (service principal + Unity Catalog governance)
• ✅ Scalable (autoscaling clusters, serverless functions)

• GitHub: https://github.com/bigdatavik/databricks-ai-ticket-vectorsearch
• LinkedIn: Connect for questions and discussions
• Databricks: Unity Catalog AI Functions

Built with ❤️ using Databricks Unity Catalog + LangChain + LangGraph