Heimdall is a lightweight, pluggable data orchestration and job execution platform that abstracts complex data infrastructure from clients while offering a secure and consistent API for submitting and managing jobs.

Originally inspired by Netflix Genie, Heimdall extends the architecture to support:

• 🔌 Pluggable commands
• ⚙️ Job queuing
• 📡 Synchronous and asynchronous execution

• 🔁 Sync & Async Job Execution
• 🧩 Plugin-Based Execution Framework: Shell, Glue, Snowflake, Spark, Trino, DynamoDB, and Ping
• 📬 REST API for programmatic access
• 🌍 Web UI for visual management
• 🔐 Secure orchestration without credential leakage
• 🧠 Dynamic routing based on command / cluster criteria
• 📦 Configurable or self-registering clusters
• 🩺 Cluster health checks with per-plugin probes and configurable timeout

Heimdall includes a web interface running alongside the API.

• API: http://localhost:9090/api/v1
• Web UI: http://localhost:9090/ui

git clone git@github.com:patterninc/heimdall.git
cd heimdall

Ensure you have Docker or a compatible alternative installed.

docker compose up --build -d

This starts:

• The Heimdall server on port 9090
• The database and other dependencies

curl -X POST -H "X-Heimdall-User: test_user" -H "Content-Type: application/json" \
     -d '{
           "name": "ping-test",
           "version": "0.0.1",
           "context": {},
           "command_criteria": ["type:ping"],
           "cluster_criteria": ["type:localhost"]
         }' \
     http://127.0.0.1:9090/api/v1/job

Use the UI (http://127.0.0.1:9090/) or the following endpoints:

# Job status
GET /api/v1/job/<job_id>

# Job stdout
GET /api/v1/job/<job_id>/stdout

# Job stderr
GET /api/v1/job/<job_id>/stderr

Heimdall supports a growing set of pluggable command types:

Defines a reusable unit of work with associated tags and plugin logic.

An execution environment abstracted from its physical form. It can represent localhost, EMR, Kubernetes, a DB, a piece of your infrastructure that has context and a name, etc.

The orchestration request. It combines:

• Command criteria
• Cluster criteria
• Execution context

Heimdall dynamically selects the best command-cluster pair based on these criteria.

Initially, Commands and Clusters are configured via a static config file (see config.yml). Heimdall is evolving toward support for:

• Self-registering clusters
• Health-based routing
• API-based dynamic configuration

1. Commands: Must be active and match all tags in command_criteria.

2. Compatible Clusters: Found via the command’s own cluster_criteria.

3. Final Selection:

   • Filters clusters using the job’s cluster_criteria.
   • If multiple pairs match, one is selected randomly (a capability for custom "routing" is in works and will be represented as a plugin).
   • If no match, the job fails with a detailed error.

Heimdall removes the need for:

• Embedding credentials in user environments
• Direct user and services access to infrastructure

It centralizes execution logic, logging, and auditing—all accessible via API or UI.

Commands may also restrict invocation via allowed_callers — a list of anchored regex patterns matched against X-Heimdall-User. Omitted/empty means open; non-matching callers are rejected at submit.

Opt any cluster into health probing by setting health_check: true in its config. Each plugin performs a lightweight connectivity check against its backend (see the Health Check column in the plugins table above).

Configure the global probe timeout (default 30s):

health_check:
  timeout_seconds: 10

GET /api/v1/clusters/health returns 200 if all probes pass, 503 if any fail. status per check: ok, error, or unchecked. Failing or misconfigured health checks have no effect on job execution or startup.

Heimdall was created at Pattern, Inc by Stan Babourine, with contributions from Will Graham, Gaurav Warale and Josh Diaz.