A GRC Lakehouse on Databricks: Medallion, Rules, and Risk Prediction

The repo is here: ethanolivertroy/databricks-GRC-demo.

What we’re building

#

Four capabilities sitting on top of a single Unity Catalog catalog called grc_compliance_dev:

1. A normalized control catalog (NIST 800-53 Rev 5 as the authoritative 188-control spine) with mapping tables that project assessments onto SOC 2, ISO 27001, and PCI DSS.
2. Assessment and evidence tracking for a fictional 50-system environment with ~1,500 assessments and 200 evidence records.
3. A rule engine that flags the usual audit surprises before they become audit surprises: expired evidence, overdue remediations, stale assessments, critical-system coverage gaps.
4. A risk prediction model that scores each control 0 to 1 for near-term failure, tracked in MLflow.

They all roll up into one Databricks SQL dashboard.

The stack

#

• Databricks workspace with Unity Catalog for governance and Delta Lake for ACID writes on every layer.
• PySpark notebooks for Bronze > Silver > Gold transforms.
• MLflow for experiment tracking and model registry on the risk prediction side.
• scikit-learn Random Forest with SHAP for per-prediction explanations.
• Databricks SQL Dashboards for the Trust Center view.

Nothing exotic. The point is that GRC data behaves like any other analytical workload once you stop treating spreadsheets as the source of truth.

Medallion architecture

#

The pipeline is the standard Bronze/Silver/Gold progression. Landing volumes hold raw CSVs for the NIST catalog plus synthetic systems, assessments, and evidence. Bronze is the raw Delta landing zone with ingestion timestamps. Silver does cleanup, validation, and adds computed fields like compliance_score, is_overdue, and days_to_remediation. Gold is where the business tables live: per-system scorecards, cross-framework rollups, audit readiness metrics, and the risk predictions.

Bronze

#

Eight raw tables: nist_800_53_controls, control_mapping, systems_inventory, control_assessments, evidence, and the mapping CSVs for SOC 2, ISO, and PCI. Nothing here is interpreted. It’s just “what was uploaded, when.”

Silver

#

Silver is where validation lives: standardized control IDs, date sanity checks, enum constraints on status fields, and the first computed columns (compliance score per assessment, days since last evidence refresh, overdue flags). If a row doesn’t pass validation, it gets quarantined rather than propagated.

Gold

#

Gold has nine business-facing tables. The ones that do the most work:

• system_compliance_scorecard rolls up per-system posture across frameworks
• control_risk_predictions holds the ML output (risk_score, risk_level)
• audit_readiness_metrics is the number the executive dashboard reads from
• compliance_alerts is the rule-engine output

Cross-framework rollups

#

The mapping table control_mapping is where most of the value lives for organizations that answer to more than one framework. NIST 800-53 is the authoritative catalog; SOC 2, ISO 27001 Annex A, and PCI DSS v4 are mapped onto it. One assessment against a NIST control propagates to the mapped controls in every other framework it satisfies, so a SOC 2 and an ISO audit don’t re-test the same access control from scratch.

The rule engine

#

Five semantic rules live in code/05_Machine_Learning/compliance_rules.py. Each one reads from Silver and writes alerts to 03_gold.compliance_alerts. The thresholds are centralized in code/utils/config.py so you can tune without touching rule code.

1. Expired evidence. Any control where the most recent evidence is older than its refresh cadence. Default warning at 30 days past due.
2. Overdue remediation. Open findings past their remediation deadline, weighted by system criticality.
3. Stale assessments. Controls not re-assessed in more than nine months. Audit readiness tanks quietly when these pile up.
4. Critical-system coverage. High-criticality systems missing assessments on required control families.
5. Evidence gaps. Controls with an “implemented” status but zero attached evidence records.

Each rule writes alert rows with a consistent schema: control_id, system_id, rule_name, severity, detected_at, context. The dashboard and the ML features both read from this table, so the same signal feeds both the human view and the model.

Risk prediction

#

The model is a Random Forest trained on seven features:

• historical compliance score
• days to remediation (average over the last N findings)
• evidence count
• gap severity (aggregated from the rule engine)
• system criticality
• control family (NIST AC, AU, IA, etc.)
• most recent assessment recency

Output is a risk_score in [0, 1] and a bucketed risk_level of High / Medium / Low, written to 03_gold.control_risk_predictions. SHAP values go into the same table so the dashboard can show why a given control is flagged, not just that it is. Everything is MLflow-tracked: params, metrics, the fitted model, the SHAP explainer. Nothing lives as a pickle in someone’s home directory.

The honest framing: this isn’t predicting the future, it’s surfacing the controls where the existing signal (stale evidence, thin assessment history, high-criticality coverage holes) correlates with past failures. It’s a prioritization aid, not a crystal ball. Source is at code/05_Machine_Learning/risk_prediction.py.

Honest caveats

#

How to run it

#

1. Clone the repo into Databricks Workspace Repos.
2. Run code/00_Setup/setup_grc_lakehouse.py to create the catalog, schemas, and volumes.
3. Upload the data files into the Volumes (REST API or CLI, snippets in the repo README).
4. Run the pipeline in order: Bronze, Silver, Gold, then compliance_rules.py, then risk_prediction.py.
5. Import code/04_Consumption/Dashboard/GRC_Compliance_Trust_Center.lvdash.json into SQL Dashboards.

The dashboard has four pages: executive Trust Center, per-system scorecard, cross-framework mapping, and evidence tracking.

What’s next

#

A few things on the list that aren’t in the repo yet:

• Terraform for the workspace + catalog setup so this stops being a manual checklist.
• Integration tests that spin up a fresh catalog, run the whole pipeline, and assert on Gold row counts.
• Streaming evidence ingestion from a ticketing system so the refresh cadence isn’t tied to a nightly job.
• An agent layer on top. I went deeper on that side of the problem in Building a GRC Agent with the Claude Agent SDK; the lakehouse is the substrate that would feed it.

TODO: add a short section on the data dictionary once the docs/ pass is cleaned up.

Repo: github.com/ethanolivertroy/databricks-GRC-demo