DataQ measures the quality of an open data catalog encoded in RDF/Turtle (DCAT) across eight FAIR-aligned quality dimensions and produces a structured, machine-readable quality report in one command.

This repository holds two things that reinforce each other: the open-source reference implementation of the method, and a living survey of the whole research area, SURVEY.md, that maps the tools and papers on open data, linked data, and FAIR quality assessment and shows where DataQ fits.

Reference paper: Martinez-Gil, J. (2025). Framework to automatically determine the quality of open data catalogs. Expert Systems with Applications, 289, 128379. [journal] · [preprint] · [how to cite]

pip install -e .             # from this repository
dataq assess my_catalog.ttl  # full quality report in your terminal

1. The tool. Point DataQ at a DCAT/RDF catalog and it returns per-dimension scores plus a machine-readable report you can track over time, compare across portals, and cite in a paper.

2. The living survey. SURVEY.md is a maintained systematic review of automated quality assessment for open data catalogs, linked data, and FAIR resources. It gathers more than forty verified references into one taxonomy, compares every major tool in an honest feature table, and lists the open problems worth solving. If you are entering this area, start there. Corrections and additions are welcome (CONTRIBUTING.md).

Open data catalogs (data.gov, the EU Open Data Portal, regional and institutional portals) are public infrastructure, yet their metadata quality is rarely assessed in a systematic, automated, comparable way. Poor catalog quality, meaning missing fields, broken links, no license, stale or unreadable metadata, absent provenance, silently undermines discoverability, reuse, and trust.

DataQ turns "is this catalog any good?" into a reproducible number. Every metric is a short, readable module, so a reviewer can see exactly what a score means, and assessment is offline-deterministic, so the same catalog yields the same result on any machine.

Data quality is a precondition for the FAIR principles (Findable, Accessible, Interoperable, Reusable). A dataset nobody can find, that has no license, or whose provenance is unknown, is not reusable however good the underlying data is. Catalog-level quality is where findability and reuse are won or lost.

Each dimension traces to the data-quality literature. The mapping to source references is in SURVEY.md, Section 3.

The field has three communities that mostly work apart: open-data portal quality, linked-data and knowledge-graph quality, and FAIR assessment. Most tools score a single dataset; few score a whole catalog. The short table below places DataQ among representative tools. The full twelve-tool comparison, with references, is in SURVEY.md, Section 5.

DataQ occupies the open, reproducible, research-friendly niche: a place to prototype a new dimension, run a fair cross-portal comparison, or teach how catalog quality is measured. It complements the European MQA service and the dataset-level FAIR tools rather than replacing them.

Real numbers, recomputed live from the bundled data with python examples/run_benchmark.py. Nothing is hard-coded, so the table reproduces on any machine.

pip install -e .                  # core
pip install -e ".[similarity]"    # + NLTK tokenizer for the similarity dimension

Dependencies: rdflib, requests, textstat, pytz (lightweight, no GPU). textstat is pinned to 0.7.3 to compute Flesch-Kincaid grades offline; on setuptools >= 81 also run pip install "setuptools<81" (it provides pkg_resources).

One command:

dataq assess my_catalog.ttl                          # human-readable report
dataq assess my_catalog.ttl -f json -o report.json   # machine-readable
dataq assess https://example.org/catalog.ttl --check-links   # include live link checks

From Python:

from dataq import assess_catalog

report = assess_catalog("my_catalog.ttl")
print(report.to_text())          # pretty report
print(report.aggregate_score)    # convenience aggregate (see note below)
data = report.to_dict()          # full machine-readable result
for m in report.metrics:
    print(m.key, m.display_value())

Use one dimension at a time:

from dataq.io import load_graph
from dataq.metrics import completeness, provenance

g = load_graph("my_catalog.ttl")
print(completeness.completeness(g, "dcat"))   # -> 40.0
print(provenance.provenance(g))               # -> 16.7

Cross-catalog tools:

dataq similarity     a.ttl b.ttl
dataq compatibility  a.ttl b.ttl
dataq show           a.ttl

Note on the aggregate score. aggregate_score is a transparent, unweighted mean of the percentage-based dimensions (completeness, consistency, licensing, provenance, and accuracy when computed). It is a convenience indicator for ranking catalogs. It is not a weighted index defined in the source paper. Every dimension is always reported separately so you can apply your own weighting. How to weight and aggregate dimensions fairly is itself an open problem; see SURVEY.md, Section 6, and the AHP approach of Kubler et al. (2018).

{
  "source": "my_catalog.ttl",
  "dataq_version": "1.0.0",
  "n_triples": 97,
  "entities": { "Catalog": 1, "Dataset": 1, "Distribution": 1 },
  "aggregate_score": 27.04,
  "aggregated_dimensions": ["completeness_dcat", "consistency", "licensing", "provenance"],
  "metrics": [
    { "key": "completeness_dcat", "name": "Completeness (DCAT)", "value": 11.11,
      "unit": "percent", "higher_is_better": true, "skipped": false, "note": "" }
  ]
}

Stable keys make it easy to track a catalog over time or compare many portals. The schema maps cleanly onto the W3C Data Quality Vocabulary (DQV), the standard interchange format for quality scores; see SURVEY.md, Section 4.5.

The repository ships example catalogs (example001.ttl–example004.ttl) and real "Official catalogs" (Canada, Europeana, EASA, EuroMap, HADEA, Pacific Data). Reproduce the full comparison table with one command:

python examples/run_benchmark.py    # writes benchmark/results.csv and results.md

Every value is computed live from the bundled data, so the table is fully reproducible on any machine.

pip install -e ".[dev]"
pytest -q          # 213 tests

The suite includes parity tests asserting the packaged metrics return values numerically identical to the original standalone check_*.py scripts across every bundled catalog. This is the scientific-integrity guardrail that the refactor is a faithful reimplementation, not a re-derivation.

A new quality dimension is a small, self-contained module:

# dataq/metrics/my_dimension.py
from rdflib import Graph
from dataq.report import MetricResult

def my_dimension(graph: Graph) -> float:
    ...                      # compute a score from the RDF graph
    return score

def evaluate(graph: Graph) -> MetricResult:
    return MetricResult(key="my_dimension", name="My dimension",
                        value=my_dimension(graph), unit="percent",
                        higher_is_better=True)

Add its evaluate(graph) call to dataq/assess.py and it appears in every report, JSON export, and benchmark automatically. New dimensions that fill a gap named in SURVEY.md, Section 7, are especially welcome as contributions.

dataq/                 importable package
  io.py                unified RDF loading (file / URL)
  namespaces.py        shared DCAT / DCT / PROV namespaces
  report.py            MetricResult + QualityReport (JSON / Markdown / text)
  assess.py            one-call orchestrator -> QualityReport
  cli.py               `dataq` command-line interface
  metrics/             one module per quality dimension
examples/run_benchmark.py   reproducible benchmark over bundled catalogs
tests/                 pytest suite incl. parity tests
check_*.py             original standalone scripts (preserved, still runnable)
Official catalogs/     real-world DCAT catalogs for benchmarking
SURVEY.md              living survey of the research area
references.bib         curated, verified bibliography (40+ entries)
CONTRIBUTING.md        how to extend the tool and the survey

The original check_*.py scripts are kept for backward compatibility and as cited artifacts; they still run exactly as before (python check_completeness.py file.ttl).

SURVEY.md is the part of this repository meant for the wider community to read, cite, and extend. It contains:

• a lightweight systematic-review method (search strategy, inclusion criteria, verification);
• a unified taxonomy that reconciles quality dimensions across three research communities;
• a twelve-tool comparison table covering DataQ, the European MQA, F-UJI, FAIRshake, the FAIR Evaluator, FAIR-Checker, Luzzu, RDFUnit, SHACL, ShEx, Roomba, and Open Data Portal Watch;
• open problems and a research agenda, including the missing shared benchmark;
• a curated references.bib where every entry resolves to a DOI or an official standards URL.

If it helps your work, please cite the reference paper below, and open a pull request to keep it current.

open data quality · data catalog quality · metadata quality · FAIR assessment · FAIR evaluation tool · DCAT validation · linked data quality · knowledge graph quality · metadata benchmarking · open government data quality · RDF · data governance · provenance · semantic web · survey

If you use DataQ or the survey in your research, please cite the paper. The CITATION.cff file also exports this through GitHub's "Cite this repository" button.

@article{martinez2025datacatalogs,
  title   = {Framework to Automatically Determine the Quality of Open Data Catalogs},
  author  = {Jorge Martinez-Gil},
  journal = {Expert Systems with Applications},
  volume  = {289},
  pages   = {128379},
  year    = {2025},
  issn    = {0957-4174},
  doi     = {10.1016/j.eswa.2025.128379},
  url     = {https://www.sciencedirect.com/science/article/pii/S0957417425019980},
}

Martinez-Gil, J. (2025). Framework to automatically determine the quality of open data catalogs. Expert Systems with Applications, 289, 128379. https://doi.org/10.1016/j.eswa.2025.128379

MIT © Jorge Martinez-Gil.