RIR-SQL-Forge is a Go utility for building a local IP ownership and abuse-contact directory from public RPSL bulk data. It downloads RIPE, APNIC, and AFRINIC databases automatically, parses the data as streams, resolves organisation/contact relationships, and publishes SQLite, DuckDB, Parquet, and CSV artifacts suitable for security operations, abuse handling, risk systems, and network analytics.

Documentation: this README. Automated datasets: GitHub Releases produced by .github/workflows/build-db.yml.

Regional Internet Registry bulk data is operationally useful, but the raw format is not query-friendly. Network objects do not usually contain direct abuse addresses; contacts are normalized into separate RPSL objects and linked by handles. RIR-SQL-Forge turns those raw registries into a local database that can be queried without live WHOIS lookups.

The compiler currently targets public RPSL data from:

By default, rir-sql-forge compile downloads public RIPE, APNIC, and AFRINIC data itself. Local/manual sources are additive and are skipped when not provided.

public RIR FTP/HTTPS
        |
        v
 streaming downloader  ---> temp source files
        |
        v
 RPSL stream parser     ---> networks / organisations / contacts
        |
        v
 SQLite bulk writer     ---> normalized tables
        |
        v
 flatten SQL            ---> SQLite / DuckDB / Parquet / CSV artifacts

RPSL relationship resolution follows registry references rather than text scraping:

network.org
  -> organisation.organisation
  -> organisation.abuse-c
  -> role/person.nic-hdl
  -> abuse-mailbox

When org is absent, the compiler falls back to APNIC IRT or direct operational contacts on the network object:

network.mnt-irt
  -> irt
  -> abuse-mailbox

network.admin-c or network.tech-c
  -> role/person.nic-hdl
  -> abuse-mailbox or e-mail

The parser reads RPSL paragraphs from io.Reader, supports gzip input, handles continuation lines, and emits typed records into the compiler pipeline. SQLite writes use prepared statements and transactions with a 50,000-row default batch size.

• Automatic public bulk downloads for RIPE, APNIC, and AFRINIC.
• Streaming RPSL parser for plain and gzip input.
• Normalization for inetnum, inet6num, route, and route6.
• Relational contact resolution across network, organisation, role, person, and APNIC IRT objects.
• SQLite output using modernc.org/sqlite as the compatibility baseline.
• DuckDB output for local analytics and scan-heavy workflows.
• Parquet output for lakehouse, object storage, and columnar ingestion pipelines.
• Flat CSV export for downstream systems that do not need typed columnar data.
• Optional local LACNIC RPSL input.
• Explicit ARIN XML flag with safe skip behavior.
• Weekly GitHub Actions build that publishes compressed database artifacts.
• Fixture-backed tests for parser behavior, downloader failure paths, join logic, CSV export, and workflow syntax.

git clone https://github.com/ipanalytics/rir-sql-forge.git
cd rir-sql-forge

go test ./...
go build ./cmd/rir-sql-forge

./rir-sql-forge compile --output dist --work-dir tmp/rir-sql-forge

The compile command downloads the public RIPE, APNIC, and AFRINIC datasets automatically unless --skip-download is set.

Generated files:

dist/net_owner_directory.sqlite
dist/net_owner_directory.duckdb
dist/net_owner_directory.parquet
dist/net_owner_directory.csv
dist/net_owner_directory.stats.json
dist/net_owner_directory.stats.md

go install github.com/ipanalytics/rir-sql-forge/cmd/rir-sql-forge@latest

go build -o rir-sql-forge ./cmd/rir-sql-forge

go test ./...
go test -race ./...
go vet ./...

rir-sql-forge compile \
  --output dist \
  --work-dir tmp/rir-sql-forge

This is the normal production path. The compiler downloads public RIPE, APNIC, and AFRINIC source files, parses them, and writes SQLite, DuckDB, Parquet, and CSV outputs.

rir-sql-forge compile \
  --skip-download \
  --source RIPE=internal/compiler/testdata/sample.db \
  --output dist

rir-sql-forge compile \
  --lacnic-db /data/rir/lacnic.db \
  --output dist

rir-sql-forge compile \
  --arin-xml /data/rir/arin.xml \
  --output dist

The current build logs a clear ARIN skip message rather than attempting credentialed ARIN automation.

rir-sql-forge compile --afrinic=false

GitHub Releases use tags in the form:

db-YYYY-MM-DD

The SQLite database contains normalized source tables and the flat table. DuckDB and Parquet contain the flat operational dataset.

SQLite schema:

CREATE TABLE networks (
    cidr TEXT,
    rir TEXT,
    org_id TEXT,
    contact_id TEXT
);

CREATE TABLE organisations (
    org_id TEXT PRIMARY KEY,
    org_name TEXT,
    country TEXT,
    abuse_contact_id TEXT
);

CREATE TABLE contacts (
    contact_id TEXT PRIMARY KEY,
    abuse_email TEXT,
    tech_email TEXT
);

The compiler then builds the flat operational table:

CREATE TABLE ip_owner_abuse_directory AS
SELECT
    n.cidr,
    n.rir,
    o.org_name,
    o.country,
    COALESCE(NULLIF(c1.abuse_email, ''), NULLIF(c2.abuse_email, ''), NULLIF(c2.tech_email, '')) AS contact_email
FROM networks n
LEFT JOIN organisations o ON n.org_id = o.org_id
LEFT JOIN contacts c1 ON o.abuse_contact_id = c1.contact_id
LEFT JOIN contacts c2 ON n.contact_id = c2.contact_id;

CSV columns:

Every compile writes a saturation report next to the data artifacts:

net_owner_directory.stats.json
net_owner_directory.stats.md

The report is also embedded into GitHub Release notes. It is meant to answer the operational question that matters for this dataset: how much of the registry space resolved to useful ownership/contact data.

Tracked metrics:

Low coverage is useful signal, not hidden failure. It usually means the upstream registry objects do not expose contacts, the reference points to ARIN/LACNIC material that was not provided, or the registry data is incomplete for that range.

This block is updated by the release workflow after a successful dataset build.

Low contact coverage usually means the upstream RPSL objects do not expose an abuse mailbox or the contact reference points to a restricted/manual registry source.

• Public registry downloads are network-bound and should run in CI or on a host with stable outbound access.
• Use --work-dir on persistent runners to control temporary source storage.
• The compiler removes and recreates the SQLite output file on each run.
• Tests do not require live registry access; they use local fixtures and fake HTTP transports.
• Release automation uses GITHUB_TOKEN and repository contents: write permission.
• For reproducible internal pipelines, mirror source files and run with --skip-download --source RIR=/path/to/file.

RIR-SQL-Forge focuses on turning bulk registry data into local query artifacts. It is designed for scheduled offline builds, not live WHOIS serving.

In scope:

• public RIPE/APNIC/AFRINIC RPSL ingestion
• local/manual source files
• SQLite, DuckDB, Parquet, and CSV outputs
• CI-driven release asset publishing
• parser and join correctness tests

Out of scope for the current build:

• credential management for restricted registry portals
• hosted lookup APIs
• real-time WHOIS querying
• full ARIN XML normalization

• abuse desk enrichment and routing
• fraud and risk scoring pipelines
• security telemetry enrichment
• network ownership analytics
• offline investigations where live WHOIS calls are slow or rate-limited
• scheduled internal datasets for SOC, CTI, and infrastructure teams

• Registry data quality varies by source and by object owner.
• Some networks do not resolve to an email address after RPSL joins.
• ARIN bulk XML support is represented by a local flag but not parsed in this version.
• DuckDB and Parquet contain the flat operational view; use SQLite when preserving normalized source joins matters.

.
├── cmd/rir-sql-forge/          # CLI entrypoint
├── internal/app/               # Command parsing and application wiring
├── internal/compiler/          # End-to-end compile orchestration
├── internal/artifacts/         # DuckDB and Parquet artifact writers
├── internal/db/                # SQLite schema, bulk inserts, flattening, CSV export
├── internal/downloader/        # Streaming HTTP downloads
├── internal/parser/            # RPSL stream parser and CIDR normalization
├── internal/sources/           # Public RIR source catalog
├── .github/workflows/          # Weekly release automation
└── site/                       # Repository visual assets

The repository includes .github/workflows/build-db.yml.

Schedule:

cron: '0 0 * * 0'

The workflow:

1. checks out the repository
2. installs Go
3. runs tests
4. builds rir-sql-forge
5. downloads and compiles public RIPE/APNIC/AFRINIC data
6. writes SQLite, DuckDB, Parquet, CSV, and saturation reports
7. compresses SQLite, DuckDB, and CSV outputs
8. embeds saturation stats into release notes
9. creates or updates a GitHub Release tagged db-YYYY-MM-DD

Manual runs are available through workflow_dispatch.

RIR-SQL-Forge is licensed under the Apache License 2.0.

RIR-SQL-Forge processes registry data as published by the relevant RIRs. Review each registry's terms before redistributing derived datasets outside your organisation.