Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,
or an AI coding agent—dlt can be dropped in anywhere.

dlt supports Python 3.10 through Python 3.14. Note that some optional extras are not yet available for Python 3.14, so support for this version is considered experimental.

pip install dlt

Add the extras you need for your sources and destinations, for example:

pip install "dlt[duckdb]"        # local DuckDB destination
pip install "dlt[bigquery]"      # or snowflake, postgres, redshift, databricks, athena, ...
pip install "dlt[s3]"            # or gs, az for cloud filesystems
pip install "dlt[sql_database]"  # read from any SQL database
pip install "dlt[hub]"           # data quality, transformations, and AI (see below)

Prefer uv? uv add "dlt[duckdb]".

Describe an API declaratively and load it into DuckDB — dlt handles requests, pagination, schema inference, and typing for you:

import dlt
from dlt.sources.rest_api import rest_api_source

# 1. Describe the API declaratively
source = rest_api_source({
    "client": {"base_url": "https://pokeapi.co/api/v2/"},
    "resources": [
        {"name": "pokemon", "endpoint": {"path": "pokemon", "params": {"limit": 1000}}},
    ],
})

# 2. Point a pipeline at any destination
pipeline = dlt.pipeline(
    pipeline_name="pokemon",
    destination="duckdb",
    dataset_name="pokemon_data",
)

# 3. Extract, normalize, and load
print(pipeline.run(source))

# 4. ...and read it straight back as a DataFrame
print(pipeline.dataset().pokemon.df())

...or load any Python iterable — a resource is just a generator, and dlt infers the schema, types the columns, and writes the table:

import dlt

@dlt.resource(table_name="players", primary_key="id", write_disposition="merge")
def players():
    yield {"id": 1, "name": "Magnus", "rating": 2839}
    yield {"id": 2, "name": "Pragg", "rating": 2758}

dlt.pipeline(destination="duckdb", dataset_name="chess").run(players())

Check out a super simple demo in Colab or a more advanced Hugging Face demo with Marimo notebooks.

dlt loads data from messy, often unstructured sources into well-structured, typed datasets. It's a library, not a platform — you pip install it into your existing code and keep your workflow and the other tools you already use. No black boxes: clean Pythonic interfaces, human-readable file formats, schemas you can inspect, no hidden side effects.

dlt and its docs are built from the ground up for LLMs and coding agents. Pair the typed, declarative primitives below with dlthub.com/context and the LLM-native workflow to go from prompt to working pipeline — across 5000+ sources — often in a single shot.

REST APIs — describe the endpoints declaratively; filter, map, and flatten records right at the source (docs):

from dlt.sources.rest_api import rest_api_source

source = rest_api_source({
    "client": {
        "base_url": "https://api.example.com/v1",
        "paginator": {"type": "cursor", "cursor_path": "next_cursor"},
    },
    "resources": [
        {
            "name": "guests",
            "endpoint": {"path": "events/guests"},
            "processing_steps": [
                {"filter": lambda r: r["approval_status"] == "approved"},
                {"map": lambda r: {**r, "email": r["email"].lower()}},
            ],
        },
    ],
})

SQL databases — reflect tables and types straight from the database (docs):

from dlt.sources.sql_database import sql_database

source = sql_database("mysql+pymysql://user:pass@host/db")

Files in any bucket — list, then parse CSV / JSONL / Parquet from local disk, S3, GCS, or Azure (docs):

from dlt.sources.filesystem import filesystem, read_csv_duckdb

source = (
    filesystem(bucket_url="s3://my-bucket/data", file_glob="*.csv")
    | read_csv_duckdb()
).with_name("events")

DataFrames & Arrow — pandas, Polars, and Arrow tables load directly; Arrow-backed frames move with zero copies:

import dlt
import pandas as pd

df = pd.DataFrame({"event": ["dlt summit", "DuckCon"], "signups": [1240, 860]})
dlt.pipeline(destination="duckdb", dataset_name="events").run(df, table_name="events")

See many more sources in the ecosystem.

The same resource runs anywhere. Change the destination string and dlt takes care of credentials, DDL in the target dialect, staging, and schema drift:

pipeline = dlt.pipeline(
    pipeline_name="luma",
    destination="duckdb",       # → snowflake, bigquery, postgres, redshift, databricks,
    dataset_name="luma_data",   #   athena, clickhouse, motherduck, filesystem (S3/GCS/Azure),
)                               #   iceberg, delta, ... and custom reverse-ETL destinations
pipeline.run(source)

dlt handles the parts you'd rather not:

• Credentials → secrets.toml / env vars, injected automatically
• DDL → CREATE TABLE in the target's dialect
• Type mapping → source types converted to the destination's types
• Staging → S3 / GCS for warehouses that need it
• Schema drift → ALTER TABLE on the fly

Browse all supported destinations, or build a custom one.

Decorators let you declare what you want — incremental loading, merge strategies, schema contracts, column hints — instead of hand-rolling it. Every knob can be overridden at runtime (docs):

import dlt

@dlt.resource(
    primary_key="id",
    write_disposition="merge",                       # upsert on the primary key
    columns={"email": {"x-annotation-pii": True}},   # type and annotate columns
    schema_contract={"columns": "freeze"},           # reject unexpected columns
)
def events(
    updated_at=dlt.sources.incremental("updated_at"),  # load only new/changed rows
):
    yield from fetch_events(since=updated_at.last_value)


@dlt.source
def luma(api_key: str = dlt.secrets.value):
    return events(), guests()   # group one or more resources behind shared config/auth

Schema contracts enforce the shape at the gate, with three modes — evolve (accept and adapt the schema), freeze (reject the record), and discard (drop the offending row/column) — applied independently to tables, columns, and data_type. You also get schema inference, normalization of nested data, incremental loading, and secrets & config injection out of the box.

A pipeline is durable. Reconnect to one by name with dlt.attach and read any table back in the shape that fits your tool (docs):

import dlt

pipeline = dlt.attach(pipeline_name="luma", destination="duckdb", dataset_name="luma_data")

dataset = pipeline.dataset()
dataset.tables               # ['events', 'guests', ...]

guests = dataset.guests      # a lazy dlt.Relation
guests.df()                  # pandas DataFrame
guests.arrow()               # pyarrow.Table (zero-copy)
guests.to_ibis()             # ibis expression — lazy, composable

Lift any loaded table into an Ibis expression, compose group-bys, joins, and window functions in Python, and let dlt compile it to SQL in the destination's dialect. Nothing runs until you ask for the result:

import ibis

guests = pipeline.dataset().guests.to_ibis()

guests_by_event = (
    guests
    .group_by("event_id")
    .aggregate(n_guests=ibis._.api_id.count())
)

guests_by_event.to_pyarrow()   # compiles to SQL and runs on the destination

dlt also supports Python and SQL data access, transformations, pipeline inspection, and visualizing data in Marimo notebooks.

For detailed usage and configuration, please refer to the official documentation.

You can find examples for various use cases in the examples folder, or in the code examples section of our docs page.

dlt follows the semantic versioning with the MAJOR.MINOR.PATCH pattern.

• major means breaking changes and removed deprecations
• minor new features, sometimes automatic migrations
• patch bug fixes

We suggest that you allow only patch level updates automatically using the Compatible Release Specifier. For example dlt~=1.23.0 allows only versions >=1.23.0 and less than <1.24.0

Please also see our release notes for notable changes between versions.

The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:

• Connect with the Community: Join other dlt users and contributors on our Slack
• Report issues and suggest features: Please use the GitHub Issues to report bugs or suggest new features. Before creating a new issue, make sure to search the tracker for possible duplicates and add a comment if you find one.
• Track progress of our work and our plans: Please check out our public Github project
• Improve documentation: Help us enhance the dlt documentation.

Please read CONTRIBUTING before you make a PR.

• 📣 New destinations are unlikely to be merged due to high maintenance cost (but we are happy to improve SQLAlchemy destination to handle more dialects)
• Significant changes require tests and docs and in many cases writing tests will be more laborious than writing code
• Bugfixes and improvements are welcome! You'll get help with writing tests and docs + a decent review.

Blacksmith is a drop-in replacement for GitHub-hosted runners that speed up our CI/CD pipelines by 2x and up to 75% cheaper. We're grateful to Blacksmith for sponsoring us with free CI/CD minutes--which helps us keep builds fast and our costs lower.

dlt is released under the Apache 2.0 License.