Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit a444692

Browse filesBrowse files
YassinNouh21YassinNouh21
authored
feat: Add integration tests for dbt import (#5899)
* Add comprehensive integration tests for dbt import feature - Add 82 integration tests covering dbt manifest parsing, mapper, codegen, and CLI - Add real dbt project with DuckDB for testing - Add GitHub Actions workflow for dbt integration tests - Fix entity column handling: codegen now includes entity columns in schema to match mapper behavior (FeatureView.__init__ expects them) - Fix type mapping: Int32→ValueType.INT32, Float32→ValueType.FLOAT (instead of widening to INT64/DOUBLE) - Add multi-entity column support tests - Add comprehensive test coverage for edge cases, validation errors, special characters, and CLI commands Signed-off-by: yassinnouh21 <yassinnouh21@gmail.com> * Add ok-to-test label check to dbt integration workflow - Add PR trigger types (opened, synchronize, labeled) - Add label check condition requiring ok-to-test/approved/lgtm - Prevents resource spin-up for every PR Signed-off-by: yassinnouh21 <yassinnouh21@gmail.com> --------- Signed-off-by: yassinnouh21 <yassinnouh21@gmail.com>
1 parent 19f9bb8 commit a444692
Copy full SHA for a444692

18 files changed

+1,645-3Lines changed: 1645 additions & 3 deletions
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎.github/workflows/dbt-integration-tests.yml‎

Copy file name to clipboard
+53Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
name: dbt-integration-tests
2+
3+
# Run dbt integration tests on PRs
4+
on:
5+
pull_request:
6+
types:
7+
- opened
8+
- synchronize
9+
- labeled
10+
paths:
11+
- 'sdk/python/feast/dbt/**'
12+
- 'sdk/python/tests/integration/dbt/**'
13+
- 'sdk/python/tests/unit/dbt/**'
14+
- '.github/workflows/dbt-integration-tests.yml'
15+
16+
jobs:
17+
dbt-integration-test:
18+
if:
19+
((github.event.action == 'labeled' && (github.event.label.name == 'approved' || github.event.label.name == 'lgtm' || github.event.label.name == 'ok-to-test')) ||
20+
(github.event.action != 'labeled' && (contains(github.event.pull_request.labels.*.name, 'ok-to-test') || contains(github.event.pull_request.labels.*.name, 'approved') || contains(github.event.pull_request.labels.*.name, 'lgtm')))) &&
21+
github.event.pull_request.base.repo.full_name == 'feast-dev/feast'
22+
runs-on: ubuntu-latest
23+
strategy:
24+
matrix:
25+
python-version: ["3.11", "3.12"]
26+
env:
27+
PYTHON: ${{ matrix.python-version }}
28+
steps:
29+
- uses: actions/checkout@v4
30+
31+
- name: Setup Python
32+
uses: actions/setup-python@v5
33+
with:
34+
python-version: ${{ matrix.python-version }}
35+
architecture: x64
36+
37+
- name: Install the latest version of uv
38+
uses: astral-sh/setup-uv@v5
39+
with:
40+
enable-cache: true
41+
42+
- name: Install dependencies
43+
run: make install-python-dependencies-ci
44+
45+
- name: Install dbt and dbt-duckdb
46+
run: |
47+
uv pip install --system dbt-core dbt-duckdb
48+
49+
- name: Run dbt integration tests
50+
run: make test-python-integration-dbt
51+
52+
- name: Minimize uv cache
53+
run: uv cache prune --ci
Collapse file

‎Makefile‎

Copy file name to clipboardExpand all lines: Makefile
+22Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,28 @@ test-python-integration-container: ## Run Python integration tests using Docker
187187
python -m pytest -n 8 --integration sdk/python/tests \
188188
) || echo "This script uses Docker, and it isn't running - please start the Docker Daemon and try again!";
189189

190+
test-python-integration-dbt: ## Run dbt integration tests
191+
@echo "Running dbt integration tests..."
192+
@cd sdk/python/tests/integration/dbt/test_dbt_project && \
193+
echo "Installing dbt dependencies..." && \
194+
dbt deps && \
195+
echo "Building dbt models..." && \
196+
dbt build
197+
@cd sdk/python/tests/integration/dbt && \
198+
echo "Setting up Feast project..." && \
199+
mkdir -p feast_repo/data && \
200+
echo "project: feast_dbt_test\nregistry: data/registry.db\nprovider: local\nonline_store:\n type: sqlite\n path: data/online_store.db" > feast_repo/feature_store.yaml
201+
@cd sdk/python/tests/integration/dbt/feast_repo && \
202+
echo "Testing feast dbt import..." && \
203+
feast dbt import -m ../test_dbt_project/target/manifest.json -e driver_id -d file --tag feast && \
204+
echo "Verifying Feast objects..." && \
205+
feast feature-views list && \
206+
feast entities list
207+
@cd sdk/python && \
208+
echo "Running pytest integration tests..." && \
209+
python -m pytest tests/integration/dbt/test_dbt_integration.py -v --tb=short
210+
@echo "✓ dbt integration tests completed successfully!"
211+
190212
test-python-universal-spark: ## Run Python Spark integration tests
191213
PYTHONPATH='.' \
192214
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.spark_repo_configuration \
Collapse file

‎sdk/python/feast/dbt/codegen.py‎

Copy file name to clipboardExpand all lines: sdk/python/feast/dbt/codegen.py
+3-1Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -248,7 +248,9 @@ def generate(
248248
if not entity_cols:
249249
raise ValueError("At least one entity column must be specified")
250250

251-
excluded = set(entity_cols) | {self.timestamp_field}
251+
# Note: entity columns should NOT be excluded - FeatureView.__init__
252+
# expects entity columns to be in the schema and will extract them
253+
excluded = {self.timestamp_field}
252254
if exclude_columns:
253255
excluded.update(exclude_columns)
254256

Collapse file

‎sdk/python/feast/dbt/mapper.py‎

Copy file name to clipboardExpand all lines: sdk/python/feast/dbt/mapper.py
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@
2929
# Mapping from FeastType to ValueType for entity value inference
3030
FEAST_TYPE_TO_VALUE_TYPE: Dict[FeastType, ValueType] = {
3131
String: ValueType.STRING,
32-
Int32: ValueType.INT64,
32+
Int32: ValueType.INT32,
3333
Int64: ValueType.INT64,
34-
Float32: ValueType.DOUBLE,
34+
Float32: ValueType.FLOAT,
3535
Float64: ValueType.DOUBLE,
3636
Bool: ValueType.BOOL,
3737
Bytes: ValueType.BYTES,
Collapse file

‎sdk/python/tests/integration/dbt/__init__.py‎

Copy file name to clipboard
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Integration tests for dbt import functionality
Collapse file

‎sdk/python/tests/integration/dbt/conftest.py‎

Copy file name to clipboard
+32Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
"""
2+
Conftest for dbt integration tests.
3+
4+
This is a standalone conftest that doesn't depend on the main Feast test infrastructure.
5+
"""
6+
7+
from pathlib import Path
8+
9+
import pytest
10+
11+
# This conftest is minimal and doesn't import the main feast conftest
12+
# to avoid complex dependency chains for dbt-specific tests
13+
14+
# Path to the test dbt project manifest
15+
TEST_DBT_PROJECT_DIR = Path(__file__).parent / "test_dbt_project"
16+
TEST_MANIFEST_PATH = TEST_DBT_PROJECT_DIR / "target" / "manifest.json"
17+
18+
19+
def pytest_collection_modifyitems(config, items): # noqa: ARG001
20+
"""
21+
Skip dbt integration tests if manifest.json doesn't exist.
22+
23+
These tests require running 'dbt build' first to generate the manifest.
24+
The dbt-integration-test workflow handles this, but regular unit test
25+
runs don't, so we skip them to avoid failures.
26+
"""
27+
if not TEST_MANIFEST_PATH.exists():
28+
skip_marker = pytest.mark.skip(
29+
reason="dbt manifest.json not found - run 'dbt build' first or use dbt-integration-test workflow"
30+
)
31+
for item in items:
32+
item.add_marker(skip_marker)
Collapse file

‎sdk/python/tests/integration/dbt/pytest.ini‎

Copy file name to clipboard
+9Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[pytest]
2+
# Prevent loading parent conftest.py files which may import heavy dependencies
3+
# like ray that are not needed for dbt integration tests
4+
norecursedirs = ..
5+
asyncio_mode = auto
6+
7+
# Test markers
8+
markers =
9+
dbt: dbt integration tests

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.