Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

fix: harden warehouse SQL identifier quoting#7187

Open
rudder-devbox[bot] wants to merge 5 commits into
masterrudderlabs/rudder-server:masterfrom
harness/INT-6791-fix-harden-warehouse-sql-identi-362a1da7rudderlabs/rudder-server:harness/INT-6791-fix-harden-warehouse-sql-identi-362a1da7Copy head branch name to clipboard
Open

fix: harden warehouse SQL identifier quoting#7187
rudder-devbox[bot] wants to merge 5 commits into
masterrudderlabs/rudder-server:masterfrom
harness/INT-6791-fix-harden-warehouse-sql-identi-362a1da7rudderlabs/rudder-server:harness/INT-6791-fix-harden-warehouse-sql-identi-362a1da7Copy head branch name to clipboard

Conversation

@rudder-devbox

@rudder-devbox rudder-devbox Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Description

Summary

  • Introduce shared identifier-quoting helpers in warehouse/utils: dialect quoters (double-quote, backtick, bracket), qualified-name and comma-separated joiners, a composite-key splitter (QuoteCommaSeparatedIdentifiers), and a string-literal escaper (SQLStringLiteral). Every quoter escapes by doubling its delimiter — """, ```, ]]], '''.
  • Route all warehouse SQL construction through these helpers — Postgres, Redshift, Snowflake, ClickHouse, MSSQL, Azure Synapse, Delta Lake, and BigQuery — replacing raw and %q interpolation of schema, table, column, and staging-table names.
  • Extend coverage beyond DDL to partition keys, dedup/load/users queries, metadata literals, TestLoadTable, and BigQuery view/delete SQL.
  • Split composite partition keys (e.g. the discards row_id, column_name, table_name) into individually quoted columns instead of quoting the whole string as one identifier.
  • Replace interpolation in schemaExists with bind parameters.
  • Leave the datalake schema-repository untouched — it builds schemas via the AWS Glue API / in-memory and constructs no warehouse SQL.

Linear Ticket

INT-6791

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@rudder-devbox

rudder-devbox Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

👋 I'm rudder-devbox. I opened this PR from a Linear ticket.

Mention @devbox or @rudder-devbox and I'll act on it — on a top-level PR comment,
in a review summary, or on an inline comment on the diff. Without a mention I won't see it.

@achettyiitr

Copy link
Copy Markdown
Member

@rudder-devbox With the recent changes, can we update the postgres integration as well.

@rudder-devbox

rudder-devbox Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Updated the Postgres integration as well. The Postgres SQL-building paths now use the shared warehouse identifier helpers instead of direct pq.QuoteIdentifier, composite partition-key quoting goes through the shared comma-separated identifier helper, and the schema existence lookup now uses a bind parameter for the namespace.

Validation:

  • go test ./warehouse/integrations/postgres ./warehouse/utils
  • go test ./warehouse/integrations/postgres -run 'Test(ColumnsWithDataTypesQuotesIdentifiers|CreateTableQuotesSchemaAndTableIdentifiers)$'
  • git diff --check

Commit: e47314f

@rudder-devbox

rudder-devbox Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

I fixed the CI failure and pushed the change.

Root cause from the Snowflake integration log: LoadUserTables created the users staging table with a quoted mixed-case name like "RUDDER_STAGING_USERS_291ed6a224b24369b0eb46c0b527fa76", but the later deduplication MERGE read it as "TEST_TKZVPZVIFTFJ_1784497134".RUDDER_STAGING_USERS_291ed6a224b24369b0eb46c0b527fa76 without quoting the table identifier. Snowflake folded that unquoted identifier to uppercase and failed with 002003 (42S02): SQL compilation error: Object '...RUDDER_STAGING_USERS_291ED6A224B24369B0EB46C0B527FA76' does not exist or not authorized, and the test eventually timed out in TestIntegration/Event_flow/Upload_Job_with_Role.

Fix: quoted the Snowflake users MERGE target/source table identifiers with whutils.DoubleQuoteIdentifier, and fixed the adjacent append-mode temporary identifies table SQL to quote generated staging table identifiers instead of using Go %q.

Validation passed:

  • go test ./warehouse/integrations/snowflake
  • go test ./warehouse/utils
  • git diff --check

Commit: ca04eb1

@codecov

codecov Bot commented Jul 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.26308% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.07%. Comparing base (c3c283c) to head (c99e78f).
⚠️ Report is 16 commits behind head on master.

Files with missing lines Patch % Lines
warehouse/integrations/snowflake/snowflake.go 84.26% 14 Missing ⚠️
warehouse/integrations/redshift/redshift.go 96.99% 4 Missing ⚠️
...ehouse/integrations/azure-synapse/azure-synapse.go 97.91% 2 Missing ⚠️
warehouse/integrations/mssql/mssql.go 97.87% 2 Missing ⚠️
warehouse/integrations/bigquery/bigquery.go 95.45% 1 Missing ⚠️
warehouse/integrations/clickhouse/clickhouse.go 96.96% 1 Missing ⚠️
warehouse/integrations/deltalake/deltalake.go 99.03% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7187      +/-   ##
==========================================
- Coverage   80.13%   80.07%   -0.07%     
==========================================
  Files         595      596       +1     
  Lines       66151    66442     +291     
==========================================
+ Hits        53011    53201     +190     
- Misses      10021    10096      +75     
- Partials     3119     3145      +26     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@achettyiitr
achettyiitr marked this pull request as ready for review July 20, 2026 05:53
@krishna2020

krishna2020 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@achettyiitr shouldn't there be any updates in each warehouse test cases ?

🔒 Scanned for secrets using gitleaks 8.29.1
@achettyiitr

Copy link
Copy Markdown
Member

@achettyiitr shouldn't there be any updates in each warehouse test cases ?

Nothing changes per integrations; currently, in some places, we were not using quoted identifiers, so we added those. With or without quote identifiers, SQL would work. Everything is covered in integration tests.

Comment thread warehouse/integrations/bigquery/bigquery.go
Comment thread warehouse/integrations/clickhouse/clickhouse.go Outdated
Comment thread warehouse/integrations/redshift/redshift.go Outdated
@krishna2020

Copy link
Copy Markdown
Contributor

Remaining injection vector not covered by this pass — warehouse/integrations/deltalake/deltalake.go:571 (tableLocationQuery):

return fmt.Sprintf("LOCATION '%s/%s/%s'", externalLocation, d.Namespace, tableName)

d.Namespace / tableName are interpolated into a single-quoted string literal without escaping, so a value containing a ' breaks out of the literal. This is a string-literal context (needs warehouseutils.SQLStringLiteral on the segments), not identifier quoting — but since the PR hardened other literal contexts (schemaExists, fetchTables LIKE, etc.), leaving this one is an inconsistency worth closing or explicitly scoping out.

Low severity (namespace is config-controlled and it is a storage-path literal rather than DDL), but it is a genuine breakout path.

(Posted as a top-level comment because line 571 is outside this PR's diff, so GitHub could not anchor it inline.)

🔒 Scanned for secrets using gitleaks 8.29.1
@achettyiitr
achettyiitr force-pushed the harness/INT-6791-fix-harden-warehouse-sql-identi-362a1da7 branch from d2314fb to c99e78f Compare July 23, 2026 06:42
@krishna2020

Copy link
Copy Markdown
Contributor

@achettyiitr shouldn't there be any updates in each warehouse test cases ?

Nothing changes per integrations; currently, in some places, we were not using quoted identifiers, so we added those. With or without quote identifiers, SQL would work. Everything is covered in integration tests.

@achettyiitr shouldn't there be any updates in each warehouse test cases ?

Nothing changes per integrations; currently, in some places, we were not using quoted identifiers, so we added those. With or without quote identifiers, SQL would work. Everything is covered in integration tests.

Integration Tests are covering only the happy path and proving current changes is not causing a regression. But we need negative test cases with malicious SQL's for proving that SQL has been hardened. In the previous Postgres PR, unit tests did that - so good to add for all warehouses too ?

@rudder-devbox

rudder-devbox Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Sandbox became unreachable (its file API was unavailable too long).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.