Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Filesystem: Add read support for more fsspec-based filesystems#254

Open
amotl wants to merge 6 commits into
mainpanodata/omniload:mainfrom
filesystems-read-202607bpanodata/omniload:filesystems-read-202607bCopy head branch name to clipboard
Open

Filesystem: Add read support for more fsspec-based filesystems#254
amotl wants to merge 6 commits into
mainpanodata/omniload:mainfrom
filesystems-read-202607bpanodata/omniload:filesystems-read-202607bCopy head branch name to clipboard

Conversation

@amotl

@amotl amotl commented Jul 25, 2026

Copy link
Copy Markdown
Member

About

  • A few more filesystem wrappers based on fsspec: Databricks files, Dropbox, FTP, Google Drive, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV, WebHDFS.
  • Filesystem documentation improvements across the board.

Documentation

Preview: https://omniload--254.org.readthedocs.build/supported-sources/filesystem.html#supported-filesystems

References

@coderabbitai

coderabbitai Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Summary by CodeRabbit

  • New Features

    • Added file reading support for Databricks, Dropbox, FTP, Google Drive, HDFS, OCI, OSS, R2, SharePoint, OneDrive, SMB, WebDAV, and WebHDFS.
    • Added Azure Blob Storage and ADLS Gen2 guidance under a unified Azure Storage experience.
    • Added broader support for remote filesystem locations and improved S3-compatible storage handling.
  • Documentation

    • Added setup, authentication, URI, and ingestion examples for newly supported sources.
    • Improved filesystem capability tables and file-format source references.
  • Bug Fixes

    • Improved endpoint parsing and validation messages for remote sources.

Walkthrough

Changes

Filesystem source expansion

Layer / File(s) Summary
Shared locator and resource construction
src/dlt_filesystem/source/{base,core,model,router}.py, src/dlt_filesystem/util/*
Adds normalized URL parsing, filesystem locators, reader-resource inference, endpoint namespaces, and option-casting helpers.
Remote connector implementations
src/dlt_filesystem/source/fsspec/*, src/dlt_filesystem/source/impl/remote.py
Adds integrations for multiple fsspec-backed systems and refactors shared S3-compatible, Azure, GCS, and SFTP handling.
Dispatch, dependencies, and validation
src/omniload/core/*, pyproject.toml, tests/main/filesystem/*, tests/dlt_filesystem/*
Registers new schemes, reorganizes filesystem extras, and adds URI, validation, incremental, and utility tests.
Supported-source documentation
docs/supported-sources/*, docs/changelog.md, docs/conf.py
Documents new remote sources and revises filesystem, Azure, S3, SFTP, and cross-reference guidance.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant SourceDestinationFactory
  participant FilesystemSource
  participant fsspecFilesystem
  participant DLTResource
  CLI->>SourceDestinationFactory: resolve source URI scheme
  SourceDestinationFactory->>FilesystemSource: create dlt_source(uri, table)
  FilesystemSource->>fsspecFilesystem: instantiate connector with decoded options
  FilesystemSource->>DLTResource: infer filesystem and reader resources
  DLTResource-->>CLI: return composed source resource
Loading

Possibly related PRs

Suggested reviewers: hampsterx

Poem

I’m a rabbit with sources to hop,
From Dropbox to WebDAV, no stop.
New paths bloom bright,
Tests guard them tight,
While DuckDB tables gently pop.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 28.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: adding read support for additional fsspec-based filesystems.
Description check ✅ Passed The description is clearly related and correctly mentions the new filesystem wrappers and documentation improvements.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch filesystems-read-202607b

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@amotl amotl added the coderabbit-review Tell CodeRabbit to review the patch. label Jul 25, 2026
@amotl
amotl marked this pull request as ready for review July 25, 2026 17:32
@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.48872% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.51%. Comparing base (7b24bb8) to head (3b9b008).

Files with missing lines Patch % Lines
src/dlt_filesystem/util/fsspec.py 76.92% 9 Missing ⚠️
src/dlt_filesystem/source/fsspec/webdav.py 87.87% 4 Missing ⚠️
src/dlt_filesystem/source/impl/remote.py 92.68% 3 Missing ⚠️
src/dlt_filesystem/source/model.py 97.14% 2 Missing ⚠️
src/dlt_filesystem/source/router.py 84.61% 2 Missing ⚠️
src/dlt_filesystem/source/fsspec/ftp.py 95.83% 1 Missing ⚠️
src/dlt_filesystem/source/fsspec/hdfs.py 96.15% 1 Missing ⚠️
src/dlt_filesystem/util/python.py 97.43% 1 Missing ⚠️
src/omniload/core/factory.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #254      +/-   ##
==========================================
+ Coverage   55.71%   57.51%   +1.80%     
==========================================
  Files         210      227      +17     
  Lines        9894    10448     +554     
==========================================
+ Hits         5512     6009     +497     
- Misses       4382     4439      +57     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 19

🧹 Nitpick comments (5)
tests/main/filesystem/test_remote_read.py (1)

52-56: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy lift

Resolve the FTP mock limitation and restore both coverage paths.

FTP is the only connector with these URI/error variants disabled, leaving both table-splitting and unsupported-format behavior unvalidated.

  • tests/main/filesystem/test_remote_read.py#L52-L56: restore the URI-with-separate-table FTP case.
  • tests/main/filesystem/test_remote_read.py#L107-L111: restore the FTP unknown-extension failure case.

As per path instructions, “Prefer flagging missing edge-case coverage over style nits.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/main/filesystem/test_remote_read.py` around lines 52 - 56, Resolve the
FTP mocking limitation in tests/main/filesystem/test_remote_read.py by updating
the shared FTP mock setup so both disabled coverage paths work, then restore the
URI-with-separate-table Item case at lines 52-56 and the FTP unknown-extension
failure case at lines 107-111. Preserve the existing test structure and
assertions for the other connectors.

Source: Path instructions

src/dlt_filesystem/util/python.py (3)

92-102: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

asbool duplicates bool-parsing already in _cast_value, with a different truthy/falsey vocabulary.

_cast_value's bool branch recognizes _TRUE_VALUES/_FALSE_VALUES (true/1/yes/on), while asbool additionally accepts y/t/n/f. Two independent bool-parsers with diverging accepted tokens is a minor DRY/consistency smell; consider consolidating if both are meant to serve the same "truthy string" concept.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/dlt_filesystem/util/python.py` around lines 92 - 102, Consolidate the
boolean parsing in asbool and _cast_value so both use the same established
_TRUE_VALUES and _FALSE_VALUES vocabulary. Remove the duplicate token lists and
preserve the existing invalid-string ValueError behavior while keeping
non-string values converted via bool.

145-149: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

apply_alias silently overwrites on key conflict.

If both name and effective_name are present in data, apply_alias silently discards effective_name's original value in favor of name's, with no warning. A user supplying both the alias and the canonical key (e.g. block_size and default_block_size) would get surprising, unannounced precedence.

♻️ Suggested guard
 def apply_alias(data: Dict[str, Any], name: str, effective_name: str) -> Dict[str, Any]:
     """Apply aliasing to dictionary keys."""
-    if name in data:
+    if name in data and effective_name in data:
+        raise ValueError(f"Both '{name}' and '{effective_name}' were provided; use only one")
+    if name in data:
         data[effective_name] = data.pop(name)
     return data
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/dlt_filesystem/util/python.py` around lines 145 - 149, Update apply_alias
to detect when both name and effective_name already exist in data before moving
the alias, and raise the established conflict error or otherwise reject the
operation instead of overwriting effective_name’s value. Preserve the current
rename behavior when only name is present and return unchanged data when name is
absent.

129-142: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

cast_to_dict/cast_to_list assume string input.

Both call json.loads(data[field_name]) unconditionally. If a caller ever passes an already-parsed dict/list for one of these fields (rather than a JSON-encoded string), this raises a TypeError instead of passing the value through. Current call sites feed these from URL-decoded strings, so this is defensive rather than an active bug, but a guard would make the helpers robust for future non-URL callers.

🛡️ Suggested guard
 def cast_to_dict(data: Dict[str, Any], names: List[str]) -> Dict[str, Any]:
     """Cast dictionary values from JSON."""
     for field_name in names:
-        if field_name in data:
+        if field_name in data and isinstance(data[field_name], str):
             data[field_name] = json.loads(data[field_name])
     return data
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/dlt_filesystem/util/python.py` around lines 129 - 142, Update
cast_to_dict and cast_to_list to call json.loads only for string-encoded field
values, passing already-parsed dict or list values through unchanged. Preserve
the current conversion behavior for JSON strings and return the modified data as
before.
src/dlt_filesystem/source/router.py (1)

177-196: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider narrowing the blind except Exception.

Static analysis flags the bare except Exception (Line 188) used to gate the table→path fallback. The two-stage retry logic is fine, but catching everything (including bugs like AttributeError/TypeError) makes failures harder to diagnose. Narrowing to (UnsupportedEndpointError, ValueError, IndexError) (whatever parse_endpoint/reader_for_format can actually raise) would preserve behavior while improving error clarity.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/dlt_filesystem/source/router.py` around lines 177 - 196, Narrow the first
fallback handler in determine_endpoint to the specific parse_endpoint failure
types that legitimately indicate the table endpoint is invalid, such as
UnsupportedEndpointError, ValueError, and IndexError. Preserve the table-to-path
retry and final error propagation behavior, while allowing unrelated programming
errors like AttributeError or TypeError to surface immediately.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.gitleaksignore:
- Around line 6-9: Extend the .gitleaksignore entries for
tests/main/filesystem/test_remote_read.py to include the generic-api-key finding
at line 59, or adjust the fake credential fixture so it no longer matches the
rule while preserving the test’s behavior.

In `@docs/changelog.md`:
- Around line 6-7: Update the Filesystem changelog entry to include Databricks
Files, Google Drive, and WebHDFS, or revise its wording to clearly indicate the
list is non-exhaustive while preserving the existing connector entries.

In `@docs/conf.py`:
- Line 98: Update the Medium URL entry in the link-check ignore configuration to
match only the specific in-repo article URL, replacing the broad domain pattern
while preserving other ignore entries.

In `@docs/supported-sources/azure-storage.md`:
- Around line 121-124: The format-hint documentation is incomplete and
inconsistent across the Azure Storage and S3 source guides. Update the
format-hint sections in docs/supported-sources/azure-storage.md lines 121-124
and docs/supported-sources/s3.md lines 94-99 to include CBOR and every currently
supported hint, or clearly mark the examples as illustrative and link to the
canonical complete format list; keep both source pages consistent.
- Around line 81-84: Clarify the SAS token encoding in the example near the
ingest command: make the `sig` value unambiguous and consistent with the
documented rule that embedded `=` is encoded as `%3D`. Remove the extra `%25`
encoding unless the example explicitly identifies the value as a raw token
requiring it, and align the surrounding prose with the chosen representation.

In `@docs/supported-sources/dropbox.md`:
- Around line 13-15: Replace the Dropbox URI’s real-looking token with an
explicit synthetic placeholder and add warnings about credential exposure and
the supported secure credential mechanism. In docs/supported-sources/dropbox.md
lines 13-15, update the URI and warning; in lines 30-33, retain a non-secret CLI
example and warn against passing real tokens as arguments. In
docs/supported-sources/ftp.md lines 14-16 and 75-78, add equivalent warnings
that copied username/password command values must not be real credentials.

In `@docs/supported-sources/ftp.md`:
- Around line 63-71: Update the FTP Authentication and Examples documentation to
consistently state that username/password credentials are required only when
connecting to an authenticated server, while servers permitting anonymous access
can be used without them.
- Around line 51-61: Update the :tls: documentation to present secure FTP-TLS as
the expected configuration, recommend auto-negotiated TLS or TLS 1.2+, and
remove TLS 1.0/1.1 from normal accepted options. Revise the FTP URI/CLI examples
to use secure transport and avoid embedding plaintext credentials, while
preserving the documented option syntax.

In `@docs/supported-sources/onedrive.md`:
- Around line 16-34: Standardize credential guidance across all four
documentation sites: in docs/supported-sources/onedrive.md lines 16-34, remove
client_secret from URI examples and avoid presenting per-connection environment
variables as the general credential mechanism; in
docs/supported-sources/google-drive.md lines 37-45, replace base64-encoded
service-account JSON credentials with file- or ADC-backed authentication and
warn against private keys in URIs; in docs/supported-sources/nextcloud.md lines
13-19, remove the real-password URL and direct users to generated app
credentials or passwordless authentication; in docs/supported-sources/oss.md
lines 23-29, remove key, secret, and token query-string examples in favor of
credential-file or environment-backed authentication. Add a concise warning that
credential-bearing URIs must not be logged or committed.

In `@docs/supported-sources/s3.md`:
- Around line 160-171: Replace the real-looking S3 credential values in the
examples at docs/supported-sources/s3.md lines 160-171, including the upload
example, with the declared YOUR_ACCESS_KEY_ID and YOUR_SECRET_ACCESS_KEY
placeholders. Also replace the UUIDs and secret-shaped value in the credential
examples at docs/supported-sources/sharepoint.md lines 16-18, including the
command example, with <CLIENT_ID>, <TENANT_ID>, and <CLIENT_SECRET>; ensure all
credential-bearing URI examples use unmistakable documentation-only
placeholders.

In `@docs/supported-sources/webdav.md`:
- Around line 20-22: Update the retry parameter documentation in the WebDAV
source reference to describe it as enabling or disabling HTTP client retries,
with true enabling retries by default. Remove the misleading wording that leads
with disabling retry while preserving the type and default details.

In `@docs/supported-sources/webhdfs.md`:
- Around line 100-103: Update the WebHDFS example heading or description around
the DuckDB copy command to replace “OSS location” with “WebHDFS location,”
leaving the rest of the example unchanged.

In `@src/dlt_filesystem/source/core.py`:
- Around line 52-82: Preserve the filesystem_incremental option through
infer_resource and FilesystemReference by reading it from locator options and
passing it as the reference field, while removing it from connector fs_kwargs
before fsspec construction. Update the affected connector paths to pop the flag
from fs_kwargs and forward it only to FilesystemReference, allowing optional
mtime-based incremental loading without sending the option to fsspec
constructors.

In `@src/dlt_filesystem/source/fsspec/ftp.py`:
- Around line 34-38: Update the tls normalization block to reject invalid values
rather than swallowing ValueError and retaining the original input. Ensure only
values successfully converted by asbool are assigned to fs_kwargs["tls"], and
propagate a clear validation error for malformed tls values before FTPFileSystem
receives them.

In `@src/dlt_filesystem/source/fsspec/gdrive.py`:
- Around line 40-48: Update the filesystem option normalization in the source
flow around fs_kwargs and GoogleDriveFileSystem construction: remove the
top-level use_local_webserver boolean cast and ensure that option is handled
within auth_kwargs, while preserving the existing token, creds, and auth_kwargs
mappings before self.fs_class is instantiated.

In `@src/dlt_filesystem/source/fsspec/oci.py`:
- Around line 37-47: Update the option decoding near cast_to_dict and
cast_to_int so cast_to_dict only JSON-decodes config_kwargs and
oci_additional_kwargs. Preserve config values as either filepath strings or
already-parsed dictionaries for OCIFileSystem.

In `@src/dlt_filesystem/source/fsspec/smb.py`:
- Around line 1-8: Validate that the required SMB host option is present before
constructing the filesystem in the SMB source flow. Add the same missing-option
guard used by the FTP/HDFS connectors, returning the established error or
validation result for a hostless URI before calling SMBFileSystem with
fs_kwargs.

In `@src/dlt_filesystem/source/fsspec/webdav.py`:
- Around line 1-8: Update the WebDAV endpoint construction to parse the URI,
normalize only its scheme from +dav to dav, and rebuild base_url without query
parameters or fragments. Preserve the path unchanged, while passing parsed query
options separately through fs_kwargs so credentials and other options are not
retained in the endpoint URL.

In `@tests/dlt_filesystem/test_util.py`:
- Around line 10-15: Extend test_asbool with assertions that invalid strings
such as "", "enabled", and "2" raise ValueError, while preserving the existing
true and false cases.

---

Nitpick comments:
In `@src/dlt_filesystem/source/router.py`:
- Around line 177-196: Narrow the first fallback handler in determine_endpoint
to the specific parse_endpoint failure types that legitimately indicate the
table endpoint is invalid, such as UnsupportedEndpointError, ValueError, and
IndexError. Preserve the table-to-path retry and final error propagation
behavior, while allowing unrelated programming errors like AttributeError or
TypeError to surface immediately.

In `@src/dlt_filesystem/util/python.py`:
- Around line 92-102: Consolidate the boolean parsing in asbool and _cast_value
so both use the same established _TRUE_VALUES and _FALSE_VALUES vocabulary.
Remove the duplicate token lists and preserve the existing invalid-string
ValueError behavior while keeping non-string values converted via bool.
- Around line 145-149: Update apply_alias to detect when both name and
effective_name already exist in data before moving the alias, and raise the
established conflict error or otherwise reject the operation instead of
overwriting effective_name’s value. Preserve the current rename behavior when
only name is present and return unchanged data when name is absent.
- Around line 129-142: Update cast_to_dict and cast_to_list to call json.loads
only for string-encoded field values, passing already-parsed dict or list values
through unchanged. Preserve the current conversion behavior for JSON strings and
return the modified data as before.

In `@tests/main/filesystem/test_remote_read.py`:
- Around line 52-56: Resolve the FTP mocking limitation in
tests/main/filesystem/test_remote_read.py by updating the shared FTP mock setup
so both disabled coverage paths work, then restore the URI-with-separate-table
Item case at lines 52-56 and the FTP unknown-extension failure case at lines
107-111. Preserve the existing test structure and assertions for the other
connectors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 031f9127-19ee-4e94-af4a-0577eea5848a

📥 Commits

Reviewing files that changed from the base of the PR and between 7b24bb8 and 3b9b008.

⛔ Files ignored due to path filters (1)
  • tests/assets/privatekey.pem is excluded by !**/*.pem
📒 Files selected for processing (74)
  • .gitleaksignore
  • docs/changelog.md
  • docs/conf.py
  • docs/supported-sources/azure-blob-storage.md
  • docs/supported-sources/azure-data-lake-storage.md
  • docs/supported-sources/azure-storage.md
  • docs/supported-sources/cbor.md
  • docs/supported-sources/databricks-files.md
  • docs/supported-sources/databricks-warehouse.md
  • docs/supported-sources/dropbox.md
  • docs/supported-sources/filesystem.md
  • docs/supported-sources/ftp.md
  • docs/supported-sources/google-cloud-storage.md
  • docs/supported-sources/google-drive.md
  • docs/supported-sources/hdfs.md
  • docs/supported-sources/msgpack.md
  • docs/supported-sources/nextcloud.md
  • docs/supported-sources/ods.md
  • docs/supported-sources/onedrive.md
  • docs/supported-sources/oracle-oci.md
  • docs/supported-sources/oss.md
  • docs/supported-sources/r2.md
  • docs/supported-sources/s3.md
  • docs/supported-sources/sftp.md
  • docs/supported-sources/sharepoint.md
  • docs/supported-sources/smb.md
  • docs/supported-sources/webdav.md
  • docs/supported-sources/webhdfs.md
  • docs/supported-sources/xlsx.md
  • docs/supported-sources/xml.md
  • docs/supported-sources/yaml.md
  • pyproject.toml
  • src/dlt_filesystem/error.py
  • src/dlt_filesystem/source/adapter.py
  • src/dlt_filesystem/source/api.py
  • src/dlt_filesystem/source/base.py
  • src/dlt_filesystem/source/core.py
  • src/dlt_filesystem/source/error.py
  • src/dlt_filesystem/source/fsspec/databricks.py
  • src/dlt_filesystem/source/fsspec/dropbox.py
  • src/dlt_filesystem/source/fsspec/ftp.py
  • src/dlt_filesystem/source/fsspec/gdrive.py
  • src/dlt_filesystem/source/fsspec/hdfs.py
  • src/dlt_filesystem/source/fsspec/local.py
  • src/dlt_filesystem/source/fsspec/oci.py
  • src/dlt_filesystem/source/fsspec/onedrive.py
  • src/dlt_filesystem/source/fsspec/oss.py
  • src/dlt_filesystem/source/fsspec/r2.py
  • src/dlt_filesystem/source/fsspec/sharepoint.py
  • src/dlt_filesystem/source/fsspec/smb.py
  • src/dlt_filesystem/source/fsspec/webdav.py
  • src/dlt_filesystem/source/fsspec/webhdfs.py
  • src/dlt_filesystem/source/impl/remote.py
  • src/dlt_filesystem/source/model.py
  • src/dlt_filesystem/source/router.py
  • src/dlt_filesystem/util/fsspec.py
  • src/dlt_filesystem/util/python.py
  • src/dlt_filesystem/util/web.py
  • src/omniload/core/factory.py
  • src/omniload/core/registry.py
  • tests/assets/privatekey-fingerprint.txt
  • tests/dlt_filesystem/format/test_bson.py
  • tests/dlt_filesystem/format/test_cbor.py
  • tests/dlt_filesystem/format/test_msgpack.py
  • tests/dlt_filesystem/format/test_xml.py
  • tests/dlt_filesystem/format/test_yaml.py
  • tests/dlt_filesystem/test_source_hints.py
  • tests/dlt_filesystem/test_source_incremental.py
  • tests/dlt_filesystem/test_source_local.py
  • tests/dlt_filesystem/test_source_remote.py
  • tests/dlt_filesystem/test_util.py
  • tests/main/filesystem/test_file_incremental.py
  • tests/main/filesystem/test_local_read.py
  • tests/main/filesystem/test_remote_read.py
💤 Files with no reviewable changes (3)
  • docs/supported-sources/azure-data-lake-storage.md
  • docs/supported-sources/azure-blob-storage.md
  • src/dlt_filesystem/source/api.py

Comment thread .gitleaksignore
Comment on lines +6 to +9
tests/main/filesystem/test_remote_read.py:generic-api-key:43
tests/main/filesystem/test_remote_read.py:generic-api-key:55
tests/main/filesystem/test_remote_read.py:generic-api-key:56
tests/main/filesystem/test_remote_read.py:generic-api-key:58

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Add the failing fixture to the allowlist.

The new entries stop at Line 58, but Gitleaks still reports generic-api-key at tests/main/filesystem/test_remote_read.py Line 59, blocking CI. Add the current fixture location (or rewrite that fake credential fixture so it does not match the rule).

Proposed fix
 tests/main/filesystem/test_remote_read.py:generic-api-key:58
+tests/main/filesystem/test_remote_read.py:generic-api-key:59
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
tests/main/filesystem/test_remote_read.py:generic-api-key:43
tests/main/filesystem/test_remote_read.py:generic-api-key:55
tests/main/filesystem/test_remote_read.py:generic-api-key:56
tests/main/filesystem/test_remote_read.py:generic-api-key:58
tests/main/filesystem/test_remote_read.py:generic-api-key:43
tests/main/filesystem/test_remote_read.py:generic-api-key:55
tests/main/filesystem/test_remote_read.py:generic-api-key:56
tests/main/filesystem/test_remote_read.py:generic-api-key:58
tests/main/filesystem/test_remote_read.py:generic-api-key:59
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.gitleaksignore around lines 6 - 9, Extend the .gitleaksignore entries for
tests/main/filesystem/test_remote_read.py to include the generic-api-key finding
at line 59, or adjust the fake credential fixture so it no longer matches the
rule while preserving the test’s behavior.

Source: Pipeline failures

Comment thread docs/changelog.md
Comment on lines +6 to +7
- Filesystem: Added support for reading from
Dropbox, FTP, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Include all connectors added by this PR in the changelog.

The bullet omits Databricks Files, Google Drive, and WebHDFS. Add them or explicitly make the wording non-exhaustive.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/changelog.md` around lines 6 - 7, Update the Filesystem changelog entry
to include Databricks Files, Google Drive, and WebHDFS, or revise its wording to
clearly indicate the list is non-exhaustive while preserving the existing
connector entries.

Comment thread docs/conf.py
r"https://github.com/",
r"https://web.archive.org/",
r"https://images.minimus.io/",
r"https://medium.com/",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Changed lines files:"
git diff --stat || true
echo
if [ -f docs/conf.py ]; then
  echo "docs/conf.py outline:"
  ast-grep outline docs/conf.py || true
  echo
  echo "docs/conf.py relevant lines:"
  nl -ba docs/conf.py | sed -n '80,110p'
fi

echo
echo "Search for medium ignore pattern:"
rg -n "medium\.com" docs/conf.py . || true

Repository: panodata/omniload

Length of output: 1399


🏁 Script executed:

#!/bin/bash
set -euo pipefail

if [ -f docs/conf.py ]; then
  echo "docs/conf.py relevant linkcheck lines:"
  sed -n '77,105p' docs/conf.py
fi

echo
echo "Medium references:"
rg -n "medium\.com|linkcheck_ignore|linkcheck_anchors_ignore_for_url" docs/conf.py . || true

Repository: panodata/omniload

Length of output: 1719


Narrow the Medium link-check ignore pattern.

r"https://medium.com/" skips validation for every Medium URL. Since the only in-repo Medium link is https://medium.com/@pavithrasainath7/mastering-file-access-in-sharepoint-with-oauth-2.0-a-comprehensive-guide-0a6b2d53736a, ignore that exact URL instead of the whole domain.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/conf.py` at line 98, Update the Medium URL entry in the link-check
ignore configuration to match only the specific in-repo article URL, replacing
the broad domain pattern while preserving other ignore entries.

Comment thread docs/supported-sources/azure-storage.md
Comment on lines +121 to +124
If your files are properly encoded but lack the correct file extension (BSON,
CSV, JSONL, or Parquet), you can provide a file type hint to inform `omniload`
about the format of the files. This is done by appending a fragment identifier
(`#format`) to the end of the path in your `--source-table` parameter.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Keep format-hint documentation consistent across remote sources.

The expanded shared readers support more formats than these source-specific lists expose, including CBOR.

  • docs/supported-sources/azure-storage.md#L121-L124: add the newer formats or state that the list is illustrative and link the canonical complete list.
  • docs/supported-sources/s3.md#L94-L99: add #cbor and the other supported hints, or link to the canonical complete list.
📍 Affects 2 files
  • docs/supported-sources/azure-storage.md#L121-L124 (this comment)
  • docs/supported-sources/s3.md#L94-L99
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/supported-sources/azure-storage.md` around lines 121 - 124, The
format-hint documentation is incomplete and inconsistent across the Azure
Storage and S3 source guides. Update the format-hint sections in
docs/supported-sources/azure-storage.md lines 121-124 and
docs/supported-sources/s3.md lines 94-99 to include CBOR and every currently
supported hint, or clearly mark the examples as illustrative and link to the
canonical complete format list; keep both source pages consistent.

Comment on lines +40 to +48
fs_kwargs = locator.options.fs_kwargs
fs_kwargs.update(kwargs)
apply_alias(fs_kwargs, "auth", "auth_kwargs")
cast_to_dict(fs_kwargs, ["creds", "auth_kwargs"])
cast_to_bool(fs_kwargs, ["use_local_webserver"])

# Create filesystem and dlt resource wrapper.
fs = self.fs_class(**fs_kwargs)
return infer_resource(fs=fs, locator=locator)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

gdrive-fsspec GoogleDriveFileSystem constructor parameters auth_kwargs creds use_local_webserver

💡 Result:

In the gdrive-fsspec library, the GoogleDriveFileSystem class provides several ways to handle authentication. Key constructor parameters related to your query include: creds: This parameter is used when employing service account authentication. It accepts a dictionary containing your service account credentials (equivalent to the JSON file downloaded from the Google Cloud Platform console) [1][2][3]. token: This parameter defines the authentication method. Common values include 'service_account', 'browser' (for initial user OAuth), 'cache' (for reusing stored credentials), and 'anon' (for public, read-only access) [1][2][3]. auth_kwargs: This is a dictionary of keyword arguments passed to the underlying authentication handler. A critical usage for this parameter is for headless or remote environments (such as CI/CD, containers, or SSH sessions) where a browser cannot be opened. By setting 'use_local_webserver': False within auth_kwargs, you can force the system to request an authentication token via the console instead of attempting to bind a local callback server [3]. Example usage for headless environments: fs = GoogleDriveFileSystem( token=token, auth_kwargs={'use_local_webserver': False},) [3]

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate gdrive connector =="
fd -a 'gdrive\.py$' . | sed 's#^\./##'

echo "== relevant source =="
file="$(fd 'gdrive\.py$' . | head -n 1 || true)"
if [ -n "${file:-}" ]; then
  wc -l "$file"
  sed -n '1,140p' "$file"
fi

echo "== inspect installed package metadata/source if present =="
python3 - <<'PY'
import sys, importlib.util
for name in ["gdrive_fsspec", "gdrive_fsspec.google_drive", "gdrive_fsspec.gdrive_fsspec"]:
    spec = importlib.util.find_spec(name)
    print(name, "found", bool(spec), "origin", getattr(spec, "origin", None))
PY
python3 - <<'PY'
try:
    import inspect
    import gdrive_fsspec
    print("gdrive_fsspec module:", gdrive_fsspec.__file__)
    print("GoogleDriveFileSystem:", getattr(gdrive_fsspec, "GoogleDriveFileSystem", None))
    cls = gdrive_fsspec.GoogleDriveFileSystem
    sig = inspect.signature(cls.__init__)
    print("signature:", sig)
    print("params:", list(sig.parameters.keys()))
except Exception as e:
    print("failed:", type(e).__name__, e)
PY

echo "== GitHub raw source snippets =="
urls=(
  "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/gdrive_fsspec/gdrive_fsspec.py"
  "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/google_drivefs.py"
  "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/main/gdrive_fsspec/gdrive_fsspec.py"
)
for url in "${urls[@]}"; do
  echo "--- $url"
  if command -v curl >/dev/null 2>&1; then
    curl -fsSL "$url" | sed -n '1,220p'
  else
    echo "curl unavailable"
  fi
done

Repository: panodata/omniload

Length of output: 2281


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
import json
import urllib.request

repos = ["fsspec/gdrive-fsspec", "fsspec/gdrive-fsspec"]
files = [
    "gdrive_fsspec/gdrive_fsspec.py",
    "gdrive_fsspec/google_drive.py",
    "google_drive.py",
    "google_drivefs.py",
]

for repo in repos:
    print(f"--- repo {repo} tree top 200")
    url = f"https://api.github.com/repos/{repo}/git/trees/main?recursive=1"
    try:
        with urllib.request.urlopen(url, timeout=20) as r:
            data = json.load(r)
    except Exception as e:
        print("tree lookup failed:", type(e).__name__, e)
        data = None
    if data:
        for blob in sorted(data.get("tree", []), key=lambda b: b.get("path", ""))[:200]:
            p = blob.get("path")
            if p.endswith(".py") and ("gdrive" in p.lower() or "google" in p.lower() or "oauth2client" in p.lower()):
                print(blob.get("path"), blob.get("type"))
PY

python3 - <<'PY'
import re
import urllib.request

urls = [
    "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/main/gdrive_fsspec/gdrive_fsspec.py",
    "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/main/gdrive_fsspec/google_drive.py",
    "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/gdrive_fsspec/gdrive_fsspec.py",
    "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/gdrive_fsspec/google_drive.py",
    "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/google_drive.py",
    "https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/google_drivefs.py",
]
pat = re.compile(r"\b(class\s+GoogleDriveFileSystem|def\s+__init__|auth_kwargs|creds|use_local_webserver|token=|service_account|browser|cache|anon)\b")

for url in urls:
    try:
        text = urllib.request.urlopen(url, timeout=20).read().decode("utf-8", "replace")
    except Exception as e:
        print("FETCH:", url, "FAILED", type(e).__name__, e)
        continue
    print("FETCH:", url, "OK", len(text))
    for i, line in enumerate(text.splitlines(), 1):
        if pat.search(line):
            start=max(1,i-5); end=min(len(text.splitlines()),i+8)
            lines=text.splitlines()
            print(f"{url}:{i}: {line.strip()}")
            for j in range(start,end):
                print(f"  {j}: {lines[j-1]}")
            print()
PY

Repository: panodata/omniload

Length of output: 2033


🌐 Web query:

site:github.com/fsspec/gdrive-fsspec "class GoogleDriveFileSystem" "auth_kwargs" "use_local_webserver"

💡 Result:

The GoogleDriveFileSystem class, as implemented in the gdrive-fsspec library, does not appear to support a parameter named use_local_webserver directly [1][2]. Authentication for GoogleDriveFileSystem is typically handled through the token argument [1][2]. The supported authentication methods include [1][2]: 1. Browser-based OAuth: Setting token='browser' triggers a flow that opens a browser window to complete the authentication process [1][2]. 2. Cached Tokens: Setting token='cache' allows the use of previously authenticated sessions [1][2]. 3. Service Account Credentials: Providing credentials and setting token='service_account' [1][2]. 4. Anonymous (read-only): Setting token='anon' for publicly accessible files [1][2]. If you are looking for local webserver-based OAuth authentication (often used in other Google Cloud Python client libraries to capture the redirect URL), this functionality is generally handled by the underlying authentication utility—typically pydata-google-auth in the case of gdrive-fsspec [3]—rather than as a direct parameter in the GoogleDriveFileSystem constructor [1][2]. If you encounter issues with the default authentication flow, it is often recommended to create your own Google Cloud Platform (GCP) project to manage your own OAuth credentials [3], rather than relying on the default shared credentials provided by the library [3].

Citations:


Align Google Drive auth options with GoogleDriveFileSystem’s constructor signature.

gdrive-fsspec documents GoogleDriveFileSystem constructor auth options as token, creds, and auth_kwargs; use_local_webserver is nested inside auth_kwargs, so cast_to_bool(fs_kwargs, ["use_local_webserver"]) casts the wrong key.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/dlt_filesystem/source/fsspec/gdrive.py` around lines 40 - 48, Update the
filesystem option normalization in the source flow around fs_kwargs and
GoogleDriveFileSystem construction: remove the top-level use_local_webserver
boolean cast and ensure that option is handled within auth_kwargs, while
preserving the existing token, creds, and auth_kwargs mappings before
self.fs_class is instantiated.

Comment on lines +37 to +47
# Decode individual options (type casting, default values, sanity checks). Schema:
# {"default_block_size": int, "config": dict, "config_kwargs": dict, "oci_additional_kwargs": dict}
fs_kwargs = locator.options.fs_kwargs
fs_kwargs.update(kwargs)

# Decode dict-typed `config`, `config_kwargs`, `oci_additional_kwargs` from JSON.
cast_to_dict(fs_kwargs, ["config", "config_kwargs", "oci_additional_kwargs"])
# The `default_` prefix seems unnecessary. Let's make it optional by using an alias.
apply_alias(fs_kwargs, "block_size", "default_block_size")
# Convert to integers.
cast_to_int(fs_kwargs, ["default_block_size"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

ocifs OCIFileSystem config parameter filepath vs dict

💡 Result:

In the ocifs library, the config parameter of the OCIFileSystem class accepts either a string (representing a file path) or a dictionary (representing a configuration object) to authenticate and connect to Oracle Cloud Infrastructure [1]. Key differences and usage: Filepath (str): You can provide the location of your OCI configuration file (e.g., "/.oci/config") as a string [1][2]. If provided as a file path, the library uses this file to load the connection details [1]. Dictionary (dict): You can provide a configuration dictionary, typically the object returned by the oci.config.from_file method from the OCI Python SDK [1][2]. This is useful if you have already loaded or processed the configuration in your code before initializing the filesystem [2]. If the config parameter is set to None, the library will attempt to use a Resource Principal or Instance Principal configured in your environment [1]. When using higher-level libraries like pandas or dask with OCI URLs (oci://), you pass these options via the storage_options dictionary, for example: storage_options={"config": "/.oci/config"} [2][3].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Locate oci.py and cast/resolve helpers"
rg -n "def (cast_to_dict|cast_to_int|apply_alias)|cast_to_dict|apply_alias|class.*Options|fs_kwargs" src/dlt_filesystem source . 2>/dev/null || true

echo
echo "Relevant oci.py section"
if [ -f src/dlt_filesystem/source/fsspec/oci.py ]; then
  nl -ba src/dlt_filesystem/source/fsspec/oci.py | sed -n '1,120p'
fi

echo
echo "Search definitions"
rg -n "def (cast_to_dict|cast_to_int|apply_alias)|def .*locator|locator =|dataclasses|options" src/dlt_filesystem/source/fsspec src/dlt_filesystem/source 2>/dev/null || true

Repository: panodata/omniload

Length of output: 17607


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Read relevant util.py helper implementations"
python3 - <<'PY'
from pathlib import Path
p=Path("src/dlt_filesystem/util/python.py")
lines=p.read_text().splitlines()
for name, start, end in [("cast_to_dict", 129, 144), ("apply_alias", 145, 162)]:
    print(f"\n--- {p}:{name}:{start}-{end} ---")
    for i in range(max(1,start-5), min(len(lines),end)+1):
        print(f"{i:4d} {lines[i-1]}")
PY

echo
echo "Read model options"
python3 - <<'PY'
from pathlib import Path
p=Path("src/dlt_filesystem/source/model.py")
lines=p.read_text().splitlines()
for i in range(max(1,30), min(len(lines),75)+1):
    print(f"{i:4d} {lines[i-1]}")
PY

echo
echo "Relevant oci.py source"
python3 - <<'PY'
from pathlib import Path
p=Path("src/dlt_filesystem/source/fsspec/oci.py")
lines=p.read_text().splitlines()
for i in range(max(1,1), min(len(lines),80)+1):
    print(f"{i:4d} {lines[i-1]}")
PY

Repository: panodata/omniload

Length of output: 5486


Don’t JSON-decode the OCI config string option.

ocifs.OCIFileSystem accepts config as either a filepath string or a parsed config dict, while config_kwargs/oci_additional_kwargs are dict-shaped. Including "config" in cast_to_dict() unconditionally runs json.loads() on values like "~/.oci/config", so oci://...?config=~/.oci/config raises JSONDecodeError.

🐛 Proposed fix
-        cast_to_dict(fs_kwargs, ["config", "config_kwargs", "oci_additional_kwargs"])
+        # `config` may legitimately be a filepath string, not JSON — leave it untouched.
+        cast_to_dict(fs_kwargs, ["config_kwargs", "oci_additional_kwargs"])
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Decode individual options (type casting, default values, sanity checks). Schema:
# {"default_block_size": int, "config": dict, "config_kwargs": dict, "oci_additional_kwargs": dict}
fs_kwargs = locator.options.fs_kwargs
fs_kwargs.update(kwargs)
# Decode dict-typed `config`, `config_kwargs`, `oci_additional_kwargs` from JSON.
cast_to_dict(fs_kwargs, ["config", "config_kwargs", "oci_additional_kwargs"])
# The `default_` prefix seems unnecessary. Let's make it optional by using an alias.
apply_alias(fs_kwargs, "block_size", "default_block_size")
# Convert to integers.
cast_to_int(fs_kwargs, ["default_block_size"])
# Decode individual options (type casting, default values, sanity checks). Schema:
# {"default_block_size": int, "config": dict, "config_kwargs": dict, "oci_additional_kwargs": dict}
fs_kwargs = locator.options.fs_kwargs
fs_kwargs.update(kwargs)
# Decode dict-typed `config_kwargs`, `oci_additional_kwargs` from JSON.
# `config` may legitimately be a filepath string, not JSON — leave it untouched.
cast_to_dict(fs_kwargs, ["config_kwargs", "oci_additional_kwargs"])
# The `default_` prefix seems unnecessary. Let's make it optional by using an alias.
apply_alias(fs_kwargs, "block_size", "default_block_size")
# Convert to integers.
cast_to_int(fs_kwargs, ["default_block_size"])
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/dlt_filesystem/source/fsspec/oci.py` around lines 37 - 47, Update the
option decoding near cast_to_dict and cast_to_int so cast_to_dict only
JSON-decodes config_kwargs and oci_additional_kwargs. Preserve config values as
either filepath strings or already-parsed dictionaries for OCIFileSystem.

Comment on lines +1 to +8
from typing import Type

from fsspec import AbstractFileSystem

from dlt_filesystem.source.base import FilesystemSource
from dlt_filesystem.source.core import infer_resource
from dlt_filesystem.source.model import FilesystemLocator
from dlt_filesystem.util.python import cast_to_bool, cast_to_int

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files:"
fd -a 'smb.py|base.py|core.py|error.py' . | sed 's#^\./##'

echo
echo "smb.py excerpt:"
file="$(fd 'smb.py' . | head -n 1)"
if [ -n "${file:-}" ]; then
  nl -ba "$file" | sed -n '1,120p'
fi

echo
echo "Relevant searches:"
rg -n "SMBFileSystem|MissingConnectorOption|host.*not in fs_kwargs|def infer_resource|class FilesystemLocator|smb://" .

Repository: panodata/omniload

Length of output: 750


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "smb.py excerpt:"
sed -n '1,120p' src/dlt_filesystem/source/fsspec/smb.py

echo
echo "source error module excerpt:"
sed -n '1,160p' src/dlt_filesystem/source/error.py

echo
echo "SMB connector files and validation patterns:"
sed -n '1,120p' src/dlt_filesystem/source/fsspec/ftp.py 2>/dev/null || true
sed -n '1,120p' src/dlt_filesystem/source/fsspec/hdfs.py 2>/dev/null || true

echo
echo "SMBFileSystem references:"
rg -n "SMBFileSystem|MissingConnectorOption|host.*not in fs_kwargs|fs_kwargs" src/dlt_filesystem/source/fsspec src/dlt_filesystem

Repository: panodata/omniload

Length of output: 23990


🌐 Web query:

fsspec SMBFileSystem constructor host required source host parameter

💡 Result:

Yes, the host parameter is a required argument for the fsspec SMBFileSystem constructor [1][2]. In the fsspec implementation for SMB, the init method is defined with host as the first positional argument [1][2]. Official documentation and source code indicate that the host parameter is used to specify the remote server name or IP address to connect to [1][2]. When using fsspec, it is recommended to provide a valid hostname or IP address for the host component. The documentation explicitly notes that you should not use the Windows/NetBIOS machine name for the host component [1][2]. While other parameters such as port, username, and password may be optional depending on your specific authentication setup (e.g., if using Kerberos or anonymous access), the host is necessary to identify the destination server [1][2].

Citations:


Validate the required SMB host before constructing the filesystem.

SMB is the only fsspec connector here that lets a hostless URI reach SMBFileSystem(**fs_kwargs), where host is required. Add the same missing-option guard used by FTP/HDFS before constructing the filesystem.

Proposed fix
+from dlt_filesystem.error import MissingConnectorOption
 from dlt_filesystem.source.base import FilesystemSource
...
         fs_kwargs = locator.options.fs_kwargs
         fs_kwargs.update(kwargs)
+        if "host" not in fs_kwargs or not fs_kwargs["host"]:
+            raise MissingConnectorOption("host", "SMB")
         cast_to_int(
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/dlt_filesystem/source/fsspec/smb.py` around lines 1 - 8, Validate that
the required SMB host option is present before constructing the filesystem in
the SMB source flow. Add the same missing-option guard used by the FTP/HDFS
connectors, returning the established error or validation result for a hostless
URI before calling SMBFileSystem with fs_kwargs.

Comment on lines +1 to +8
from typing import Type

from fsspec import AbstractFileSystem

from dlt_filesystem.source.base import FilesystemSource
from dlt_filesystem.source.core import infer_resource
from dlt_filesystem.source.model import FilesystemLocator
from dlt_filesystem.util.python import asbool, cast_to_bool, cast_to_dict, cast_to_int

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Parse and rebuild the WebDAV endpoint instead of mutating the full URI.

Line 29 also rewrites +dav inside file paths or query values. More importantly, Line 64 passes the option-bearing URI as base_url, so query credentials parsed into fs_kwargs can remain in the endpoint URL. Normalize only the scheme and remove query/fragment from base_url.

Proposed fix
 from typing import Type
+from urllib.parse import urlsplit, urlunsplit
...
-        uri = uri.replace("+webdav", "").replace("+dav", "")
+        parts = urlsplit(uri)
+        scheme = parts.scheme.removesuffix("+webdav").removesuffix("+dav")
+        uri = urlunsplit(parts._replace(scheme=scheme))
+        base_url = urlunsplit(
+            parts._replace(scheme=scheme, query="", fragment="")
+        )
...
-        fs = self.fs_class(base_url=uri, auth=auth, **fs_kwargs)
+        fs = self.fs_class(base_url=base_url, auth=auth, **fs_kwargs)

Also applies to: 25-29, 37-40, 63-65

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/dlt_filesystem/source/fsspec/webdav.py` around lines 1 - 8, Update the
WebDAV endpoint construction to parse the URI, normalize only its scheme from
+dav to dav, and rebuild base_url without query parameters or fragments.
Preserve the path unchanged, while passing parsed query options separately
through fs_kwargs so credentials and other options are not retained in the
endpoint URL.

Comment on lines +10 to +15
def test_asbool():
"""Validate the `asbool` utility function."""
assert asbool("true") is True
assert asbool("yes") is True
assert asbool("false") is False
assert asbool("no") is False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Cover invalid boolean inputs.

asbool is expected to reject unrecognized strings with ValueError; add cases such as "", "enabled", and "2" so connector-option parsing cannot regress into accepting arbitrary values. As per path instructions, “Prefer flagging missing edge-case coverage over style nits.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/dlt_filesystem/test_util.py` around lines 10 - 15, Extend test_asbool
with assertions that invalid strings such as "", "enabled", and "2" raise
ValueError, while preserving the existing true and false cases.

Source: Path instructions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coderabbit-review Tell CodeRabbit to review the patch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Morty Proxy This is a proxified and sanitized view of the page, visit original site.