Filesystem: Add read support for more fsspec-based filesystems#254
Filesystem: Add read support for more fsspec-based filesystems#254amotl wants to merge 6 commits intomainpanodata/omniload:mainfrom filesystems-read-202607bpanodata/omniload:filesystems-read-202607bCopy head branch name to clipboard
Conversation
- Add support for Databricks files, Dropbox, FTP, Google Drive, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV, WebHDFS. - Filesystem documentation improvements across the board.
Summary by CodeRabbit
WalkthroughChangesFilesystem source expansion
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant SourceDestinationFactory
participant FilesystemSource
participant fsspecFilesystem
participant DLTResource
CLI->>SourceDestinationFactory: resolve source URI scheme
SourceDestinationFactory->>FilesystemSource: create dlt_source(uri, table)
FilesystemSource->>fsspecFilesystem: instantiate connector with decoded options
FilesystemSource->>DLTResource: infer filesystem and reader resources
DLTResource-->>CLI: return composed source resource
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Documentation build overview
30 files changed ·
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #254 +/- ##
==========================================
+ Coverage 55.71% 57.51% +1.80%
==========================================
Files 210 227 +17
Lines 9894 10448 +554
==========================================
+ Hits 5512 6009 +497
- Misses 4382 4439 +57 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 19
🧹 Nitpick comments (5)
tests/main/filesystem/test_remote_read.py (1)
52-56: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftResolve the FTP mock limitation and restore both coverage paths.
FTP is the only connector with these URI/error variants disabled, leaving both table-splitting and unsupported-format behavior unvalidated.
tests/main/filesystem/test_remote_read.py#L52-L56: restore the URI-with-separate-tableFTP case.tests/main/filesystem/test_remote_read.py#L107-L111: restore the FTP unknown-extension failure case.As per path instructions, “Prefer flagging missing edge-case coverage over style nits.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/main/filesystem/test_remote_read.py` around lines 52 - 56, Resolve the FTP mocking limitation in tests/main/filesystem/test_remote_read.py by updating the shared FTP mock setup so both disabled coverage paths work, then restore the URI-with-separate-table Item case at lines 52-56 and the FTP unknown-extension failure case at lines 107-111. Preserve the existing test structure and assertions for the other connectors.Source: Path instructions
src/dlt_filesystem/util/python.py (3)
92-102: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
asboolduplicates bool-parsing already in_cast_value, with a different truthy/falsey vocabulary.
_cast_value'sboolbranch recognizes_TRUE_VALUES/_FALSE_VALUES(true/1/yes/on), whileasbooladditionally acceptsy/t/n/f. Two independent bool-parsers with diverging accepted tokens is a minor DRY/consistency smell; consider consolidating if both are meant to serve the same "truthy string" concept.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/dlt_filesystem/util/python.py` around lines 92 - 102, Consolidate the boolean parsing in asbool and _cast_value so both use the same established _TRUE_VALUES and _FALSE_VALUES vocabulary. Remove the duplicate token lists and preserve the existing invalid-string ValueError behavior while keeping non-string values converted via bool.
145-149: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
apply_aliassilently overwrites on key conflict.If both
nameandeffective_nameare present indata,apply_aliassilently discardseffective_name's original value in favor ofname's, with no warning. A user supplying both the alias and the canonical key (e.g.block_sizeanddefault_block_size) would get surprising, unannounced precedence.♻️ Suggested guard
def apply_alias(data: Dict[str, Any], name: str, effective_name: str) -> Dict[str, Any]: """Apply aliasing to dictionary keys.""" - if name in data: + if name in data and effective_name in data: + raise ValueError(f"Both '{name}' and '{effective_name}' were provided; use only one") + if name in data: data[effective_name] = data.pop(name) return data🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/dlt_filesystem/util/python.py` around lines 145 - 149, Update apply_alias to detect when both name and effective_name already exist in data before moving the alias, and raise the established conflict error or otherwise reject the operation instead of overwriting effective_name’s value. Preserve the current rename behavior when only name is present and return unchanged data when name is absent.
129-142: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
cast_to_dict/cast_to_listassume string input.Both call
json.loads(data[field_name])unconditionally. If a caller ever passes an already-parseddict/listfor one of these fields (rather than a JSON-encoded string), this raises aTypeErrorinstead of passing the value through. Current call sites feed these from URL-decoded strings, so this is defensive rather than an active bug, but a guard would make the helpers robust for future non-URL callers.🛡️ Suggested guard
def cast_to_dict(data: Dict[str, Any], names: List[str]) -> Dict[str, Any]: """Cast dictionary values from JSON.""" for field_name in names: - if field_name in data: + if field_name in data and isinstance(data[field_name], str): data[field_name] = json.loads(data[field_name]) return data🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/dlt_filesystem/util/python.py` around lines 129 - 142, Update cast_to_dict and cast_to_list to call json.loads only for string-encoded field values, passing already-parsed dict or list values through unchanged. Preserve the current conversion behavior for JSON strings and return the modified data as before.src/dlt_filesystem/source/router.py (1)
177-196: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider narrowing the blind
except Exception.Static analysis flags the bare
except Exception(Line 188) used to gate the table→path fallback. The two-stage retry logic is fine, but catching everything (including bugs likeAttributeError/TypeError) makes failures harder to diagnose. Narrowing to(UnsupportedEndpointError, ValueError, IndexError)(whateverparse_endpoint/reader_for_formatcan actually raise) would preserve behavior while improving error clarity.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/dlt_filesystem/source/router.py` around lines 177 - 196, Narrow the first fallback handler in determine_endpoint to the specific parse_endpoint failure types that legitimately indicate the table endpoint is invalid, such as UnsupportedEndpointError, ValueError, and IndexError. Preserve the table-to-path retry and final error propagation behavior, while allowing unrelated programming errors like AttributeError or TypeError to surface immediately.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.gitleaksignore:
- Around line 6-9: Extend the .gitleaksignore entries for
tests/main/filesystem/test_remote_read.py to include the generic-api-key finding
at line 59, or adjust the fake credential fixture so it no longer matches the
rule while preserving the test’s behavior.
In `@docs/changelog.md`:
- Around line 6-7: Update the Filesystem changelog entry to include Databricks
Files, Google Drive, and WebHDFS, or revise its wording to clearly indicate the
list is non-exhaustive while preserving the existing connector entries.
In `@docs/conf.py`:
- Line 98: Update the Medium URL entry in the link-check ignore configuration to
match only the specific in-repo article URL, replacing the broad domain pattern
while preserving other ignore entries.
In `@docs/supported-sources/azure-storage.md`:
- Around line 121-124: The format-hint documentation is incomplete and
inconsistent across the Azure Storage and S3 source guides. Update the
format-hint sections in docs/supported-sources/azure-storage.md lines 121-124
and docs/supported-sources/s3.md lines 94-99 to include CBOR and every currently
supported hint, or clearly mark the examples as illustrative and link to the
canonical complete format list; keep both source pages consistent.
- Around line 81-84: Clarify the SAS token encoding in the example near the
ingest command: make the `sig` value unambiguous and consistent with the
documented rule that embedded `=` is encoded as `%3D`. Remove the extra `%25`
encoding unless the example explicitly identifies the value as a raw token
requiring it, and align the surrounding prose with the chosen representation.
In `@docs/supported-sources/dropbox.md`:
- Around line 13-15: Replace the Dropbox URI’s real-looking token with an
explicit synthetic placeholder and add warnings about credential exposure and
the supported secure credential mechanism. In docs/supported-sources/dropbox.md
lines 13-15, update the URI and warning; in lines 30-33, retain a non-secret CLI
example and warn against passing real tokens as arguments. In
docs/supported-sources/ftp.md lines 14-16 and 75-78, add equivalent warnings
that copied username/password command values must not be real credentials.
In `@docs/supported-sources/ftp.md`:
- Around line 63-71: Update the FTP Authentication and Examples documentation to
consistently state that username/password credentials are required only when
connecting to an authenticated server, while servers permitting anonymous access
can be used without them.
- Around line 51-61: Update the :tls: documentation to present secure FTP-TLS as
the expected configuration, recommend auto-negotiated TLS or TLS 1.2+, and
remove TLS 1.0/1.1 from normal accepted options. Revise the FTP URI/CLI examples
to use secure transport and avoid embedding plaintext credentials, while
preserving the documented option syntax.
In `@docs/supported-sources/onedrive.md`:
- Around line 16-34: Standardize credential guidance across all four
documentation sites: in docs/supported-sources/onedrive.md lines 16-34, remove
client_secret from URI examples and avoid presenting per-connection environment
variables as the general credential mechanism; in
docs/supported-sources/google-drive.md lines 37-45, replace base64-encoded
service-account JSON credentials with file- or ADC-backed authentication and
warn against private keys in URIs; in docs/supported-sources/nextcloud.md lines
13-19, remove the real-password URL and direct users to generated app
credentials or passwordless authentication; in docs/supported-sources/oss.md
lines 23-29, remove key, secret, and token query-string examples in favor of
credential-file or environment-backed authentication. Add a concise warning that
credential-bearing URIs must not be logged or committed.
In `@docs/supported-sources/s3.md`:
- Around line 160-171: Replace the real-looking S3 credential values in the
examples at docs/supported-sources/s3.md lines 160-171, including the upload
example, with the declared YOUR_ACCESS_KEY_ID and YOUR_SECRET_ACCESS_KEY
placeholders. Also replace the UUIDs and secret-shaped value in the credential
examples at docs/supported-sources/sharepoint.md lines 16-18, including the
command example, with <CLIENT_ID>, <TENANT_ID>, and <CLIENT_SECRET>; ensure all
credential-bearing URI examples use unmistakable documentation-only
placeholders.
In `@docs/supported-sources/webdav.md`:
- Around line 20-22: Update the retry parameter documentation in the WebDAV
source reference to describe it as enabling or disabling HTTP client retries,
with true enabling retries by default. Remove the misleading wording that leads
with disabling retry while preserving the type and default details.
In `@docs/supported-sources/webhdfs.md`:
- Around line 100-103: Update the WebHDFS example heading or description around
the DuckDB copy command to replace “OSS location” with “WebHDFS location,”
leaving the rest of the example unchanged.
In `@src/dlt_filesystem/source/core.py`:
- Around line 52-82: Preserve the filesystem_incremental option through
infer_resource and FilesystemReference by reading it from locator options and
passing it as the reference field, while removing it from connector fs_kwargs
before fsspec construction. Update the affected connector paths to pop the flag
from fs_kwargs and forward it only to FilesystemReference, allowing optional
mtime-based incremental loading without sending the option to fsspec
constructors.
In `@src/dlt_filesystem/source/fsspec/ftp.py`:
- Around line 34-38: Update the tls normalization block to reject invalid values
rather than swallowing ValueError and retaining the original input. Ensure only
values successfully converted by asbool are assigned to fs_kwargs["tls"], and
propagate a clear validation error for malformed tls values before FTPFileSystem
receives them.
In `@src/dlt_filesystem/source/fsspec/gdrive.py`:
- Around line 40-48: Update the filesystem option normalization in the source
flow around fs_kwargs and GoogleDriveFileSystem construction: remove the
top-level use_local_webserver boolean cast and ensure that option is handled
within auth_kwargs, while preserving the existing token, creds, and auth_kwargs
mappings before self.fs_class is instantiated.
In `@src/dlt_filesystem/source/fsspec/oci.py`:
- Around line 37-47: Update the option decoding near cast_to_dict and
cast_to_int so cast_to_dict only JSON-decodes config_kwargs and
oci_additional_kwargs. Preserve config values as either filepath strings or
already-parsed dictionaries for OCIFileSystem.
In `@src/dlt_filesystem/source/fsspec/smb.py`:
- Around line 1-8: Validate that the required SMB host option is present before
constructing the filesystem in the SMB source flow. Add the same missing-option
guard used by the FTP/HDFS connectors, returning the established error or
validation result for a hostless URI before calling SMBFileSystem with
fs_kwargs.
In `@src/dlt_filesystem/source/fsspec/webdav.py`:
- Around line 1-8: Update the WebDAV endpoint construction to parse the URI,
normalize only its scheme from +dav to dav, and rebuild base_url without query
parameters or fragments. Preserve the path unchanged, while passing parsed query
options separately through fs_kwargs so credentials and other options are not
retained in the endpoint URL.
In `@tests/dlt_filesystem/test_util.py`:
- Around line 10-15: Extend test_asbool with assertions that invalid strings
such as "", "enabled", and "2" raise ValueError, while preserving the existing
true and false cases.
---
Nitpick comments:
In `@src/dlt_filesystem/source/router.py`:
- Around line 177-196: Narrow the first fallback handler in determine_endpoint
to the specific parse_endpoint failure types that legitimately indicate the
table endpoint is invalid, such as UnsupportedEndpointError, ValueError, and
IndexError. Preserve the table-to-path retry and final error propagation
behavior, while allowing unrelated programming errors like AttributeError or
TypeError to surface immediately.
In `@src/dlt_filesystem/util/python.py`:
- Around line 92-102: Consolidate the boolean parsing in asbool and _cast_value
so both use the same established _TRUE_VALUES and _FALSE_VALUES vocabulary.
Remove the duplicate token lists and preserve the existing invalid-string
ValueError behavior while keeping non-string values converted via bool.
- Around line 145-149: Update apply_alias to detect when both name and
effective_name already exist in data before moving the alias, and raise the
established conflict error or otherwise reject the operation instead of
overwriting effective_name’s value. Preserve the current rename behavior when
only name is present and return unchanged data when name is absent.
- Around line 129-142: Update cast_to_dict and cast_to_list to call json.loads
only for string-encoded field values, passing already-parsed dict or list values
through unchanged. Preserve the current conversion behavior for JSON strings and
return the modified data as before.
In `@tests/main/filesystem/test_remote_read.py`:
- Around line 52-56: Resolve the FTP mocking limitation in
tests/main/filesystem/test_remote_read.py by updating the shared FTP mock setup
so both disabled coverage paths work, then restore the URI-with-separate-table
Item case at lines 52-56 and the FTP unknown-extension failure case at lines
107-111. Preserve the existing test structure and assertions for the other
connectors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 031f9127-19ee-4e94-af4a-0577eea5848a
⛔ Files ignored due to path filters (1)
tests/assets/privatekey.pemis excluded by!**/*.pem
📒 Files selected for processing (74)
.gitleaksignoredocs/changelog.mddocs/conf.pydocs/supported-sources/azure-blob-storage.mddocs/supported-sources/azure-data-lake-storage.mddocs/supported-sources/azure-storage.mddocs/supported-sources/cbor.mddocs/supported-sources/databricks-files.mddocs/supported-sources/databricks-warehouse.mddocs/supported-sources/dropbox.mddocs/supported-sources/filesystem.mddocs/supported-sources/ftp.mddocs/supported-sources/google-cloud-storage.mddocs/supported-sources/google-drive.mddocs/supported-sources/hdfs.mddocs/supported-sources/msgpack.mddocs/supported-sources/nextcloud.mddocs/supported-sources/ods.mddocs/supported-sources/onedrive.mddocs/supported-sources/oracle-oci.mddocs/supported-sources/oss.mddocs/supported-sources/r2.mddocs/supported-sources/s3.mddocs/supported-sources/sftp.mddocs/supported-sources/sharepoint.mddocs/supported-sources/smb.mddocs/supported-sources/webdav.mddocs/supported-sources/webhdfs.mddocs/supported-sources/xlsx.mddocs/supported-sources/xml.mddocs/supported-sources/yaml.mdpyproject.tomlsrc/dlt_filesystem/error.pysrc/dlt_filesystem/source/adapter.pysrc/dlt_filesystem/source/api.pysrc/dlt_filesystem/source/base.pysrc/dlt_filesystem/source/core.pysrc/dlt_filesystem/source/error.pysrc/dlt_filesystem/source/fsspec/databricks.pysrc/dlt_filesystem/source/fsspec/dropbox.pysrc/dlt_filesystem/source/fsspec/ftp.pysrc/dlt_filesystem/source/fsspec/gdrive.pysrc/dlt_filesystem/source/fsspec/hdfs.pysrc/dlt_filesystem/source/fsspec/local.pysrc/dlt_filesystem/source/fsspec/oci.pysrc/dlt_filesystem/source/fsspec/onedrive.pysrc/dlt_filesystem/source/fsspec/oss.pysrc/dlt_filesystem/source/fsspec/r2.pysrc/dlt_filesystem/source/fsspec/sharepoint.pysrc/dlt_filesystem/source/fsspec/smb.pysrc/dlt_filesystem/source/fsspec/webdav.pysrc/dlt_filesystem/source/fsspec/webhdfs.pysrc/dlt_filesystem/source/impl/remote.pysrc/dlt_filesystem/source/model.pysrc/dlt_filesystem/source/router.pysrc/dlt_filesystem/util/fsspec.pysrc/dlt_filesystem/util/python.pysrc/dlt_filesystem/util/web.pysrc/omniload/core/factory.pysrc/omniload/core/registry.pytests/assets/privatekey-fingerprint.txttests/dlt_filesystem/format/test_bson.pytests/dlt_filesystem/format/test_cbor.pytests/dlt_filesystem/format/test_msgpack.pytests/dlt_filesystem/format/test_xml.pytests/dlt_filesystem/format/test_yaml.pytests/dlt_filesystem/test_source_hints.pytests/dlt_filesystem/test_source_incremental.pytests/dlt_filesystem/test_source_local.pytests/dlt_filesystem/test_source_remote.pytests/dlt_filesystem/test_util.pytests/main/filesystem/test_file_incremental.pytests/main/filesystem/test_local_read.pytests/main/filesystem/test_remote_read.py
💤 Files with no reviewable changes (3)
- docs/supported-sources/azure-data-lake-storage.md
- docs/supported-sources/azure-blob-storage.md
- src/dlt_filesystem/source/api.py
| tests/main/filesystem/test_remote_read.py:generic-api-key:43 | ||
| tests/main/filesystem/test_remote_read.py:generic-api-key:55 | ||
| tests/main/filesystem/test_remote_read.py:generic-api-key:56 | ||
| tests/main/filesystem/test_remote_read.py:generic-api-key:58 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Add the failing fixture to the allowlist.
The new entries stop at Line 58, but Gitleaks still reports generic-api-key at tests/main/filesystem/test_remote_read.py Line 59, blocking CI. Add the current fixture location (or rewrite that fake credential fixture so it does not match the rule).
Proposed fix
tests/main/filesystem/test_remote_read.py:generic-api-key:58
+tests/main/filesystem/test_remote_read.py:generic-api-key:59📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| tests/main/filesystem/test_remote_read.py:generic-api-key:43 | |
| tests/main/filesystem/test_remote_read.py:generic-api-key:55 | |
| tests/main/filesystem/test_remote_read.py:generic-api-key:56 | |
| tests/main/filesystem/test_remote_read.py:generic-api-key:58 | |
| tests/main/filesystem/test_remote_read.py:generic-api-key:43 | |
| tests/main/filesystem/test_remote_read.py:generic-api-key:55 | |
| tests/main/filesystem/test_remote_read.py:generic-api-key:56 | |
| tests/main/filesystem/test_remote_read.py:generic-api-key:58 | |
| tests/main/filesystem/test_remote_read.py:generic-api-key:59 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.gitleaksignore around lines 6 - 9, Extend the .gitleaksignore entries for
tests/main/filesystem/test_remote_read.py to include the generic-api-key finding
at line 59, or adjust the fake credential fixture so it no longer matches the
rule while preserving the test’s behavior.
Source: Pipeline failures
| - Filesystem: Added support for reading from | ||
| Dropbox, FTP, HDFS, OCI, OneDrive, OSS, R2, SharePoint, SMB, WebDAV. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Include all connectors added by this PR in the changelog.
The bullet omits Databricks Files, Google Drive, and WebHDFS. Add them or explicitly make the wording non-exhaustive.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/changelog.md` around lines 6 - 7, Update the Filesystem changelog entry
to include Databricks Files, Google Drive, and WebHDFS, or revise its wording to
clearly indicate the list is non-exhaustive while preserving the existing
connector entries.
| r"https://github.com/", | ||
| r"https://web.archive.org/", | ||
| r"https://images.minimus.io/", | ||
| r"https://medium.com/", |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Changed lines files:"
git diff --stat || true
echo
if [ -f docs/conf.py ]; then
echo "docs/conf.py outline:"
ast-grep outline docs/conf.py || true
echo
echo "docs/conf.py relevant lines:"
nl -ba docs/conf.py | sed -n '80,110p'
fi
echo
echo "Search for medium ignore pattern:"
rg -n "medium\.com" docs/conf.py . || trueRepository: panodata/omniload
Length of output: 1399
🏁 Script executed:
#!/bin/bash
set -euo pipefail
if [ -f docs/conf.py ]; then
echo "docs/conf.py relevant linkcheck lines:"
sed -n '77,105p' docs/conf.py
fi
echo
echo "Medium references:"
rg -n "medium\.com|linkcheck_ignore|linkcheck_anchors_ignore_for_url" docs/conf.py . || trueRepository: panodata/omniload
Length of output: 1719
Narrow the Medium link-check ignore pattern.
r"https://medium.com/" skips validation for every Medium URL. Since the only in-repo Medium link is https://medium.com/@pavithrasainath7/mastering-file-access-in-sharepoint-with-oauth-2.0-a-comprehensive-guide-0a6b2d53736a, ignore that exact URL instead of the whole domain.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/conf.py` at line 98, Update the Medium URL entry in the link-check
ignore configuration to match only the specific in-repo article URL, replacing
the broad domain pattern while preserving other ignore entries.
| If your files are properly encoded but lack the correct file extension (BSON, | ||
| CSV, JSONL, or Parquet), you can provide a file type hint to inform `omniload` | ||
| about the format of the files. This is done by appending a fragment identifier | ||
| (`#format`) to the end of the path in your `--source-table` parameter. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Keep format-hint documentation consistent across remote sources.
The expanded shared readers support more formats than these source-specific lists expose, including CBOR.
docs/supported-sources/azure-storage.md#L121-L124: add the newer formats or state that the list is illustrative and link the canonical complete list.docs/supported-sources/s3.md#L94-L99: add#cborand the other supported hints, or link to the canonical complete list.
📍 Affects 2 files
docs/supported-sources/azure-storage.md#L121-L124(this comment)docs/supported-sources/s3.md#L94-L99
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/supported-sources/azure-storage.md` around lines 121 - 124, The
format-hint documentation is incomplete and inconsistent across the Azure
Storage and S3 source guides. Update the format-hint sections in
docs/supported-sources/azure-storage.md lines 121-124 and
docs/supported-sources/s3.md lines 94-99 to include CBOR and every currently
supported hint, or clearly mark the examples as illustrative and link to the
canonical complete format list; keep both source pages consistent.
| fs_kwargs = locator.options.fs_kwargs | ||
| fs_kwargs.update(kwargs) | ||
| apply_alias(fs_kwargs, "auth", "auth_kwargs") | ||
| cast_to_dict(fs_kwargs, ["creds", "auth_kwargs"]) | ||
| cast_to_bool(fs_kwargs, ["use_local_webserver"]) | ||
|
|
||
| # Create filesystem and dlt resource wrapper. | ||
| fs = self.fs_class(**fs_kwargs) | ||
| return infer_resource(fs=fs, locator=locator) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
gdrive-fsspec GoogleDriveFileSystem constructor parameters auth_kwargs creds use_local_webserver
💡 Result:
In the gdrive-fsspec library, the GoogleDriveFileSystem class provides several ways to handle authentication. Key constructor parameters related to your query include: creds: This parameter is used when employing service account authentication. It accepts a dictionary containing your service account credentials (equivalent to the JSON file downloaded from the Google Cloud Platform console) [1][2][3]. token: This parameter defines the authentication method. Common values include 'service_account', 'browser' (for initial user OAuth), 'cache' (for reusing stored credentials), and 'anon' (for public, read-only access) [1][2][3]. auth_kwargs: This is a dictionary of keyword arguments passed to the underlying authentication handler. A critical usage for this parameter is for headless or remote environments (such as CI/CD, containers, or SSH sessions) where a browser cannot be opened. By setting 'use_local_webserver': False within auth_kwargs, you can force the system to request an authentication token via the console instead of attempting to bind a local callback server [3]. Example usage for headless environments: fs = GoogleDriveFileSystem( token=token, auth_kwargs={'use_local_webserver': False},) [3]
Citations:
- 1: https://github.com/fsspec/gdrive-fsspec
- 2: https://github.com/fsspec/gdrive-fsspec/blob/master/README.md
- 3: https://github.com/intake/gdrivefs
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate gdrive connector =="
fd -a 'gdrive\.py$' . | sed 's#^\./##'
echo "== relevant source =="
file="$(fd 'gdrive\.py$' . | head -n 1 || true)"
if [ -n "${file:-}" ]; then
wc -l "$file"
sed -n '1,140p' "$file"
fi
echo "== inspect installed package metadata/source if present =="
python3 - <<'PY'
import sys, importlib.util
for name in ["gdrive_fsspec", "gdrive_fsspec.google_drive", "gdrive_fsspec.gdrive_fsspec"]:
spec = importlib.util.find_spec(name)
print(name, "found", bool(spec), "origin", getattr(spec, "origin", None))
PY
python3 - <<'PY'
try:
import inspect
import gdrive_fsspec
print("gdrive_fsspec module:", gdrive_fsspec.__file__)
print("GoogleDriveFileSystem:", getattr(gdrive_fsspec, "GoogleDriveFileSystem", None))
cls = gdrive_fsspec.GoogleDriveFileSystem
sig = inspect.signature(cls.__init__)
print("signature:", sig)
print("params:", list(sig.parameters.keys()))
except Exception as e:
print("failed:", type(e).__name__, e)
PY
echo "== GitHub raw source snippets =="
urls=(
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/gdrive_fsspec/gdrive_fsspec.py"
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/google_drivefs.py"
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/main/gdrive_fsspec/gdrive_fsspec.py"
)
for url in "${urls[@]}"; do
echo "--- $url"
if command -v curl >/dev/null 2>&1; then
curl -fsSL "$url" | sed -n '1,220p'
else
echo "curl unavailable"
fi
doneRepository: panodata/omniload
Length of output: 2281
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
import urllib.request
repos = ["fsspec/gdrive-fsspec", "fsspec/gdrive-fsspec"]
files = [
"gdrive_fsspec/gdrive_fsspec.py",
"gdrive_fsspec/google_drive.py",
"google_drive.py",
"google_drivefs.py",
]
for repo in repos:
print(f"--- repo {repo} tree top 200")
url = f"https://api.github.com/repos/{repo}/git/trees/main?recursive=1"
try:
with urllib.request.urlopen(url, timeout=20) as r:
data = json.load(r)
except Exception as e:
print("tree lookup failed:", type(e).__name__, e)
data = None
if data:
for blob in sorted(data.get("tree", []), key=lambda b: b.get("path", ""))[:200]:
p = blob.get("path")
if p.endswith(".py") and ("gdrive" in p.lower() or "google" in p.lower() or "oauth2client" in p.lower()):
print(blob.get("path"), blob.get("type"))
PY
python3 - <<'PY'
import re
import urllib.request
urls = [
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/main/gdrive_fsspec/gdrive_fsspec.py",
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/main/gdrive_fsspec/google_drive.py",
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/gdrive_fsspec/gdrive_fsspec.py",
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/gdrive_fsspec/google_drive.py",
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/google_drive.py",
"https://raw.githubusercontent.com/fsspec/gdrive-fsspec/master/google_drivefs.py",
]
pat = re.compile(r"\b(class\s+GoogleDriveFileSystem|def\s+__init__|auth_kwargs|creds|use_local_webserver|token=|service_account|browser|cache|anon)\b")
for url in urls:
try:
text = urllib.request.urlopen(url, timeout=20).read().decode("utf-8", "replace")
except Exception as e:
print("FETCH:", url, "FAILED", type(e).__name__, e)
continue
print("FETCH:", url, "OK", len(text))
for i, line in enumerate(text.splitlines(), 1):
if pat.search(line):
start=max(1,i-5); end=min(len(text.splitlines()),i+8)
lines=text.splitlines()
print(f"{url}:{i}: {line.strip()}")
for j in range(start,end):
print(f" {j}: {lines[j-1]}")
print()
PYRepository: panodata/omniload
Length of output: 2033
🌐 Web query:
site:github.com/fsspec/gdrive-fsspec "class GoogleDriveFileSystem" "auth_kwargs" "use_local_webserver"
💡 Result:
The GoogleDriveFileSystem class, as implemented in the gdrive-fsspec library, does not appear to support a parameter named use_local_webserver directly [1][2]. Authentication for GoogleDriveFileSystem is typically handled through the token argument [1][2]. The supported authentication methods include [1][2]: 1. Browser-based OAuth: Setting token='browser' triggers a flow that opens a browser window to complete the authentication process [1][2]. 2. Cached Tokens: Setting token='cache' allows the use of previously authenticated sessions [1][2]. 3. Service Account Credentials: Providing credentials and setting token='service_account' [1][2]. 4. Anonymous (read-only): Setting token='anon' for publicly accessible files [1][2]. If you are looking for local webserver-based OAuth authentication (often used in other Google Cloud Python client libraries to capture the redirect URL), this functionality is generally handled by the underlying authentication utility—typically pydata-google-auth in the case of gdrive-fsspec [3]—rather than as a direct parameter in the GoogleDriveFileSystem constructor [1][2]. If you encounter issues with the default authentication flow, it is often recommended to create your own Google Cloud Platform (GCP) project to manage your own OAuth credentials [3], rather than relying on the default shared credentials provided by the library [3].
Citations:
- 1: https://github.com/fsspec/gdrive-fsspec
- 2: https://github.com/fsspec/gdrive-fsspec/blob/master/README.md
- 3: "This app is blocked" on read_only access fsspec/gdrive-fsspec#28
Align Google Drive auth options with GoogleDriveFileSystem’s constructor signature.
gdrive-fsspec documents GoogleDriveFileSystem constructor auth options as token, creds, and auth_kwargs; use_local_webserver is nested inside auth_kwargs, so cast_to_bool(fs_kwargs, ["use_local_webserver"]) casts the wrong key.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/dlt_filesystem/source/fsspec/gdrive.py` around lines 40 - 48, Update the
filesystem option normalization in the source flow around fs_kwargs and
GoogleDriveFileSystem construction: remove the top-level use_local_webserver
boolean cast and ensure that option is handled within auth_kwargs, while
preserving the existing token, creds, and auth_kwargs mappings before
self.fs_class is instantiated.
| # Decode individual options (type casting, default values, sanity checks). Schema: | ||
| # {"default_block_size": int, "config": dict, "config_kwargs": dict, "oci_additional_kwargs": dict} | ||
| fs_kwargs = locator.options.fs_kwargs | ||
| fs_kwargs.update(kwargs) | ||
|
|
||
| # Decode dict-typed `config`, `config_kwargs`, `oci_additional_kwargs` from JSON. | ||
| cast_to_dict(fs_kwargs, ["config", "config_kwargs", "oci_additional_kwargs"]) | ||
| # The `default_` prefix seems unnecessary. Let's make it optional by using an alias. | ||
| apply_alias(fs_kwargs, "block_size", "default_block_size") | ||
| # Convert to integers. | ||
| cast_to_int(fs_kwargs, ["default_block_size"]) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
ocifs OCIFileSystem config parameter filepath vs dict
💡 Result:
In the ocifs library, the config parameter of the OCIFileSystem class accepts either a string (representing a file path) or a dictionary (representing a configuration object) to authenticate and connect to Oracle Cloud Infrastructure [1]. Key differences and usage: Filepath (str): You can provide the location of your OCI configuration file (e.g., "/.oci/config") as a string [1][2]. If provided as a file path, the library uses this file to load the connection details [1]. Dictionary (dict): You can provide a configuration dictionary, typically the object returned by the oci.config.from_file method from the OCI Python SDK [1][2]. This is useful if you have already loaded or processed the configuration in your code before initializing the filesystem [2]. If the config parameter is set to None, the library will attempt to use a Resource Principal or Instance Principal configured in your environment [1]. When using higher-level libraries like pandas or dask with OCI URLs (oci://), you pass these options via the storage_options dictionary, for example: storage_options={"config": "/.oci/config"} [2][3].
Citations:
- 1: https://ocifs.readthedocs.io/en/latest/ocifs.html
- 2: https://ocifs.readthedocs.io/en/latest/
- 3: https://ocifs.readthedocs.io/en/latest/getting-started.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate oci.py and cast/resolve helpers"
rg -n "def (cast_to_dict|cast_to_int|apply_alias)|cast_to_dict|apply_alias|class.*Options|fs_kwargs" src/dlt_filesystem source . 2>/dev/null || true
echo
echo "Relevant oci.py section"
if [ -f src/dlt_filesystem/source/fsspec/oci.py ]; then
nl -ba src/dlt_filesystem/source/fsspec/oci.py | sed -n '1,120p'
fi
echo
echo "Search definitions"
rg -n "def (cast_to_dict|cast_to_int|apply_alias)|def .*locator|locator =|dataclasses|options" src/dlt_filesystem/source/fsspec src/dlt_filesystem/source 2>/dev/null || trueRepository: panodata/omniload
Length of output: 17607
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Read relevant util.py helper implementations"
python3 - <<'PY'
from pathlib import Path
p=Path("src/dlt_filesystem/util/python.py")
lines=p.read_text().splitlines()
for name, start, end in [("cast_to_dict", 129, 144), ("apply_alias", 145, 162)]:
print(f"\n--- {p}:{name}:{start}-{end} ---")
for i in range(max(1,start-5), min(len(lines),end)+1):
print(f"{i:4d} {lines[i-1]}")
PY
echo
echo "Read model options"
python3 - <<'PY'
from pathlib import Path
p=Path("src/dlt_filesystem/source/model.py")
lines=p.read_text().splitlines()
for i in range(max(1,30), min(len(lines),75)+1):
print(f"{i:4d} {lines[i-1]}")
PY
echo
echo "Relevant oci.py source"
python3 - <<'PY'
from pathlib import Path
p=Path("src/dlt_filesystem/source/fsspec/oci.py")
lines=p.read_text().splitlines()
for i in range(max(1,1), min(len(lines),80)+1):
print(f"{i:4d} {lines[i-1]}")
PYRepository: panodata/omniload
Length of output: 5486
Don’t JSON-decode the OCI config string option.
ocifs.OCIFileSystem accepts config as either a filepath string or a parsed config dict, while config_kwargs/oci_additional_kwargs are dict-shaped. Including "config" in cast_to_dict() unconditionally runs json.loads() on values like "~/.oci/config", so oci://...?config=~/.oci/config raises JSONDecodeError.
🐛 Proposed fix
- cast_to_dict(fs_kwargs, ["config", "config_kwargs", "oci_additional_kwargs"])
+ # `config` may legitimately be a filepath string, not JSON — leave it untouched.
+ cast_to_dict(fs_kwargs, ["config_kwargs", "oci_additional_kwargs"])📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Decode individual options (type casting, default values, sanity checks). Schema: | |
| # {"default_block_size": int, "config": dict, "config_kwargs": dict, "oci_additional_kwargs": dict} | |
| fs_kwargs = locator.options.fs_kwargs | |
| fs_kwargs.update(kwargs) | |
| # Decode dict-typed `config`, `config_kwargs`, `oci_additional_kwargs` from JSON. | |
| cast_to_dict(fs_kwargs, ["config", "config_kwargs", "oci_additional_kwargs"]) | |
| # The `default_` prefix seems unnecessary. Let's make it optional by using an alias. | |
| apply_alias(fs_kwargs, "block_size", "default_block_size") | |
| # Convert to integers. | |
| cast_to_int(fs_kwargs, ["default_block_size"]) | |
| # Decode individual options (type casting, default values, sanity checks). Schema: | |
| # {"default_block_size": int, "config": dict, "config_kwargs": dict, "oci_additional_kwargs": dict} | |
| fs_kwargs = locator.options.fs_kwargs | |
| fs_kwargs.update(kwargs) | |
| # Decode dict-typed `config_kwargs`, `oci_additional_kwargs` from JSON. | |
| # `config` may legitimately be a filepath string, not JSON — leave it untouched. | |
| cast_to_dict(fs_kwargs, ["config_kwargs", "oci_additional_kwargs"]) | |
| # The `default_` prefix seems unnecessary. Let's make it optional by using an alias. | |
| apply_alias(fs_kwargs, "block_size", "default_block_size") | |
| # Convert to integers. | |
| cast_to_int(fs_kwargs, ["default_block_size"]) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/dlt_filesystem/source/fsspec/oci.py` around lines 37 - 47, Update the
option decoding near cast_to_dict and cast_to_int so cast_to_dict only
JSON-decodes config_kwargs and oci_additional_kwargs. Preserve config values as
either filepath strings or already-parsed dictionaries for OCIFileSystem.
| from typing import Type | ||
|
|
||
| from fsspec import AbstractFileSystem | ||
|
|
||
| from dlt_filesystem.source.base import FilesystemSource | ||
| from dlt_filesystem.source.core import infer_resource | ||
| from dlt_filesystem.source.model import FilesystemLocator | ||
| from dlt_filesystem.util.python import cast_to_bool, cast_to_int |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
fd -a 'smb.py|base.py|core.py|error.py' . | sed 's#^\./##'
echo
echo "smb.py excerpt:"
file="$(fd 'smb.py' . | head -n 1)"
if [ -n "${file:-}" ]; then
nl -ba "$file" | sed -n '1,120p'
fi
echo
echo "Relevant searches:"
rg -n "SMBFileSystem|MissingConnectorOption|host.*not in fs_kwargs|def infer_resource|class FilesystemLocator|smb://" .Repository: panodata/omniload
Length of output: 750
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "smb.py excerpt:"
sed -n '1,120p' src/dlt_filesystem/source/fsspec/smb.py
echo
echo "source error module excerpt:"
sed -n '1,160p' src/dlt_filesystem/source/error.py
echo
echo "SMB connector files and validation patterns:"
sed -n '1,120p' src/dlt_filesystem/source/fsspec/ftp.py 2>/dev/null || true
sed -n '1,120p' src/dlt_filesystem/source/fsspec/hdfs.py 2>/dev/null || true
echo
echo "SMBFileSystem references:"
rg -n "SMBFileSystem|MissingConnectorOption|host.*not in fs_kwargs|fs_kwargs" src/dlt_filesystem/source/fsspec src/dlt_filesystemRepository: panodata/omniload
Length of output: 23990
🌐 Web query:
fsspec SMBFileSystem constructor host required source host parameter
💡 Result:
Yes, the host parameter is a required argument for the fsspec SMBFileSystem constructor [1][2]. In the fsspec implementation for SMB, the init method is defined with host as the first positional argument [1][2]. Official documentation and source code indicate that the host parameter is used to specify the remote server name or IP address to connect to [1][2]. When using fsspec, it is recommended to provide a valid hostname or IP address for the host component. The documentation explicitly notes that you should not use the Windows/NetBIOS machine name for the host component [1][2]. While other parameters such as port, username, and password may be optional depending on your specific authentication setup (e.g., if using Kerberos or anonymous access), the host is necessary to identify the destination server [1][2].
Citations:
- 1: https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/smb.html
- 2: https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/smb.py
Validate the required SMB host before constructing the filesystem.
SMB is the only fsspec connector here that lets a hostless URI reach SMBFileSystem(**fs_kwargs), where host is required. Add the same missing-option guard used by FTP/HDFS before constructing the filesystem.
Proposed fix
+from dlt_filesystem.error import MissingConnectorOption
from dlt_filesystem.source.base import FilesystemSource
...
fs_kwargs = locator.options.fs_kwargs
fs_kwargs.update(kwargs)
+ if "host" not in fs_kwargs or not fs_kwargs["host"]:
+ raise MissingConnectorOption("host", "SMB")
cast_to_int(🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/dlt_filesystem/source/fsspec/smb.py` around lines 1 - 8, Validate that
the required SMB host option is present before constructing the filesystem in
the SMB source flow. Add the same missing-option guard used by the FTP/HDFS
connectors, returning the established error or validation result for a hostless
URI before calling SMBFileSystem with fs_kwargs.
| from typing import Type | ||
|
|
||
| from fsspec import AbstractFileSystem | ||
|
|
||
| from dlt_filesystem.source.base import FilesystemSource | ||
| from dlt_filesystem.source.core import infer_resource | ||
| from dlt_filesystem.source.model import FilesystemLocator | ||
| from dlt_filesystem.util.python import asbool, cast_to_bool, cast_to_dict, cast_to_int |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Parse and rebuild the WebDAV endpoint instead of mutating the full URI.
Line 29 also rewrites +dav inside file paths or query values. More importantly, Line 64 passes the option-bearing URI as base_url, so query credentials parsed into fs_kwargs can remain in the endpoint URL. Normalize only the scheme and remove query/fragment from base_url.
Proposed fix
from typing import Type
+from urllib.parse import urlsplit, urlunsplit
...
- uri = uri.replace("+webdav", "").replace("+dav", "")
+ parts = urlsplit(uri)
+ scheme = parts.scheme.removesuffix("+webdav").removesuffix("+dav")
+ uri = urlunsplit(parts._replace(scheme=scheme))
+ base_url = urlunsplit(
+ parts._replace(scheme=scheme, query="", fragment="")
+ )
...
- fs = self.fs_class(base_url=uri, auth=auth, **fs_kwargs)
+ fs = self.fs_class(base_url=base_url, auth=auth, **fs_kwargs)Also applies to: 25-29, 37-40, 63-65
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/dlt_filesystem/source/fsspec/webdav.py` around lines 1 - 8, Update the
WebDAV endpoint construction to parse the URI, normalize only its scheme from
+dav to dav, and rebuild base_url without query parameters or fragments.
Preserve the path unchanged, while passing parsed query options separately
through fs_kwargs so credentials and other options are not retained in the
endpoint URL.
| def test_asbool(): | ||
| """Validate the `asbool` utility function.""" | ||
| assert asbool("true") is True | ||
| assert asbool("yes") is True | ||
| assert asbool("false") is False | ||
| assert asbool("no") is False |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Cover invalid boolean inputs.
asbool is expected to reject unrecognized strings with ValueError; add cases such as "", "enabled", and "2" so connector-option parsing cannot regress into accepting arbitrary values. As per path instructions, “Prefer flagging missing edge-case coverage over style nits.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/dlt_filesystem/test_util.py` around lines 10 - 15, Extend test_asbool
with assertions that invalid strings such as "", "enabled", and "2" raise
ValueError, while preserving the existing true and false cases.
Source: Path instructions
About
Documentation
Preview: https://omniload--254.org.readthedocs.build/supported-sources/filesystem.html#supported-filesystems
References
oss://#180