SEA: re-fetch links in case of expiry #635
Draft
varun-edachali-dbx wants to merge 382 commits into
Draft
SEA: re-fetch links in case of expiry #635varun-edachali-dbx wants to merge 382 commits into
varun-edachali-dbx wants to merge 382 commits into
Commits
This pull request is big! We're only showing the most recent 250 commits
Commits on Jun 16, 2025
use more fetch methods
Show description for 5380c7acommittedremove redundant schema_bytes from parent constructor
Show description for 27b781fcommittedonly call get_chunk_link with non null chunk index
Show description for 238dc0acommittedalign SeaResultSet structure with ThriftResultSet
Show description for b3bb07ecommittedremvoe _fill_result_buffer from SeaResultSet
Show description for 13e6346committedreduce code repetition
Show description for f90b4d4committedalign SeaResultSet with ext-links-sea
Show description for 23963fccommittedremove redundant methods
Show description for dd43715committedupdate unit tests
Show description for 34a7f66committedremove accidental venv changes
Show description for 715cc13committed
Commits on Jun 17, 2025
pre-fetch next chunk link on processing current
Show description for fb53dd9committedreduce nesting
Show description for d893877committedline break after multi line pydoc
Show description for a165f1ccommittedre-introduce schema_bytes for better abstraction (likely temporary)
Show description for d68e4eacommitted- committed
add fetchmany_arrow and fetchall_arrow
Show description for a0705bccommitted- committed
remove accidental changes in sea backend tests
Show description for f7c11b9committed- committed
remove irrelevant changes
Show description for 6229848committedremove un-necessary test changes
Show description for fd52356committedremove un-necessary changes in thrift backend tests
Show description for 64e58b0committedremove unimplemented methods test
Show description for 2903473committed- committed
remove unimplemented method tests
Show description for 021ff4ccommittedmodify example scripts to include fetch calls
Show description for adecd53committed- committed
add GetChunksResponse
Show description for e3cef5ccommittedremove changes to sea test
Show description for ac50669committedre-introduce accidentally removed description extraction method
Show description for 03cdc4fcommittedfix type errors (ssl_options, CHUNK_PATH_WITH_ID..., etc.)
Show description for e1842d8committedaccess ssl_options through connection
Show description for 89a46afcommittedDEBUG level
Show description for 1d0b28bcommittedremove explicit multi chunk test
Show description for c8820d4committedmove cloud fetch queues back into utils.py
Show description for fe47787committedremove excess docstrings
Show description for 74f59b7committedmove ThriftCloudFetchQueue above SeaCloudFetchQueue
Show description for 4b456b2committedfix sea connector tests
Show description for bfc1f01committed- committed
correct patch module path in cloud fetch queue tests
Show description for 4883affcommittedremove unimplemented methods test
Show description for 0a2cdfdcommittedcorrect add_link docstring
Show description for cd3378ccommitted- committed
remove invalid import
Show description for cd22389committed- committed
- committed
- committed
- committed
Commits on Jun 18, 2025
better align queries with JDBC impl
Show description for 5ab9bbecommittedline breaks after multi-line PRs
Show description for 1ab6e87committedremove unused imports
Show description for f469c24committedfix: introduce ExecuteResponse import
Show description for 68ec65fcommitted- committed
remove unimplemented metadata methods test, un-necessary imports
Show description for f6d873dcommittedintroduce unit tests for metadata methods
Show description for 28675f5committed
Commits on Jun 20, 2025
remove verbosity in ResultSetFilter docstring
Show description for 3578659andauthoredremove un-necessary info in ResultSetFilter docstring
Show description for 8713023committedremove explicit type checking, string literals around forward annotations
Show description for 22dc252committedhouse SQL commands in constants
Show description for 390f592committed
Commits on Jun 23, 2025
convert complex types to string if not _use_arrow_native_complex_types
Show description for dd7dc6acommitted- committed
introduce unit tests for altered functionality
Show description for 2712d1ccommitted- committed
Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
Show description for 48ad7b3committedreduce verbosity of ResultSetFilter docstring
Show description for a1f9b9ccommittedremove unused imports
Show description for 984e8eecommitted- committed
Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
Show description for c313c2bcommittedRevert "reduce verbosity of ResultSetFilter docstring"
Show description for 3bc615ecommittedReapply "Merge branch 'fetch-json-inline' into ext-links-sea"
Show description for b6e1a10committedRevert "Merge branch 'fetch-json-inline' into ext-links-sea"
Show description for 2df3d39committedremove un-necessary filters changes
Show description for 5e75fb5committedremove un-necessary backend changes
Show description for 20822e4committedremove constants changes
Show description for 802d045committedremove changes in filters tests
Show description for f3f795acommittedremove unit test backend and JSON queue changes
Show description for f6c5950committedremove changes in sea result set testing
Show description for d210ccdcommittedRevert "remove changes in sea result set testing"
Show description for 22a953ecommittedRevert "remove unit test backend and JSON queue changes"
Show description for 3aed144committedRevert "remove changes in filters tests"
Show description for 0fe4da4committedRevert "remove constants changes"
Show description for 0e3c0a1committedRevert "remove un-necessary backend changes"
Show description for 93edb93committedRevert "remove un-necessary filters changes"
Show description for 871a44fcommittedremove unused imports
Show description for 0ce144dcommitted- committed
working version
Show description for 8c5cc77committedadopy _wait_until_command_done
Show description for 7f5c715committedintroduce metadata commands
Show description for 9ef5fadcommitteduse new backend structure
Show description for 44183dbcommittedconstrain backend diff
Show description for d59b351committedremove changes to filters
Show description for 1edc80acommittedmake _parse methods in models internal
Show description for f82658acommittedreduce changes in unit tests
Show description for 54eb0a4committed
Commits on Jun 24, 2025
run small queries with SEA during integration tests
Show description for 50cc1e2committedrun some tests for sea
Show description for 242307acommitted
Commits on Jun 25, 2025
allow empty schema bytes for alignment with SEA
Show description for 8a138e8committedpass is_vl_op to Sea backend ExecuteResponse
Show description for 82f9d6bcommitted
Commits on Jun 26, 2025
remove catalog requirement in get_tables
Show description for 35f1ef0committedmove filters.py to SEA utils
Show description for a515d26committedensure SeaResultSet
Show description for 59b1330committed- committed
prevent circular imports
Show description for dd40bebcommittedremove unused imports
Show description for 14057accommittedremove cast, throw error if not SeaResultSet
Show description for a4d5bdbcommitted- committed
pass param as TSparkParameterValue
Show description for eb1a9b4committed- committed
remove failing test (temp)
Show description for a3ca7c7committedremove SeaResultSet type assertion
Show description for 2c22010committedchange errors to align with spec, instead of arbitrary ValueError
Show description for c09508ecommittedmake SEA backend methods return SeaResultSet
Show description for e9b1314committeduse spec-aligned Exceptions in SEA backend
Show description for 8ede414committedremove defensive row type check
Show description for 09a1b11committed- committed
- committed
raise ProgrammingError for invalid id
Show description for a026d31committedmake is_volume_operation strict bool
Show description for 4446a9ecommittedremove complex types code
Show description for 138359dcommittedRevert "remove complex types code"
Show description for b99d0c4committed
Commits on Jun 27, 2025
introduce type conversion for primitive types for JSON + INLINE
Show description for 21c389dcommitted- committed
remove SEA running on metadata queries (known failures
Show description for 9f0f969committedremove un-necessary docstrings
Show description for 04a1936committedalign expected types with databricks sdk
Show description for 278b8cdcommittedlink rest api reference to validate types
Show description for 91b7f7fcommittedremove test_catalogs_returns_arrow_table test
Show description for 7a5ae13committedfix fetchall_arrow and fetchmany_arrow
Show description for f1776f3committedremove thrift aligned test_cancel_during_execute from SEA tests
Show description for 6143331committed- committed
remove un-necessary changes in example scripts
Show description for 5eaded4committedremove un-necessary chagnes in example scripts
Show description for eeed9a1committed_convert_json_table -> _create_json_table
Show description for f233886committedremove accidentally removed test
Show description for 68ac437committedremove new unit tests (to be re-added based on new arch)
Show description for 7fd0845committedremove changes in sea_result_set functionality (to be re-added)
Show description for ea7ff73committedintroduce more integration tests
Show description for 563da71committedremove SEA tests in parameterized queries
Show description for a018273committedremove partial parameter fix changes
Show description for c0e98f4committedremove un-necessary timestamp tests
Show description for 7343035committedslightly stronger typing of _convert_json_types
Show description for ec500b6committedstronger typing of json utility func s
Show description for 0b3e91dcommittedstronger typing of fetch*_json
Show description for 7664e44committedremove unused helper methods in SqlType
Show description for db7b8e5committedline breaks after multi line pydocs, remove excess logs
Show description for f75f2b5committedline breaks after multi line pydocs, reduce diff of redundant changes
Show description for e2d4ef5committedreduce diff of redundant changes
Show description for 21e3078committedmandate ResultData in SeaResultSet constructor
Show description for bb015e6committed- committed
remove complex type conversion
Show description for b3273c7committedcorrect fetch*_arrow
Show description for 38c2b88committed
Commits on Jul 3, 2025
- committed
recover old sea tests
Show description for fa2359dcommittedmove queue and result set into SEA specific dir
Show description for c07f709committedpass ssl_options into CloudFetchQueue
Show description for 9e4ef2ecommittedreduce diff
Show description for b00c06ccommittedremove redundant conversion.py
Show description for 10f55f0committedfix type issues
Show description for cd119e9committedValueError not ProgrammingError
Show description for d79638bcommittedreduce diff
Show description for f84578acommittedintroduce SEA cloudfetch e2e tests
Show description for c621c0ccommittedallow empty cloudfetch result
Show description for 7958cd9committedadd unit tests for CloudFetchQueue and SeaResultSet
Show description for e2d17ffcommittedskip pyarrow dependent tests
Show description for d348b35committed- committed
Commits on Jul 4, 2025
simplify download process: no pre-fetching
Show description for 4bd290ecommittedcorrect class name in logs
Show description for dfbbf79committed
Commits on Jul 7, 2025
- committed
align with old impl
Show description for a5e9cdfcommittedalign next_n_rows with prev imple
Show description for be16634committedalign remaining_rows with prev impl
Show description for 6ec8656committedremove un-necessary Optional params
Show description for 7ea7b75committedremove un-necessary changes in thrift field if tests
Show description for 64be07bcommittedremove unused imports
Show description for 165644ccommitted
Commits on Jul 8, 2025
- committed
run large queries
Show description for abef941committedhybrid disposition
Show description for e74ccd1committedremove un-ncessary log
Show description for 06a5d54committedformatting (black)
Show description for 9cf6e02committedremove redundant tests
Show description for b5578cacommitted
Commits on Jul 9, 2025
multi frame decompression of lz4
Show description for 7f67cfccommittedensure no compression (temp)
Show description for 2202057committedintroduce separate link fetcher
Show description for 92fe5fdcommittedlog time to create table
Show description for 54b43facommittedadd chunk index to table creation time log
Show description for 2750c5ecommittedremove custom multi-frame decompressor for lz4
Show description for 239cd8dcommitted- committed
Commits on Jul 10, 2025
remove redundant tests (temp)
Show description for 6894b4ccommittedadd link to download manager before notifying consumer
Show description for 4abb3adcommitted
Commits on Jul 11, 2025
move link fetching immediately before table creation so link expiry is not an issue
Show description for fce324bcommitted- committed
- committed
resolve merge artifacts
Show description for 9590af7committedremove redundant methods
Show description for 82fc0b6committed- committed
formatting (black)
Show description for b2d1579committedintroduce callback to handle link expiry
Show description for bd51b1ccommittedfix types
Show description for 077a71ccommittedfix param type in unit tests
Show description for 7985639committed
Commits on Jul 12, 2025
- committed
- committed
formatting + minor type fixes
Show description for 0868fe3committedRevert "introduce callback to handle link expiry"
Show description for 3f42a03committedremove unused callback (to be introduced later)
Show description for d038d84committed
Commits on Jul 14, 2025
- committed
correct param extraction
Show description for 0a0643bcommittedinit sea link retry func
Show description for f374f5fcommittedcorrect unit tests for download management
Show description for 0829e67committedcreate strongly typed Future for download tasks
Show description for a18be78committedremove common constructor for databricks client abc
Show description for f7fd1d9committedmake SEA Http Client instance a private member
Show description for 510b0a3committedmake GetChunksResponse model more robust
Show description for dd2864bcommittedadd link to doc of GetChunk response model
Show description for c32b281committedpass result_data instead of "initial links" into SeaCloudFetchQueue
Show description for 0b1eba5committedmove download_manager init into parent CloudFetchQueue
Show description for 777f7c1committedraise ServerOperationError for no 0th chunk
Show description for 130b0d3committedunused iports
Show description for 1920375committed
Commits on Jul 15, 2025
- committed
return None in case of empty respose
Show description for 5a43686committedensure table is empty on no initial link s
Show description for 28c6bb1committed- committed
account for total chunk count
Show description for 5791745committed
Commits on Jul 16, 2025
iterate by chunk index instead of link
Show description for 2701e5dcommitted- committed
Commits on Jul 17, 2025
- committed
make LinkFetcher convert link static
Show description for f56497fcommittedadd helper for link addition, check for edge case to prevent inf wait
Show description for 76fd190committedadd unit tests for LinkFetcher
Show description for ab2c2c4committedremove un-necessary download manager check
Show description for 3af2333committedremove un-necessary string literals around param type
Show description for 480528ecommittedremove duplicate download_manager init
Show description for fba1812committed
Commits on Jul 19, 2025
Commits on Jul 20, 2025
account for empty response in LinkFetcher init
Show description for 23dba80committedmake get_chunk_link return mandatory ExternalLink
Show description for 26aa45ccommittedset shutdown_event instead of breaking on completion so get_chunk_link is informed
Show description for 9fd57d8committed
Commits on Jul 21, 2025
docstrings, logging, pydoc
Show description for dd009dfcommitteduse total_chunk_cound > 0
Show description for 7c360d1committedclarify that link has already been submitted on getting row_offset
Show description for 6b1b972committedreturn None for out of range
Show description for 2ca7c2bcommitteddefault link_fetcher to None
Show description for 00db613committed- committed
reduce repeated init
Show description for 2cd802ecommitted- committed
- committed
add_link -> add_links for less heavy link additions
Show description for b114599committed- committed
move param order
Show description for 92821e7committedtype issues
Show description for 63deb04committed- committed
Commits on Aug 6, 2025
- committed
introduce create_download_handler
Show description for ac04e66committed