[PECO-1532] Ignore the excess records in query results#239
Merged
kravets-levko merged 5 commits intoMar 27, 2024
maindatabricks/databricks-sql-nodejs:mainfrom
PECO-1532-ignore-excess-recordsdatabricks/databricks-sql-nodejs:PECO-1532-ignore-excess-recordsCopy head branch name to clipboard
Merged
[PECO-1532] Ignore the excess records in query results#239kravets-levko merged 5 commits intomaindatabricks/databricks-sql-nodejs:mainfrom PECO-1532-ignore-excess-recordsdatabricks/databricks-sql-nodejs:PECO-1532-ignore-excess-recordsCopy head branch name to clipboard
kravets-levko merged 5 commits into
maindatabricks/databricks-sql-nodejs:mainfrom
PECO-1532-ignore-excess-recordsdatabricks/databricks-sql-nodejs:PECO-1532-ignore-excess-recordsCopy head branch name to clipboard
Conversation
…th raw batch data Signed-off-by: Levko Kravets <levko.ne@gmail.com>
Signed-off-by: Levko Kravets <levko.ne@gmail.com>
Signed-off-by: Levko Kravets <levko.ne@gmail.com>
Signed-off-by: Levko Kravets <levko.ne@gmail.com>
benc-db
approved these changes
Mar 26, 2024
benc-db
left a comment
Contributor
There was a problem hiding this comment.
Looks reasonable to me, but I'm neither a node nor thrift semantics expert.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PECO-1532
Note for reviewers: This PR contains some refactoring needed to implement the fix, so probably it's easier to review commit by commit
When client library executes query and wants an Arrow-based or Cloudfetch results - server will return records as Arrow batches. Batch size may vary, server makes the decision on that depending on count of records, record size, etc. But usually all batches will have the same size, with the only exception - the last batch, which usually contains less records. And there are two possibilities:
rowCountfield which defines how may "valid" records are in the batch. Client should use only that records and discard the remaining ones.(I guess that different workspaces may be configured differently and will behave either as described in scenario 1 or 2)
Nodejs connector doesn’t use value from
rowCountand therefore returns that extra records to user. This behavior is wrong, and this PR fixes it.