…sync cancel (#825)

* feat(kernel): wire CUJ-gap fixes — staging fail-loud, error context, sync cancel

Connector half of the API+CUJ audit fixes (kernel half: databricks-sql-kernel
PR #121). Bumps KERNEL_REV to pick up the kernel surface.

Staging fail-loud (kernel/client.py):
- Volume/staging PUT/GET/REMOVE silently no-op'd on the kernel path
  (KernelResultSet.is_staging_operation is always False, so the
  connector's _handle_staging_operation never fired and no file was
  transferred). Detect the leading verb in execute_command and raise
  NotSupportedError so ETL fails loud instead of ingesting stale data.

Error context (_errors.py):
- Forward display_message / diagnostic_info / error_details_json (now
  exposed across the pyo3 boundary in #121) onto the re-raised PEP-249
  exception, and populate ServerOperationError.context with
  "diagnostic-info" (Spark stack trace) + "operation-id" — matching
  the Thrift backend so callers reading err.context work identically.

Sync cancel wiring (client.py, kernel/client.py):
- cursor.cancel() was a silent no-op for the default blocking
  execute() (active_command_id is None until execute returns). The
  kernel backend now registers a detached StatementCanceller (keyed by
  the cursor) before the blocking execute and exposes
  cancel_running_cursor(cursor). Cursor.cancel() routes to that hook
  via getattr when there's no command id yet — opt-in, so Thrift/SEA
  backends are unaffected.

Tests: unit (staging fail-loud, _is_staging_statement, cancel registry
+ routing, error context/diagnostic-info forwarding) and e2e
(tz-aware TIMESTAMP, scientific DECIMAL, staging NotSupportedError,
diagnostic-info context, cross-thread sync cancel interrupts a running
query). All e2e verified live against dogfood with use_kernel=True.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

* fix(kernel): address review — comment-prefixed staging + tolerant sync cancel

Review fixes on PR #825 (+ KERNEL_REV bump to the amended kernel #121
which now folds the original message on attach failure and bounds
error_details_json):

P1 #1 — staging fail-loud missed comment-prefixed statements:
_is_staging_statement took the first whitespace token without
stripping SQL comments, so "-- upload\nPUT ..." / "/* c */ PUT ..."
(common in ETL) classified as non-staging and slipped into the
silent-no-op bug. Added _strip_leading_sql_comments (handles leading
-- line and /* */ block comments, multiple/mixed) before extracting
the verb. Tests for both comment forms, mixed, and verb-only-in-comment
(must NOT match).

P1 #2 — sync cancel could raise out of cursor.cancel():
cursor.cancel() is best-effort per PEP-249, but cancel_running_cursor
re-raised a canceller failure (e.g. an early cancel before the server
statement id is observed, or a transport hiccup) via the public
cancel(). Now swallow+log and still return True (a canceller was
present and attempted) so Cursor doesn't emit the misleading "no
executing command" warning.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

* chore: re-pin KERNEL_REV to merged kernel #121 (cbeaf44)

#121 (tz-aware/scientific param binds, error context, sync cancel +
Ctrl-C) is merged to kernel main. Re-pin from the orphaned branch HEAD
(f62d941) to the merged squash SHA (cbeaf44) — content-identical, but
reachable from main so no orphan-SHA risk. Verified against a wheel
built from cbeaf44: connector unit + kernel e2e (tz-aware TIMESTAMP,
scientific DECIMAL, staging fail-loud incl. comment-prefixed,
diagnostic-info context) all pass.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

---------

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

test(e2e/mst): DESCRIBE QUERY now allowed in transactions

The server's MSTCheckRule allowlist has broadened to include DESCRIBE
QUERY (DescribeQueryCommand), mirroring the earlier SHOW COLUMNS change.
It no longer throws inside an active transaction, so the prior
test_describe_query_blocked assertion (DID NOT RAISE) was stale.

Flip it to test_describe_query_not_blocked using _assert_not_blocked
(verifies it succeeds and returns >0 rows) and move DESCRIBE QUERY from
the Blocked to the Allowed list in the class docstring.

Verified against a live DBSQL warehouse: the full TestMstBlockedSql
class (9 tests) passes.

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

#824)

* feat(kernel): surface kernel logs through Python logging on use_kernel

When the kernel backend loads, auto-initialize the kernel's
tracing -> Python logging bridge so `use_kernel=True` users see kernel
logs with no extra setup. Kernel logs land under the
`databricks.sql.kernel` logger (a child of the connector's
`databricks.sql.*` namespace), so an existing
`logging.getLogger("databricks.sql").setLevel(...)` cascades to them.

- `_errors.py` calls `databricks_sql_kernel.init_logging()` once at
  extension load (it's the canonical kernel-import site). The call is
  `getattr`-guarded so an older kernel wheel without the function still
  works — just without kernel logs.
- e2e tests assert kernel records reach the `databricks.sql.kernel`
  logger (and the pyo3 boundary under `databricks.sql.kernel.pyo3`) and
  that the level set on the logger is respected. Creds-gated per the
  existing kernel e2e convention.

Requires the companion kernel/pyo3 change
(databricks-sql-kernel#120) that exposes `init_logging()`.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

* fix(kernel): harden log-bridge init + address review feedback

Review feedback from PR #824 (gopalldb):

- P1: guard the init_logging() call against throwing, not just a missing
  function. A panic across the PyO3 boundary surfaces as
  pyo3_runtime.PanicException (a BaseException, not Exception), so a bare
  call could escape module import and fail every use_kernel=True
  connection over a non-essential logging feature. Wrap in
  try/except BaseException, log a debug breadcrumb, continue. This also
  neutralizes the idempotency concern regardless of the Rust impl.
- P2: soften the level-control test docstring to make clear it asserts
  the effective customer-visible outcome (sub-threshold records don't
  surface), not source-side suppression — caplog filters after the FFI.
- P2: downgrade the databricks.sql.kernel.pyo3 assertion to a soft
  warning so a benign kernel change to the boundary breadcrumb target
  doesn't break the connector e2e suite. The core
  databricks.sql.kernel contract is still hard-asserted.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

* chore: bump KERNEL_REV to merged kernel main (f4ee6fe) for the logging bridge

The kernel tracing -> Python logging bridge (init_logging / pyo3-log,
routing under databricks.sql.kernel) landed in kernel #120 — AFTER the
previously-pinned 101aa46 (#118). The kernel-e2e built a wheel without
the bridge, so test_kernel_logs_reach_python_logging failed with
'no records delivered' (assert []).

Bump to f4ee6fe (current kernel main: includes #118, #120, #123, #125).
Verified live against a wheel built from f4ee6fe: the bridge delivers
records to the databricks.sql.kernel logger, and both
test_kernel_logs_reach_python_logging and
test_kernel_log_level_is_respected pass.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

---------

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

…V to batch 2 (#830)

* fix(kernel): stop premature sync statement close (H4); bump KERNEL_REV to batch 2

Connector half of the kernel batch-2 fixes (kernel PR #123). Bumps
KERNEL_REV to pick up the batch-2 kernel surface.

H4 — don't close the kernel Statement at sync execute-return:
execute_command's finally used to `stmt.close()` immediately after
`stmt.execute()` succeeded. For a large CloudFetch result with
paginated chunk links (all_fetched=false), the kernel fetches later
links lazily (get_result_chunks against the LIVE statement) during
consumption, so a premature CloseStatement broke those fetches. The
kernel now auto-closes the server statement when its result stream
drains (ExecutedStatement::next_batch end-of-stream), with the
executed-handle Drop as the backstop for partial/abandoned reads. So
the connector now flips close_stmt=False on a successful execute and
only closes on the error path (no executed handle was produced).

The other batch-2 fixes (cancelled-class -> OperationalError, U2M
refresh fail-fast, metadata statement close-on-drop, per-binding OAuth
client_id) are entirely kernel-side and need no connector code beyond
the KERNEL_REV bump.

Tests: unit (sync execute does-not-close on success / does-close on
failure) + e2e (large multi-chunk result drains without premature
close + cursor reuse; server cancel maps to OperationalError not
ProgrammingError). All e2e verified live against dogfood.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

* chore: re-pin KERNEL_REV to #123 HEAD after cancelled-test fix

#123 picked up a follow-up commit fixing the wiremock cancelled-state
assertions (ErrorCode::Cancelled). Bump the placeholder pin so the
connector CI builds the corrected kernel. Still to be re-pinned to the
squash-merge SHA before #830 merges.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

* chore: re-pin KERNEL_REV to merged kernel #123 (f4ee6fe)

Kernel batch-2 (#123) is merged to kernel main (f4ee6fe, current main
HEAD). Re-pin from the orphaned branch HEAD (4f7fbe7) to the merged
SHA — reachable from main, no orphan-SHA risk. Verified against a
wheel built from f4ee6fe: connector unit (102) + kernel e2e (H4
large-result drain + reuse, server-cancel -> OperationalError, staging
fail-loud, diagnostic-info) all pass against the real merged kernel.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

* docs: drop internal 'H4' audit shorthand from comments/docstrings

Address review: 'H4' is internal audit shorthand, meaningless in the
public connector codebase. Reword the affected comment + two test
docstrings to describe the behavior directly (premature sync
CloseStatement broke lazy CloudFetch chunk-link fetches). No code change.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

---------

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>