Issue Summary

On a BigQuery data source, the schema browser becomes completely empty (no tables, no columns, no autocomplete) when the service account cannot access every dataset in the project. This happens even for the datasets the account can read.

The root cause is in get_schema() (redash/query_runner/big_query.py):

• It lists all datasets in the project via datasets().list().
• It builds a single UNION ALL query over each dataset's INFORMATION_SCHEMA.COLUMN_FIELD_PATHS (and a second one over INFORMATION_SCHEMA.TABLE_OPTIONS).
• If any one dataset returns an error, _handle_run_query_error() raises and the whole refresh aborts. Nothing is cached.

So a single inaccessible dataset wipes out the schema for the entire data source.

This is common with dataset-level IAM (per-team isolation): the service account is granted only a subset of datasets, plus roles/bigquery.jobUser. Querying the datasets it cannot read returns:

403 PERMISSION_DENIED
Access Denied: Table PROJECT:DATASET.INFORMATION_SCHEMA.TABLE_OPTIONS:
User does not have permission to query table
PROJECT:DATASET.INFORMATION_SCHEMA.TABLE_OPTIONS, or perhaps it does not exist.

Why existing mechanisms do not help:

• Before Speed up BigQuery get schema operation in massively tables environment #5632, columns were fetched per-table via the metadata API, which tolerated inaccessible datasets. The UNION ALL approach introduced there is all-or-nothing.
• The location filter from bigquery load schema diff locations ignore #7289 only excludes datasets in a different location than the one configured. When every dataset shares one location (a single region), it excludes nothing and the schema is still empty.

Steps to Reproduce

1. Use a BigQuery project with multiple datasets, all in the same location (e.g. asia-northeast1).
2. Grant a service account read access to only some of those datasets, plus roles/bigquery.jobUser.
3. Configure a Redash BigQuery data source with that account and "Load Schema" enabled.
4. Trigger a schema refresh (periodic refresh_schemas, or open the query editor).

• Expected: the schema browser shows the tables of the accessible datasets.
• Actual: the schema browser is empty. The refresh_schema job fails with a 403 PERMISSION_DENIED on INFORMATION_SCHEMA.TABLE_OPTIONS / COLUMN_FIELD_PATHS of an inaccessible dataset.

Note: query execution is unaffected. Only schema loading and autocomplete break.

Possible fix

Make get_schema() tolerant of per-dataset failures:

• Run the metadata query per dataset (or per location) and skip datasets that error out, instead of aborting the whole refresh.
• The number of calls stays bounded by the dataset count, not the table count, so the performance concern behind Speed up BigQuery get schema operation in massively tables environment #5632 does not return.

Technical details:

• Redash Version: 26.3.0 (official redash/redash:26.3.0 image)
• Browser/OS: N/A (server-side schema refresh; reproduced via RQ worker logs)
• How did you install Redash: Docker (official image)