Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit f822fd7

Browse filesBrowse files
sinhasubhamsinhasubham
authored
feat: implement native asyncio support via Cross-Sync (#1509)
****Description: Native Asyncio Support**** This PR introduces comprehensive, native asyncio support to the google-cloud-spanner library. It transitions the library into a "Cross-Sync" architecture, where the asynchronous implementation serves as the source of truth, and the synchronous implementation is automatically kept in parity. ****Key Technical Changes**** Core Library Porting * Asynchronous API: Introduced AsyncClient, AsyncInstance, and AsyncDatabase classes. * Session Management: Completely refactored pool.py to support asyncio. Replaced threading.Lock and queue.Queue with their asyncio counterparts via the CrossSync abstraction. * Transactions & Snapshots: Native async implementation of run_in_transaction, including robust retry logic for Aborted exceptions and proper lock management. ****Verification & Testing**** 9 New System Tests: Created a dedicated async system test suite (tests/system/_async/) covering: Rich data types (Timestamp, JSON, Protobuf, etc.) Transaction retry loops Partitioned DML operations Session pool lifecycle 100% Pass Rate: All new async tests and existing sync tests pass reliably against the Spanner Emulator. Mock Server Updates: Added 40+ mock server tests to verify specific async behaviors like result-set iteration and error handling.
1 parent 00d5f8e commit f822fd7
Copy full SHA for f822fd7

208 files changed

+30,345-4,243Lines changed: 30345 additions & 4243 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Dismiss banner
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎packages/google-cloud-spanner/.agents/workflows/verify-asyncio.md‎

Copy file name to clipboard
+44Lines changed: 44 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
description: How to verify a Spanner Asyncio launch is ready.
3+
---
4+
# Spanner Asyncio Launch Verification Workflow
5+
6+
This workflow provides the necessary steps to verify that the Spanner Asyncio implementation is correct, stable, and ready for launch.
7+
8+
## 1. Run Async Unit Tests
9+
Run the complete suite of asynchronous unit tests across all supported Python versions.
10+
```bash
11+
nox -s unit
12+
```
13+
Ensure that all tests in `tests/unit/_async/` pass.
14+
15+
## 2. Run Async System Tests
16+
Verify the asynchronous behavior against the Spanner Emulator.
17+
// turbo
18+
```bash
19+
export SPANNER_EMULATOR_HOST="localhost:9010"
20+
export GCLOUD_PROJECT="emulator-test-project"
21+
export GOOGLE_CLOUD_TESTS_CREATE_SPANNER_INSTANCE="true"
22+
nox -s system -- tests/system/_async
23+
```
24+
**Note**: Ensure `pytest-asyncio` is installed in the system test environment.
25+
26+
## 3. Verify Sync/Async Parity
27+
Run the cross-sync generation tool and ensure no regressions in the synchronous codebase.
28+
```bash
29+
python3 .cross_sync/generate.py
30+
nox -s unit-3.14
31+
nox -s system-3.14
32+
```
33+
34+
## 4. Check for Coroutine Leaks
35+
Ensure all asynchronous GAPIC calls are properly awaited. Search for any unawaited coroutines in the `_async` directory.
36+
```bash
37+
grep -r "await " google/cloud/spanner_v1/_async | grep -v "async def"
38+
```
39+
40+
## 5. Verify Sample Code
41+
Verify that the provided samples work correctly.
42+
```bash
43+
python3 samples/async_samples.py
44+
```
Collapse file

‎packages/google-cloud-spanner/.coveragerc‎

Copy file name to clipboardExpand all lines: packages/google-cloud-spanner/.coveragerc
-40Lines changed: 0 additions & 40 deletions
This file was deleted.
Collapse file

‎packages/google-cloud-spanner/.cross_sync/README.md‎

Copy file name to clipboard
+75Lines changed: 75 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# CrossSync
2+
3+
CrossSync provides a simple way to share logic between async and sync code.
4+
It is made up of a small library that provides:
5+
1. a set of shims that provide a shared sync/async API surface
6+
2. annotations that are used to guide generation of a sync version from an async class
7+
8+
Using CrossSync, the async code is treated as the source of truth, and sync code is generated from it.
9+
10+
## Usage
11+
12+
### CrossSync Shims
13+
14+
Many Asyncio components have direct, 1:1 threaded counterparts for use in non-asyncio code. CrossSync
15+
provides a compatibility layer that works with both
16+
17+
| CrossSync | Asyncio Version | Sync Version |
18+
| --- | --- | --- |
19+
| CrossSync.Queue | asyncio.Queue | queue.Queue |
20+
| CrossSync.Condition | asyncio.Condition | threading.Condition |
21+
| CrossSync.Future | asyncio.Future | Concurrent.futures.Future |
22+
| CrossSync.Task | asyncio.Task | Concurrent.futures.Future |
23+
| CrossSync.Event | asyncio.Event | threading.Event |
24+
| CrossSync.Semaphore | asyncio.Semaphore | threading.Semaphore |
25+
| CrossSync.Awaitable | typing.Awaitable | typing.Union (no-op type) |
26+
| CrossSync.Iterable | typing.AsyncIterable | typing.Iterable |
27+
| CrossSync.Iterator | typing.AsyncIterator | typing.Iterator |
28+
| CrossSync.Generator | typing.AsyncGenerator | typing.Generator |
29+
| CrossSync.Retry | google.api_core.retry.AsyncRetry | google.api_core.retry.Retry |
30+
| CrossSync.StopIteration | StopAsyncIteration | StopIteration |
31+
| CrossSync.Mock | unittest.mock.AsyncMock | unittest.mock.Mock |
32+
33+
Custom aliases can be added using `CrossSync.add_mapping(class, name)`
34+
35+
Additionally, CrossSync provides method implementations that work equivalently in async and sync code:
36+
- `CrossSync.sleep()`
37+
- `CrossSync.gather_partials()`
38+
- `CrossSync.wait()`
39+
- `CrossSync.condition_wait()`
40+
- `CrossSync.event_wait()`
41+
- `CrossSync.create_task()`
42+
- `CrossSync.retry_target()`
43+
- `CrossSync.retry_target_stream()`
44+
45+
### Annotations
46+
47+
CrossSync provides a set of annotations to mark up async classes, to guide the generation of sync code.
48+
49+
- `@CrossSync.convert_sync`
50+
- marks classes for conversion. Unmarked classes will be copied as-is
51+
- if add_mapping is included, the async and sync classes can be accessed using a shared CrossSync.X alias
52+
- `@CrossSync.convert`
53+
- marks async functions for conversion. Unmarked methods will be copied as-is
54+
- `@CrossSync.drop`
55+
- marks functions or classes that should not be included in sync output
56+
- `@CrossSync.pytest`
57+
- marks test functions. Test functions automatically have all async keywords stripped (i.e., rm_aio is unneeded)
58+
- `CrossSync.add_mapping`
59+
- manually registers a new CrossSync.X alias, for custom types
60+
- `CrossSync.rm_aio`
61+
- Marks regions of the code that include asyncio keywords that should be stripped during generation
62+
63+
### Code Generation
64+
65+
Generation can be initiated using `nox -s generate_sync`
66+
from the root of the project. This will find all classes with the `__CROSS_SYNC_OUTPUT__ = "path/to/output"`
67+
annotation, and generate a sync version of classes marked with `@CrossSync.convert_sync` at the output path.
68+
69+
There is a unit test at `tests/unit/data/test_sync_up_to_date.py` that verifies that the generated code is up to date
70+
71+
## Architecture
72+
73+
CrossSync is made up of two parts:
74+
- the runtime shims and annotations live in `/google/cloud/aio/_cross_sync`
75+
- the code generation logic lives in `/.cross_sync/` in the repo root
Collapse file

‎packages/google-cloud-spanner/.cross_sync/generate.py‎

Copy file name to clipboard
+112Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Copyright 2024 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
from __future__ import annotations
15+
from typing import Sequence
16+
import ast
17+
"""
18+
Entrypoint for initiating an async -> sync conversion using CrossSync
19+
20+
Finds all python files rooted in a given directory, and uses
21+
transformers.CrossSyncFileProcessor to handle any files marked with
22+
__CROSS_SYNC_OUTPUT__
23+
"""
24+
25+
26+
def extract_header_comments(file_path) -> str:
27+
"""
28+
Extract the file header. Header is defined as the top-level
29+
comments before any code or imports
30+
"""
31+
header = []
32+
with open(file_path, "r", encoding="utf-8-sig") as f:
33+
for line in f:
34+
if line.startswith("#") or line.strip() == "":
35+
header.append(line)
36+
else:
37+
break
38+
header.append("\n# This file is automatically generated by CrossSync. Do not edit manually.\n\n")
39+
return "".join(header)
40+
41+
42+
class CrossSyncOutputFile:
43+
44+
def __init__(self, output_path: str, ast_tree, header: str | None = None):
45+
self.output_path = output_path
46+
self.tree = ast_tree
47+
self.header = header or ""
48+
49+
def render(self, with_formatter=True, save_to_disk: bool = True) -> str:
50+
"""
51+
Render the file to a string, and optionally save to disk
52+
53+
Args:
54+
with_formatter: whether to run the output through black before returning
55+
save_to_disk: whether to write the output to the file path
56+
"""
57+
full_str = self.header + ast.unparse(self.tree)
58+
if with_formatter:
59+
import black # type: ignore
60+
import autoflake # type: ignore
61+
62+
full_str = black.format_str(
63+
autoflake.fix_code(full_str, remove_all_unused_imports=True),
64+
mode=black.FileMode(),
65+
)
66+
if save_to_disk:
67+
import os
68+
os.makedirs(os.path.dirname(self.output_path), exist_ok=True)
69+
with open(self.output_path, "w") as f:
70+
f.write(full_str)
71+
return full_str
72+
73+
74+
def convert_files_in_dir(directory: str) -> set[CrossSyncOutputFile]:
75+
import glob
76+
from transformers import CrossSyncFileProcessor
77+
78+
# find all python files in the directory
79+
files = glob.glob(directory + "/**/*.py", recursive=True)
80+
# keep track of the output files pointed to by the annotated classes
81+
artifacts: set[CrossSyncOutputFile] = set()
82+
file_transformer = CrossSyncFileProcessor()
83+
# run each file through ast transformation to find all annotated classes
84+
for file_path in files:
85+
with open(file_path, encoding="utf-8-sig") as f:
86+
ast_tree = ast.parse(f.read())
87+
output_path = file_transformer.get_output_path(ast_tree)
88+
if output_path is not None:
89+
# contains __CROSS_SYNC_OUTPUT__ annotation
90+
converted_tree = file_transformer.visit(ast_tree)
91+
header = extract_header_comments(file_path)
92+
artifacts.add(CrossSyncOutputFile(output_path, converted_tree, header))
93+
# return set of output artifacts
94+
return artifacts
95+
96+
97+
def save_artifacts(artifacts: Sequence[CrossSyncOutputFile]):
98+
for a in artifacts:
99+
a.render(save_to_disk=True)
100+
101+
102+
if __name__ == "__main__":
103+
import sys
104+
105+
if len(sys.argv) < 2:
106+
print("Usage: python .cross_sync/generate.py <directory>")
107+
sys.exit(1)
108+
109+
search_root = sys.argv[1]
110+
outputs = convert_files_in_dir(search_root)
111+
print(f"Generated {len(outputs)} artifacts: {[a.output_path for a in outputs]}")
112+
save_artifacts(outputs)

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.