storage: delete blobs faster#2972
Merged
pongad merged 2 commits intoFeb 28, 2018
googleapis:mastergoogleapis/google-cloud-java:masterfrom
pongad:deflake-gcsCopy head branch name to clipboard
Merged
storage: delete blobs faster#2972pongad merged 2 commits intogoogleapis:mastergoogleapis/google-cloud-java:masterfrom pongad:deflake-gcsCopy head branch name to clipboard
pongad merged 2 commits into
googleapis:mastergoogleapis/google-cloud-java:masterfrom
pongad:deflake-gcsCopy head branch name to clipboard
Conversation
Fixes #2962. The stack trace from the linked issue suggests we are deleting blobs too slowly, leading to timeouts. This PR tackles the problem in two ways. 1. Unless a test is actually testing blob deletion, the test is no longer responsible for cleaning up after itself. Blobs left over are batch-deleted at the end. 2. In case the batch-deletion fails, we set the bucket to auto-delete blobs over 1 day old. Then the clean up job will delete buckets over 2 days old. These buckets should already be empty and easy to delete.
pongad
commented
Feb 28, 2018
| EasyMock.expect(storageMock.delete(BLOB_ID1)).andReturn(true); | ||
| EasyMock.expect(blob2.getBlobId()).andReturn(BLOB_ID2); | ||
| EasyMock.expect(storageMock.delete(BLOB_ID2)).andReturn(true); | ||
|
|
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
garrettjonesgoogle
left a comment
Contributor
There was a problem hiding this comment.
LGTM, I just have one naming comment
| return options; | ||
| } | ||
|
|
||
| public static void cleanOldBuckets(final Storage storage, final long olderThan, long timeoutMs) { |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
chingor13
pushed a commit
that referenced
this pull request
Feb 20, 2026
…ng to v1.50.0 (#2972) Co-authored-by: Phong Chuong <147636638+PhongChuong@users.noreply.github.com>
chingor13
pushed a commit
that referenced
this pull request
Feb 20, 2026
🤖 I have created a release *beep* *boop* --- ## [3.14.1](https://togithub.com/googleapis/java-bigquerystorage/compare/v3.14.0...v3.14.1) (2025-05-15) ### Bug Fixes * **deps:** Update the Java code generator (gapic-generator-java) to 2.57.0 ([745bedb](https://togithub.com/googleapis/java-bigquerystorage/commit/745bedbd7f907361f89daa165ef9b361188c61a1)) ### Dependencies * Update dependency com.google.cloud:google-cloud-bigquery to v2.50.0 ([#2969](https://togithub.com/googleapis/java-bigquerystorage/issues/2969)) ([7a85ae2](https://togithub.com/googleapis/java-bigquerystorage/commit/7a85ae2c616f6c851f9d45b663be5ce8ba064a4e)) * Update dependency com.google.cloud:sdk-platform-java-config to v3.48.0 ([#2975](https://togithub.com/googleapis/java-bigquerystorage/issues/2975)) ([0f94876](https://togithub.com/googleapis/java-bigquerystorage/commit/0f94876d274968bb27c748f9a58062cfb4499e3b)) * Update dependency io.opentelemetry:opentelemetry-bom to v1.50.0 ([#2971](https://togithub.com/googleapis/java-bigquerystorage/issues/2971)) ([adafd79](https://togithub.com/googleapis/java-bigquerystorage/commit/adafd799d43b5fdea8c527432930069fe01f08f6)) * Update dependency io.opentelemetry:opentelemetry-exporter-logging to v1.50.0 ([#2972](https://togithub.com/googleapis/java-bigquerystorage/issues/2972)) ([a09a457](https://togithub.com/googleapis/java-bigquerystorage/commit/a09a457a4522da71c4abf255e3e1deb23364bc73)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
suztomo
pushed a commit
that referenced
this pull request
Mar 9, 2026
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
lqiu96
pushed a commit
that referenced
this pull request
Mar 20, 2026
…ng to v1.50.0 (#2972) Co-authored-by: Phong Chuong <147636638+PhongChuong@users.noreply.github.com>
lqiu96
pushed a commit
that referenced
this pull request
Mar 20, 2026
🤖 I have created a release *beep* *boop* --- ## [3.14.1](https://togithub.com/googleapis/java-bigquerystorage/compare/v3.14.0...v3.14.1) (2025-05-15) ### Bug Fixes * **deps:** Update the Java code generator (gapic-generator-java) to 2.57.0 ([cea2f6c](https://togithub.com/googleapis/java-bigquerystorage/commit/cea2f6c4bb22222b50d6538c707ac860da535cc7)) ### Dependencies * Update dependency com.google.cloud:google-cloud-bigquery to v2.50.0 ([#2969](https://togithub.com/googleapis/java-bigquerystorage/issues/2969)) ([51ba2e5](https://togithub.com/googleapis/java-bigquerystorage/commit/51ba2e50afc815dc3bfaa057a88d6b92f5a1237b)) * Update dependency com.google.cloud:sdk-platform-java-config to v3.48.0 ([#2975](https://togithub.com/googleapis/java-bigquerystorage/issues/2975)) ([a3850df](https://togithub.com/googleapis/java-bigquerystorage/commit/a3850df11a6be25491186b39fd2bc20db1e36846)) * Update dependency io.opentelemetry:opentelemetry-bom to v1.50.0 ([#2971](https://togithub.com/googleapis/java-bigquerystorage/issues/2971)) ([887d0d9](https://togithub.com/googleapis/java-bigquerystorage/commit/887d0d91558ed6f241e1e6bbf3378283f84b79aa)) * Update dependency io.opentelemetry:opentelemetry-exporter-logging to v1.50.0 ([#2972](https://togithub.com/googleapis/java-bigquerystorage/issues/2972)) ([cdae1af](https://togithub.com/googleapis/java-bigquerystorage/commit/cdae1af42a9a9422b280b9199d86714684e3890a)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
chingor13
pushed a commit
that referenced
this pull request
Mar 30, 2026
Usage of `Storage#format(String, Object...)` will use the default locale for it's formatting. This can lead to unexpected formatting if a right-to-left language such as Arabic is the default locale. Update all usages to use `String.format(Locale.US, pattern, Object...)` so we ensure things like headers or error messages are formatted according to the US conventions which match `en_US` and ascii byte conventions. Incidentally, things like right-to-left formatting seem to only apply to java11+ not java8. Fixes #2972
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2962.
The stack trace from the linked issue suggests we are deleting blobs too
slowly, leading to timeouts. This PR tackles the problem in two ways.
Unless a test is actually testing blob deletion, the test is no
longer responsible for cleaning up after itself. Blobs left over are
batch-deleted at the end.
In case the batch-deletion fails, we set the bucket to auto-delete
blobs over 1 day old. Then the clean up job will delete buckets over 2 days old.
These buckets should already be empty and easy to delete.