Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

chore: Script to inspect and clean up stale GCFs #331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 24, 2024

Conversation

shobsi
Copy link
Contributor

@shobsi shobsi commented Jan 19, 2024

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes internal issue 319307783 🦕

Usage:

$ python scripts/manage_cloud_functions.py --help
usage: manage_cloud_functions.py [-h] -p PROJECT_ID [-r REGIONS] {summary,cleanup} ...

Manage cloud functions created to serve bigframes remote functions.

options:
  -h, --help            show this help message and exit
  -p PROJECT_ID, --project-id PROJECT_ID
                        GCP project-id.
  -r REGIONS, --regions REGIONS
                        Cloud functions region(s). If multiple regions, Specify comma separated (e.g. region1,region2)

subcommands:
  {summary,cleanup}
    summary             BigFrames cloud functions summary.
    cleanup             BigFrames cloud functions clean up.


$ python scripts/manage_cloud_functions.py summary --help
usage: manage_cloud_functions.py summary [-h]

Show the bigframes cloud functions summary.

options:
  -h, --help  show this help message and exit


$ python scripts/manage_cloud_functions.py cleanup --help
usage: manage_cloud_functions.py cleanup [-h] [-n NUMBER]

Delete the stale bigframes cloud functions.

options:
  -h, --help            show this help message and exit
  -n NUMBER, --number NUMBER
                        Number of stale (more than a day old) cloud functions to clean up.
(venv) shobs@shobs-ct-3:~/code/bigframes1$ 

Example:

$ python scripts/manage_cloud_functions.py -p bigframes-dev summary
us-central1: Total=1412, Recent=86, OlderThanADay=1326
europe-west4: Total=270, Recent=24, OlderThanADay=246
southamerica-west1: Total=269, Recent=23, OlderThanADay=246
europe-west1: Total=262, Recent=23, OlderThanADay=239
asia-southeast1: Total=260, Recent=18, OlderThanADay=242
us-east1: Total=1, Recent=0, OlderThanADay=1

$ python scripts/manage_cloud_functions.py -p bigframes-dev -r us-central1,europe-west4 summary
us-central1: Total=1412, Recent=85, OlderThanADay=1327
europe-west4: Total=270, Recent=24, OlderThanADay=246

$ python scripts/manage_cloud_functions.py -p bigframes-dev -r us-central1,europe-west4 cleanup -n 2
[us-central1]: deleted [1] projects/bigframes-dev/locations/us-central1/functions/bigframes-597cc02ef5ce0525e4f51697b5a83b6c-3pfpu6gu last updated on 2024-01-08 21:47:58.503628+00:00
[us-central1]: deleted [2] projects/bigframes-dev/locations/us-central1/functions/bigframes-68f796a13666bb3bfe354dd1adaeef71 last updated on 2024-01-09 21:52:49.620259+00:00
[europe-west4]: deleted [1] projects/bigframes-dev/locations/europe-west4/functions/bigframes-558d0ca6649537a9e45896faf08b0a7a last updated on 2024-01-12 21:15:04.379198+00:00
[europe-west4]: deleted [2] projects/bigframes-dev/locations/europe-west4/functions/bigframes-4b7705561ec336ed80722a8e6e56ac41 last updated on 2024-01-08 05:34:59.331828+00:00

$ python scripts/manage_cloud_functions.py -p bigframes-dev -r us-central1,europe-west4 summary
us-central1: Total=1410, Recent=85, OlderThanADay=1325
europe-west4: Total=269, Recent=25, OlderThanADay=244

@shobsi shobsi requested review from a team as code owners January 19, 2024 01:27
@shobsi shobsi requested a review from ashleyxuu January 19, 2024 01:27
@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Jan 19, 2024
@shobsi shobsi requested a review from GarrettWu January 19, 2024 01:40
Copy link
Contributor

@GarrettWu GarrettWu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we set a TTL to the services?

How this script is scheduled? (I assume it doesn't need to be called manually.)

@shobsi shobsi added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 23, 2024
@bigframes-bot bigframes-bot removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 23, 2024
@shobsi
Copy link
Contributor Author

shobsi commented Jan 23, 2024

Can't we set a TTL to the services?

How this script is scheduled? (I assume it doesn't need to be called manually.)

Nope, there is no TTL support in GCFs. We can work out a scheduling story after this change. We could include this in the nightly script.

@shobsi shobsi added the automerge Merge the pull request once unit tests and other checks pass. label Jan 24, 2024
@gcf-merge-on-green gcf-merge-on-green bot merged commit 47c3285 into main Jan 24, 2024
@gcf-merge-on-green gcf-merge-on-green bot deleted the shobs-gcf-cleanup branch January 24, 2024 20:40
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: m Pull request size is medium.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.