Extract test script #1476

wisechengyi · Nov 4, 2025

With a later goal to have a single command to run tests locally as well.

Problem

Duplicated build logic across CI workflows: The build and test steps were duplicated across both ci-backend-2.12.yml and ci-backend-2.13.yml workflows, with identical multi-line bash scripts. This
violates DRY principles, makes maintenance difficult, and uses the slower pip for Python dependency installation.

Solution

Extracted build logic into reusable script

Created .github/scripts/build-and-test.sh:

Single parameterized script that takes Scala version as argument
Replaces duplicated inline bash code in both workflow files
Uses uv instead of pip for significantly faster Python dependency installation
Creates and activates a Python virtual environment in .venv for better isolation
Auto-installs uv if not present
Includes proper error handling with set -euo pipefail
Can be tested locally: ./build-and-test.sh 2.12.12

Simplified CI workflows

Both ci-backend-2.12.yml and ci-backend-2.13.yml now use the same script:

- name: Build
  run: ./.github/scripts/build-and-test.sh <scala-version>

Benefits

DRY code: Build logic defined once, reused across both Scala 2.12 and 2.13 workflows
Faster CI: uv provides 10-100x faster package installation compared to pip (eyeballing reduced to 6 min from 8 min total runtime)
Easier maintenance: Changes to build process only need to be made in one place
Better debuggability: Script can be run locally to reproduce CI issues
Better isolation: Uses virtual environment for Python dependencies

wisechengyi · Nov 5, 2025

Spark 3.5 Support #1479
Use JDK 17 runtime #1471
Bump Spark to 3.3 #1478
Extract test script #1476 👈 (View in Graphite)
Shade Google dependencies in assembly #1477
Fix spark download deadlock #1475
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copilot

Pull Request Overview

This PR extracts duplicated build and test logic from two CI workflow files into a single reusable bash script, addressing DRY principles and improving CI performance by switching from pip to uv for Python dependency installation.

Key Changes:

Created .github/scripts/build-and-test.sh script that consolidates build logic with parameterized Scala version
Replaced inline bash scripts in both CI workflows with calls to the new script
Switched from pip to uv for faster Python package installation with virtual environment isolation

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
`.github/workflows/ci-backend-2.13.yml`	Replaced inline build script with call to shared build-and-test.sh script for Scala 2.13.6
`.github/workflows/ci-backend-2.12.yml`	Replaced inline build script with call to shared build-and-test.sh script for Scala 2.12.12
`.github/scripts/build-and-test.sh`	New reusable script containing build/test logic with uv-based dependency installation and virtual environment support

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · Nov 7, 2025

.github/scripts/build-and-test.sh

+
+# Get jep installation details
+echo "Configuring jep library paths..."
+jep_site_packages_path=$(uv pip show jep | grep "^Location:" | cut -d ':' -f 2 | xargs)


The script uses uv pip show jep which requires jep to be installed, but there's no guarantee that jep is in the requirements.txt or was successfully installed. If jep isn't found, jep_site_packages_path will be empty, leading to incorrect path constructions in lines 50-51 and potentially breaking the build. Consider adding error handling to verify jep was installed successfully before attempting to get its location.

Suggested change

jep_site_packages_path=$(uv pip show jep | grep "^Location:" | cut -d ':' -f 2 | xargs)

jep_site_packages_path=$(uv pip show jep | grep "^Location:" | cut -d ':' -f 2 | xargs)

if [ -z "$jep_site_packages_path" ]; then

echo "Error: 'jep' is not installed or not found in the Python environment. Please ensure 'jep' is listed in requirements.txt and installed successfully."

exit 1

fi

Copilot · Nov 7, 2025

.github/scripts/build-and-test.sh

+
+# Set up environment variables for jep
+export LD_LIBRARY_PATH="${jep_path}:${jep_site_packages_path}:${jep_lib_path}:${LD_LIBRARY_PATH:-}"
+export LD_PRELOAD="$(dirname "$(dirname "$(realpath "$(which python3)")")")/lib/libpython3.so"


This path construction assumes the virtual environment's python3 will be found by which, but there's no verification that the file exists at the constructed path. If the libpython3.so file doesn't exist at this location (e.g., different Python installation structure), LD_PRELOAD will point to a non-existent file, potentially causing runtime failures. The original code used realpath ${jep_site_packages_path}/../../ which was more reliable. Consider adding a check to verify the file exists or reverting to a more robust path resolution method.

Suggested change

export LD_PRELOAD="$(dirname "$(dirname "$(realpath "$(which python3)")")")/lib/libpython3.so"

LIBPYTHON_SO="${jep_lib_path}/libpython3.so"

if [ -f "${LIBPYTHON_SO}" ]; then

export LD_PRELOAD="${LIBPYTHON_SO}"

else

echo "Warning: libpython3.so not found at ${LIBPYTHON_SO}. LD_PRELOAD will not be set."

fi

Copilot · Nov 7, 2025

.github/scripts/build-and-test.sh

+# Create a Python virtual environment if it doesn't exist
+if [ ! -d ".venv" ]; then
+    echo "Creating Python virtual environment in .venv..."
+    python -m venv .venv


The script uses python to create the virtual environment but later uses python3 in line 55. This inconsistency could cause issues if python and python3 point to different Python versions or if python doesn't exist on the system. Use python3 consistently throughout the script.

Suggested change

python -m venv .venv

python3 -m venv .venv

jonathanindig

Looks good to me

wisechengyi marked this pull request as ready for review November 4, 2025 20:25

wisechengyi force-pushed the testextract branch from addac6b to 32c17b0 Compare November 4, 2025 22:33

wisechengyi changed the base branch from master to fixrace3 November 4, 2025 22:33

wisechengyi changed the base branch from fixrace3 to shade-guava-fixrace3 November 4, 2025 22:35

This was referenced Nov 5, 2025

Use JDK 17 runtime #1471

Merged

Shade Google dependencies in assembly #1477

Merged

Bump Spark to 3.3 #1478

Merged

wisechengyi changed the base branch from shade-guava-fixrace3 to graphite-base/1476 November 5, 2025 21:42

This was referenced Nov 5, 2025

Fix spark download deadlock #1475

Merged

Spark 3.5 Support #1479

Merged

wisechengyi force-pushed the graphite-base/1476 branch from 9a8a663 to 51beff2 Compare November 6, 2025 00:06

wisechengyi force-pushed the testextract branch from 32c17b0 to 5af9d37 Compare November 6, 2025 00:06

wisechengyi changed the base branch from graphite-base/1476 to shade-guava-fixrace3 November 6, 2025 00:06

jonathanindig requested a review from Copilot November 7, 2025 16:58

Copilot AI reviewed Nov 7, 2025

View reviewed changes

jonathanindig approved these changes Nov 7, 2025

View reviewed changes

wisechengyi force-pushed the shade-guava-fixrace3 branch 2 times, most recently from 1642615 to 8e462d6 Compare November 7, 2025 18:40

wisechengyi force-pushed the testextract branch from 5af9d37 to 647b363 Compare November 7, 2025 18:40

wisechengyi changed the base branch from shade-guava-fixrace3 to graphite-base/1476 November 7, 2025 18:44

Extract test script

6bb6f33

wisechengyi force-pushed the graphite-base/1476 branch from 8e462d6 to 593c2ba Compare November 7, 2025 18:45

wisechengyi force-pushed the testextract branch from 647b363 to f4dbb79 Compare November 7, 2025 18:45

graphite-app bot changed the base branch from graphite-base/1476 to master November 7, 2025 18:45

libpython3.so

695e42c

wisechengyi force-pushed the testextract branch from f4dbb79 to 695e42c Compare November 7, 2025 18:45

wisechengyi merged commit 13ef697 into master Nov 7, 2025
4 checks passed

wisechengyi deleted the testextract branch November 7, 2025 19:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extract test script #1476

Extract test script #1476

Uh oh!

wisechengyi commented Nov 4, 2025 •

edited

Loading

Uh oh!

wisechengyi commented Nov 5, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

Copilot AI Nov 7, 2025

Uh oh!

jonathanindig left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-export LD_PRELOAD="$(dirname "$(dirname "$(realpath "$(which python3)")")")/lib/libpython3.so"
+LIBPYTHON_SO="${jep_lib_path}/libpython3.so"
+if [ -f "${LIBPYTHON_SO}" ]; then
+    export LD_PRELOAD="${LIBPYTHON_SO}"
+else
+    echo "Warning: libpython3.so not found at ${LIBPYTHON_SO}. LD_PRELOAD will not be set."
+fi

Search code, repositories, users, issues, pull requests...

Extract test script #1476

Extract test script #1476

Uh oh!

Conversation

wisechengyi commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Extracted build logic into reusable script

Simplified CI workflows

Uh oh!

wisechengyi commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

jonathanindig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wisechengyi commented Nov 4, 2025 •

edited

Loading

wisechengyi commented Nov 5, 2025 •

edited

Loading