[JENKINS-64383] combined refrepo became our bottleneck, support a fanout location too - #644
#644[JENKINS-64383] combined refrepo became our bottleneck, support a fanout location too#644jimklimov wants to merge 183 commits into
Conversation
191ca38 to
694ee90
Compare
…RL} to replace by url => funny dir subtree in filesystem
…atibleGitAPIImpl.java so its logic (expected to grow in complexity) can be shared by both JGitAPIImpl.java and CliGitAPIImpl.java
…d ref-repos in submodule checkouts (only CliGitAPIImpl.java has it)
… findParameterizedReferenceRepository()
…SHA256} reference repos
… name for the mirrored repo
…6() with real sanity checks
…(): do not bother normalizing the URL if the string is not with supported suffix
…intsToLocal*Mirror() with custom paths and bare vs workspace repos
…6() to find expected paths
…refactor getObjectPath(referencePath) to check on git dirs elsewhere later
…erenceRepository() and isParameterizedReferenceRepository() taking a File reference (not only a String) object
…ceRepository() processing to stderr
… keep original reference intact, and as indicator to recreate referencePath object once for many cases
…256_FALLBACK suffixes for using unsuffixed directory if expanded path points nowhere useful
…ith fallback modes of operation
…d GIT_SUBMODULES_FALLBACK support
…s/ subdir When a workspace's .git is a file (the submodule redirect pattern, e.g. "gitdir: ../../.git/modules/sub"), the code resolved the gitdir path but then used that directory directly as the objects location. Because the gitdir itself is a directory, every downstream isDirectory() check passed, and the alternates file was written with a path like /workspace/.git/modules/sub instead of the correct /workspace/.git/modules/sub/objects Git silently ignores alternates entries that are not an objects/ directory, so the reference repository was never used and a full network fetch happened instead. Fix: append "/objects" to the resolved gitdir path, mirroring what the non-redirect branch already does with new File(fGit, "objects"). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In findParameterizedReferenceRepository(), the else-branch that handles
/${GIT_SUBMODULES_FALLBACK} called:
reference.replaceAll("\$\{GIT_SUBMODULES\}$", needleBasename)
The pattern targets the non-fallback token, which is not present in the
string, so replaceAll() returned the original string unchanged.
referenceExpanded therefore still ended with "/${GIT_SUBMODULES_FALLBACK}",
getObjectsFile() returned null on that literal path, and the code always
fell back to referenceBaseDir — the specific per-URL subdirectory was
never tried, defeating the whole point of the fallback mode.
Fix: use the correct pattern \$\{GIT_SUBMODULES_FALLBACK\}$ so the token
is actually replaced with the computed needleBasename.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
findParameterizedReferenceRepository() printed the raw `url` and `urlNormalized` values unconditionally to stderr via System.err.println. Remote URLs can contain embedded credentials (user:token@host), and stderr output bypasses Jenkins' credential-masking facility, potentially leaking secrets into build logs or console output. The same information is already logged at INFO level through the Jenkins task listener earlier in the method, so the debug prints carry no unique value. The stale comment claiming "unit-tests expect this markup on stderr" is incorrect: all test assertions check handler.getMessages() (the Jenkins listener log), not stderr. Remove both System.err.println calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…or absolute paths
Two related correctness fixes in LegacyCompatibleGitAPIImpl:
1. Regex: every call to replaceAll(".git$", "") used an unescaped dot, which
is a regex wildcard matching any character. A URL or directory name ending
in e.g. "xgit" or "1git" would have its last four characters incorrectly
stripped, producing a wrong basename and causing all SHA256/BASENAME probes
to miss even when a correctly-named reference directory exists.
Fix: escape to "\.git$" (and "\.[Gg][Ii][Tt]$" for the case-insensitive
variant). Same fix applied to the trailing-slash pattern "/*$" -> "/+$"
(the original was technically correct but /+ is the right idiom).
2. Absolute-path safety: in the basename-priority probe loop of
getSubmodulesUrls(), the code called referenceGit.subGit(dirname) where
dirname is an absolute path. subGit() constructs new File(workspace, subdir),
which on Windows drops the drive letter when subdir begins with a slash,
producing a wrong path on a different drive than the Jenkins workspace.
The second loop in the same method already correctly uses this.newGit(dirname).
Fix: use this.newGit(dirname) in the first loop too.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both CliGitAPIImpl and JGitAPIImpl had near-identical implementations of getRemoteUrls() and getRemotePushUrls() differing only in which config key or URIish list accessor they used. The duplication meant bug fixes or improvements had to be applied twice, and the two methods had already started to drift (a comment was present in one but not the other). CliGitAPIImpl: extract getRemoteUrlMap(String configKeySuffix) and delegate both public methods to it, parameterised on ".url=" vs ".pushurl=". JGitAPIImpl: extract collectRemoteUris(Function<RemoteConfig,List<URIish>>) and delegate both public methods via method references RemoteConfig::getURIs and RemoteConfig::getPushURIs. No behaviour change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- isParameterizedReferenceRepository(): collapse 6 independent if/return true blocks into a single return with || operators. - getSubmodulesUrls(): remove large commented-out block (TBD placeholder); replace with a concise TODO comment. Remove the duplicate `isBare = false` assignment inside the else-branch. Remove `result = new LinkedHashSet<>()` before the two exact-match early-return sites; clearing the set there discarded previously cached findings for no benefit. - findParameterizedReferenceRepository(): remove `referencePath = null; // GC` lines; the JVM does not need manual nulling hints and the pattern is noisy. - CliGitAPIImpl / JGitAPIImpl: consolidate the 4-line incremental string construction for the "Parameterized reference path ... replaced with: ..." log message into a single string expression. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…eturn LinkedHashSet with best result on top Semi-revert the AI change from commit 9cff320 with rationale: > Remove `result = new LinkedHashSet<>()` before the two exact-match > early-return sites; clearing the set there discarded previously > cached findings for no benefit. IIRC the benefit was that the caller needed not look at the other cached candidate locations. Returning a set where the exact-match is listed first, but other findings are not forgotten, may cover both bases. Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
… dropped commented-away code for not-bare repo handling Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
…; drop duplicate method WorkspaceWithRepo.localMirror(String): remove 7 unconditional System.err.println calls that were added for debugging. They fired on every test run, polluted CI stderr, and are not guarded by any logging level. Instead, add a class-level Logger and convert the 7 unconditional System.err.println calls in localMirror(String) to LOGGER.log(Level.FINE). The messages are still available when debugging (set the logger to FINE), but are suppressed in normal CI runs. GitClientCloneTest: remove 4 System.err.println calls from the two SHA256-parameterized reference repo tests (wsRefrepoBase/wsRefrepo state dumps and clone output dumps). Likewise, add a class-level Logger and convert the 4 System.err.println calls in the two SHA256-parameterized reference repo tests to LOGGER.log(Level.FINE). Also remove the no-arg assertAlternateFilePointsToLocalWorkspaceMirror() overload which was introduced alongside assertAlternateFilePointsToLocalMirror() and is identical in body; the no-arg form is never called, so it is dead code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The commented-out early-return block had a note suggesting it might "overload the flag's meaning" — it was disabled and the note itself acknowledged the idea was not fully thought through. Remove it to keep the method readable.
|
@MarkEWaite : as discussed on the Jenkins Contributor Summit(s), I was slowly getting my feet wet with AI. Now finally exploring how it (Claude) would fare with the review comments that you posted in 2022 and requested tests to be added. It seems to have done a decent job, also finding some bugs in my PR source branch. There were some changes it had done that I am less sure about, so would rather run the updated plugin for some time in practice to see if those are okay or break some smart idea I had years ago and might have forgotten about since. If all seems OK I will update this PR, maybe even rebase to lose years of "merged from master" changes if that would not be extremely much work. For now the relevant updated code boils at https://github.com/jimklimov/git-client-plugin/tree/refrepo-args-tests (you can probably see a full diff against master at https://github.com/jimklimov/git-client-plugin/pull/new/refrepo-args-tests page, and a diff of just those follow-up fixes since the PR source branch at https://github.com/jimklimov/git-client-plugin/compare/refrepo-args...jimklimov:git-client-plugin:refrepo-args-tests?expand=1). |
I'd love to see tests added to the pull request. If Claude can do it, that's great.
I'd be happy to see the pull request updated. No need to rebase unless that helps you in some way. Once your testing is completed, my review is complete, and my testing is completed, I'll most likely squash merge the pull request so that it will be a single entry in the commit history of the master branch. |
There was a problem hiding this comment.
Pull request overview
This PR introduces support for parameterized reference repository paths (fan-out layouts) so a single configured ref-repo base can resolve to smaller per-URL (or per-hash) reference repositories, reducing contention on a combined “monolith” refrepo. It also extends the GitClient API to query all remote URLs and to create same-implementation clients for other worktrees.
Changes:
- Add parameterized refrepo resolution (e.g.,
${GIT_URL_BASENAME},${GIT_URL_SHA256},${GIT_SUBMODULES}+ fallback variants) and helpers likegetObjectsFile()/ URL normalization. - Extend
GitClientwithgetRemoteUrls(),getRemotePushUrls(), andnewGit(String); implement in CLI/JGit/remoting wrappers. - Add / adjust tests and helpers to exercise parameterized refrepo behavior.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/java/org/jenkinsci/plugins/gitclient/WorkspaceWithRepo.java | Test helper updates for configurable mirror directory names and path normalization. |
| src/test/java/org/jenkinsci/plugins/gitclient/GitClientCloneTest.java | New tests for parameterized refrepo modes and refactored alternates-file assertions. |
| src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java | Delegate new GitClient API methods through the remoting proxy. |
| src/main/java/org/jenkinsci/plugins/gitclient/LegacyCompatibleGitAPIImpl.java | Core implementation of parameterized refrepo resolution, URL normalization, and repo-scanning helpers. |
| src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java | Implement new APIs and parameterized refrepo handling for the JGit path. |
| src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java | Define new GitClient interface methods (getRemoteUrls, getRemotePushUrls, newGit). |
| src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java | Implement new APIs and parameterized refrepo handling for the CLI path; improve error message with workdir. |
| src/main/java/hudson/plugins/git/GitAPI.java | Route newGit(String) to CLI or JGit implementation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| final String expectedContent = _testAltDir.replace("\\", "/") + "/objects"; | ||
| final String actualContent = Files.readString(testGitDir.toPath().resolve(alternates), StandardCharsets.UTF_8); | ||
| assertThat("Alternates file content", actualContent, is(expectedContent)); |
There was a problem hiding this comment.
Good point, need to think about it :)
| @Test | ||
| void test_clone_reference_parameterized_basename() throws Exception { | ||
| testGitClient | ||
| .clone_() | ||
| .url(workspace.localMirror()) | ||
| .repositoryName("origin") | ||
| .reference(workspace.localMirror() + "/${GIT_URL_BASENAME}") | ||
| .execute(); | ||
| testGitClient.checkout().ref("origin/master").branch("master").execute(); | ||
| check_remote_url(workspace, testGitClient, "origin"); | ||
| // Verify JENKINS-46737 expected log message is written | ||
| String messages = StringUtils.join(handler.getMessages(), ";"); | ||
| assertThat( | ||
| "Reference repo name-parsing logged in: " + messages, | ||
| handler.containsMessageSubstring("Parameterized reference path ") | ||
| && handler.containsMessageSubstring(" replaced with: "), | ||
| is(true)); | ||
| // TODO: Actually construct the local filesystem path to match |
|
Thanks @MarkEWaite for the added review points. Most of them did hit what Claude already suggested (so far shelved in the other branch), but a few comments seem new to me. Hope to review them closer later. |
|
The plugin built with recent changes at least was not caught misbehaving in the past week or so, updating this PR source branch with new tests and fixes for older implementation bugs found by AI (Claude, sponsored by dayjob - Provys). |
…il methods Addresses @MarkEWaite's Sep 2022 review request for test coverage of the new methods introduced by the PR: - GitClientTest.java: five new parameterized tests (run for git/jgit/jgitapache) covering getRemoteUrls() with one and two remotes, getRemotePushUrls() with and without an explicit pushurl configured, and newGit() verifying same-class client creation for a different directory. - LegacyCompatibleGitAPIImplStaticTest.java: new test class for the static utility methods that need no git implementation parameter — getObjectsFile() for null/missing/normal-workspace/bare-repo/submodule-pointer inputs, isParameterizedReferenceRepository() for all six recognized magic suffixes and negative cases, and normalizeGitUrl() for trailing-slash stripping, .git suffix handling, scheme inference (file://, ssh://), and URL canonicalization.
testNewGit: The assertion comparing getClass() of the gitClient field (which
is a hudson.plugins.git.GitAPI wrapper) to the class returned by newGit()
(which is the unwrapped CliGitAPIImpl) always fails. Replace the class-equality
check with the functionally meaningful assertion: the returned client can
actually init a git repo in the specified directory.
isParameterizedReferenceRepository_fileOverload: On Windows, File.getPath()
uses backslash separators while the parameterized-suffix checks use forward
slashes, so new File("/.../${GIT_URL_BASENAME}").getPath() does not end with
"/${GIT_URL_BASENAME}". Restrict the File-overload test to platform-independent
cases (null → false, non-parameterized → false).
…am; drop redundant public on interface methods GitClientCloneTest.assertAlternateFilePointsToLocalMirror(): the method takes a _testGitDir parameter but line 708 read the alternates file using the class field testGitDir instead, silently ignoring the parameter. Any caller passing a different directory would assert the wrong file content. Fix: use new File(_testGitDir).toPath() so the parameter is honoured. GitClient.java: interface methods are implicitly public; the explicit 'public' modifier on getRemoteUrls() and getRemotePushUrls() was inconsistent with every other method declaration in the interface. Remove the redundant modifier.
…t<List<String>>
String[] uses identity-based equals/hashCode, so a LinkedHashSet<String[]>
never actually removes duplicate entries — every new String[] {} is a
distinct object. The comment at the merge point ("Hopefully the Set should
deduplicate entries if something matched twice") was therefore never true.
Change the element type from String[] to List<String> throughout
getSubmodulesUrls() and its callers. List.equals() / List.hashCode() are
content-based, so the LinkedHashSet now correctly deduplicates entries
with the same (dirname, url, urlNormalized, remoteName) tuple.
Mechanical changes:
- LinkedHashSet<String[]> → LinkedHashSet<List<String>> (10 sites)
- new String[] {a,b,c,d} → List.of(a,b,c,d) (4 sites)
- entry[N] → entry.get(N) (9 sites)
- for (String[] ...) → for (List<String> ...) (4 sites)
- Javadoc updated: [N] → get(N)
…lesUrls() and findParameterizedReferenceRepository() [JENKINS-64383] Signed-off-by: Evgeny Klimov <klimov@provys.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Oh those dalmatians! :} |
Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
|
Discussion on Gitter reminded me of a loose thread in this world, needed to pull it: by now we have (along with https://github.com/jimklimov/git-refrepo-scripts helper to maintain the refrepo location) a way to store a map of URLs to dirs, speeding up the checkouts and reducing log noise. Finally got around to implementing the missing bits. Another fast-track improvement is to parse the git config files directly (via jGit) to look for remote definitions and not waste time and resources to spawn numerous git processes for that by default (only do so if such parsing attempt fails). Thanks again to Claude for heavy-lifting, especially around tests. And code. And javadocs. Going back to sip coffee and test the result... |
…fast-track correct refrepo discovery [JENKINS-64383] The https://github.com/jimklimov/git-refrepo-scripts helper can maintain a file which maps Git URLs to refrepo directory names, and now the plugin can use that file to speed up discovery of the correct dir with less noise in build log when many projects are cached and had to all/many be walked until now. Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…forking git for remote lookups during refrepo walk [JENKINS-64383] getSubmodulesUrls() previously called GitClient.getRemoteUrls() for every candidate directory while walking a fanned-out refrepo, which forks a `git config --local --list` subprocess (CliGit) or opens a full JGit Repository (JGit) per directory. Add getRemoteUrlsFast(), which first tries reading each dir's "config" file directly and parsing it in memory with JGit's own Config/RemoteConfig classes (no subprocess, no Repository object), falling back to the previous GitClient-based lookup when the config is missing, unreadable, unparseable, or uses an include/includeIf directive we do not attempt to resolve here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
|
The updated code seems to find the correct refrepo faster (at least much less noisily) when there are many fanned-out ref repo dirs to look at - if done pedantically :)
|
JENKINS-64383 - combined refrepo became our bottleneck
As detailed in the JIRA issue, our heavy use of a single combined reference repository made it more a bottleneck and cause of job timeouts than a speedup and reliability improvement which it once was. This PR explores a way to keep the single point of configuration of the reference repository directory, suffixed with some "magic variable" to substitute a path to subdirectory with a smaller-scope reference repository for a particular source Git URL. On file systems with symlinks it is possible to maintain several such names that would point to the same directory, for closely-related repositories or different URLs of the same repository.
This PoC introduces trivial support for reference repository paths ending with
/${GIT_URL}to replace by url => funny dir subtree in filesystem. Its limitation at the moment is that the URL is pasted in verbatim - this works for Linux and Unix like systems that only forbid a 0x00 and a slash from being characters in a filename, and slash suits us as a directory subtree separator. This code likely won't run on Windows as is (colon inhttps:and likely other chars - Microsoft has an extensive list of invalid chars).The next ideas, commented but not yet PoCed, are to either escape such characters (non-ASCII and offensive to at least one popular filesystem), or convert URLs into base64 strings or sha/md5/... hashes. Using submodules and finding a way to map several URLs to a certain submodule might be a good idea if they keep indexes separately. This all can be built on top of this PoCed code by introducing further suffixes and handling for them.
It was tested on a MultiBranch pipeline job, where an original definition of the reference repository was suffixed with the new magic string, yielding
/home/abuild/jenkins-gitcache/${GIT_URL}(verbatim in "Advanced clone behaviours"). During the checkout into a wiped workspace, with this plugin variant installed:This completed quickly, much faster than the usual checkout with huge refrepo in original
/home/abuild/jenkins-gitcache/, and did automatically find the "funny"/home/abuild/jenkins-gitcache/https://github.com/zeromq/czmq.gitdirectory prepared with the single repo's reference cache:DOCS NOTE: With 2.36.x and newer Git versions, if your reference repository maintenance script runs as a different user account than the Jenkins server (or Jenkins agent), safety checks about
safe.directory(see https://github.blog/2022-04-18-highlights-from-git-2-36/) can be disabled by configuring each such user account:UPDATE: My repository at https://github.com/jimklimov/git-refrepo-scripts provides the shell scripts and Jenkinsfile jobs I use to maintain the servers using this modification of the Git Client plugin. One of the jobs there allows to automatically discover and register Git repositories used by known builds on the server it runs on (might run daily or so), and another can run more regularly to update the known refrepos.