Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

[JENKINS-64383] combined refrepo became our bottleneck, support a fanout location too - #644

#644
Open
jimklimov wants to merge 183 commits into
jenkinsci:masterjenkinsci/git-client-plugin:masterfrom
jimklimov:refrepo-argsjimklimov/git-client-plugin:refrepo-argsCopy head branch name to clipboard
Open

[JENKINS-64383] combined refrepo became our bottleneck, support a fanout location too#644
jimklimov wants to merge 183 commits into
jenkinsci:masterjenkinsci/git-client-plugin:masterfrom
jimklimov:refrepo-argsjimklimov/git-client-plugin:refrepo-argsCopy head branch name to clipboard

Conversation

@jimklimov

@jimklimov jimklimov commented Dec 7, 2020

Copy link
Copy Markdown
Contributor

JENKINS-64383 - combined refrepo became our bottleneck

As detailed in the JIRA issue, our heavy use of a single combined reference repository made it more a bottleneck and cause of job timeouts than a speedup and reliability improvement which it once was. This PR explores a way to keep the single point of configuration of the reference repository directory, suffixed with some "magic variable" to substitute a path to subdirectory with a smaller-scope reference repository for a particular source Git URL. On file systems with symlinks it is possible to maintain several such names that would point to the same directory, for closely-related repositories or different URLs of the same repository.

This PoC introduces trivial support for reference repository paths ending with /${GIT_URL} to replace by url => funny dir subtree in filesystem. Its limitation at the moment is that the URL is pasted in verbatim - this works for Linux and Unix like systems that only forbid a 0x00 and a slash from being characters in a filename, and slash suits us as a directory subtree separator. This code likely won't run on Windows as is (colon in https: and likely other chars - Microsoft has an extensive list of invalid chars).

The next ideas, commented but not yet PoCed, are to either escape such characters (non-ASCII and offensive to at least one popular filesystem), or convert URLs into base64 strings or sha/md5/... hashes. Using submodules and finding a way to map several URLs to a certain submodule might be a good idea if they keep indexes separately. This all can be built on top of this PoCed code by introducing further suffixes and handling for them.

It was tested on a MultiBranch pipeline job, where an original definition of the reference repository was suffixed with the new magic string, yielding /home/abuild/jenkins-gitcache/${GIT_URL} (verbatim in "Advanced clone behaviours"). During the checkout into a wiped workspace, with this plugin variant installed:

Cloning the remote Git repository
Cloning repository https://github.com/zeromq/czmq.git
 > git init /dev/shm/jenkins-swarm-client/workspace/CZMQ-upstream_master # timeout=10
[WARNING] Parameterized reference path replaced with: /home/abuild/jenkins-gitcache/https://github.com/zeromq/czmq.git
Using reference repository: /home/abuild/jenkins-gitcache/https://github.com/zeromq/czmq.git
Fetching upstream changes from https://github.com/zeromq/czmq.git
 > git --version # timeout=10
 > git --version # 'git version 2.1.4'
 > git fetch --tags --progress https://github.com/zeromq/czmq.git +refs/heads/*:refs/remotes/origin/* # timeout=40

Avoid second fetch
Checking out Revision fbe313cd2010bace7833fe52d419f82282343bd9 (master)

 > git config remote.origin.url https://github.com/zeromq/czmq.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fbe313cd2010bace7833fe52d419f82282343bd9 # timeout=10

Commit message: "Merge pull request #2139 from bluca/ci_failures"
 > git rev-list --no-walk fbe313cd2010bace7833fe52d419f82282343bd9 # timeout=10

This completed quickly, much faster than the usual checkout with huge refrepo in original /home/abuild/jenkins-gitcache/, and did automatically find the "funny" /home/abuild/jenkins-gitcache/https://github.com/zeromq/czmq.git directory prepared with the single repo's reference cache:

# ls -la /home/abuild/jenkins-gitcache/https://github.com/zeromq/czmq.git
total 38
drwxr-xr-x 7 4294967294 4294967294   12 Dec  7 19:31 .
drwxr-xr-x 3 4294967294 4294967294    3 Dec  7 19:29 ..
-rw-r--r-- 1 4294967294 4294967294 2353 Dec  7 19:31 FETCH_HEAD
-rw-r--r-- 1 4294967294 4294967294   23 Dec  7 19:30 HEAD
drwxr-xr-x 2 4294967294 4294967294    2 Dec  7 19:30 branches
-rwxr--r-- 1 4294967294 4294967294  204 Dec  7 19:30 config
-rw-r--r-- 1 4294967294 4294967294   73 Dec  7 19:30 description
drwxr-xr-x 2 4294967294 4294967294   11 Dec  7 19:30 hooks
drwxr-xr-x 2 4294967294 4294967294    3 Dec  7 19:30 info
drwxr-xr-x 4 4294967294 4294967294    4 Dec  7 19:30 objects
drwxr-xr-x 5 4294967294 4294967294    5 Dec  7 19:31 refs
lrwxrwxrwx 1 4294967294 4294967294   43 Dec  7 19:30 register-git-cache.sh -> /mnt/jenkins-gitcache/register-git-cache.sh

DOCS NOTE: With 2.36.x and newer Git versions, if your reference repository maintenance script runs as a different user account than the Jenkins server (or Jenkins agent), safety checks about safe.directory (see https://github.blog/2022-04-18-highlights-from-git-2-36/) can be disabled by configuring each such user account:

:; git config --global --add safe.directory '*'

UPDATE: My repository at https://github.com/jimklimov/git-refrepo-scripts provides the shell scripts and Jenkinsfile jobs I use to maintain the servers using this modification of the Git Client plugin. One of the jobs there allows to automatically discover and register Git repositories used by known builds on the server it runs on (might run daily or so), and another can run more regularly to update the known refrepos.

@jimklimov
jimklimov force-pushed the refrepo-args branch 2 times, most recently from 191ca38 to 694ee90 Compare December 8, 2020 12:34
@jimklimov
jimklimov marked this pull request as draft December 10, 2020 01:43
@MarkEWaite MarkEWaite added the enhancement Improvement or new feature label Dec 13, 2020
…RL} to replace by url => funny dir subtree in filesystem
…atibleGitAPIImpl.java so its logic (expected to grow in complexity) can be shared by both JGitAPIImpl.java and CliGitAPIImpl.java
…d ref-repos in submodule checkouts (only CliGitAPIImpl.java has it)
…(): do not bother normalizing the URL if the string is not with supported suffix
…intsToLocal*Mirror() with custom paths and bare vs workspace repos
…refactor getObjectPath(referencePath) to check on git dirs elsewhere later
…erenceRepository() and isParameterizedReferenceRepository() taking a File reference (not only a String) object
… keep original reference intact, and as indicator to recreate referencePath object once for many cases
…256_FALLBACK suffixes for using unsuffixed directory if expanded path points nowhere useful
jimklimov and others added 10 commits July 15, 2026 21:58
…s/ subdir

When a workspace's .git is a file (the submodule redirect pattern, e.g.
"gitdir: ../../.git/modules/sub"), the code resolved the gitdir path but
then used that directory directly as the objects location.  Because the
gitdir itself is a directory, every downstream isDirectory() check passed,
and the alternates file was written with a path like
  /workspace/.git/modules/sub
instead of the correct
  /workspace/.git/modules/sub/objects
Git silently ignores alternates entries that are not an objects/ directory,
so the reference repository was never used and a full network fetch happened
instead.

Fix: append "/objects" to the resolved gitdir path, mirroring what the
non-redirect branch already does with new File(fGit, "objects").

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In findParameterizedReferenceRepository(), the else-branch that handles
/${GIT_SUBMODULES_FALLBACK} called:

  reference.replaceAll("\$\{GIT_SUBMODULES\}$", needleBasename)

The pattern targets the non-fallback token, which is not present in the
string, so replaceAll() returned the original string unchanged.
referenceExpanded therefore still ended with "/${GIT_SUBMODULES_FALLBACK}",
getObjectsFile() returned null on that literal path, and the code always
fell back to referenceBaseDir — the specific per-URL subdirectory was
never tried, defeating the whole point of the fallback mode.

Fix: use the correct pattern \$\{GIT_SUBMODULES_FALLBACK\}$ so the token
is actually replaced with the computed needleBasename.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
findParameterizedReferenceRepository() printed the raw `url` and
`urlNormalized` values unconditionally to stderr via System.err.println.
Remote URLs can contain embedded credentials (user:token@host), and stderr
output bypasses Jenkins' credential-masking facility, potentially leaking
secrets into build logs or console output.

The same information is already logged at INFO level through the Jenkins
task listener earlier in the method, so the debug prints carry no unique
value.  The stale comment claiming "unit-tests expect this markup on stderr"
is incorrect: all test assertions check handler.getMessages() (the Jenkins
listener log), not stderr.

Remove both System.err.println calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…or absolute paths

Two related correctness fixes in LegacyCompatibleGitAPIImpl:

1. Regex: every call to replaceAll(".git$", "") used an unescaped dot, which
   is a regex wildcard matching any character.  A URL or directory name ending
   in e.g. "xgit" or "1git" would have its last four characters incorrectly
   stripped, producing a wrong basename and causing all SHA256/BASENAME probes
   to miss even when a correctly-named reference directory exists.
   Fix: escape to "\.git$" (and "\.[Gg][Ii][Tt]$" for the case-insensitive
   variant).  Same fix applied to the trailing-slash pattern "/*$" -> "/+$"
   (the original was technically correct but /+ is the right idiom).

2. Absolute-path safety: in the basename-priority probe loop of
   getSubmodulesUrls(), the code called referenceGit.subGit(dirname) where
   dirname is an absolute path.  subGit() constructs new File(workspace, subdir),
   which on Windows drops the drive letter when subdir begins with a slash,
   producing a wrong path on a different drive than the Jenkins workspace.
   The second loop in the same method already correctly uses this.newGit(dirname).
   Fix: use this.newGit(dirname) in the first loop too.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both CliGitAPIImpl and JGitAPIImpl had near-identical implementations of
getRemoteUrls() and getRemotePushUrls() differing only in which config key
or URIish list accessor they used.  The duplication meant bug fixes or
improvements had to be applied twice, and the two methods had already
started to drift (a comment was present in one but not the other).

CliGitAPIImpl: extract getRemoteUrlMap(String configKeySuffix) and delegate
  both public methods to it, parameterised on ".url=" vs ".pushurl=".

JGitAPIImpl: extract collectRemoteUris(Function<RemoteConfig,List<URIish>>)
  and delegate both public methods via method references RemoteConfig::getURIs
  and RemoteConfig::getPushURIs.

No behaviour change.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- isParameterizedReferenceRepository(): collapse 6 independent if/return
  true blocks into a single return with || operators.

- getSubmodulesUrls(): remove large commented-out block (TBD placeholder);
  replace with a concise TODO comment.  Remove the duplicate `isBare = false`
  assignment inside the else-branch.  Remove `result = new LinkedHashSet<>()`
  before the two exact-match early-return sites; clearing the set there
  discarded previously cached findings for no benefit.

- findParameterizedReferenceRepository(): remove `referencePath = null; // GC`
  lines; the JVM does not need manual nulling hints and the pattern is noisy.

- CliGitAPIImpl / JGitAPIImpl: consolidate the 4-line incremental string
  construction for the "Parameterized reference path ... replaced with: ..."
  log message into a single string expression.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…eturn LinkedHashSet with best result on top

Semi-revert the AI change from commit 9cff320 with rationale:
> Remove `result = new LinkedHashSet<>()` before the two exact-match
> early-return sites; clearing the set there discarded previously
> cached findings for no benefit.

IIRC the benefit was that the caller needed not look at the other cached
candidate locations. Returning a set where the exact-match is listed
first, but other findings are not forgotten, may cover both bases.

Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
… dropped commented-away code for not-bare repo handling

Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
…; drop duplicate method

WorkspaceWithRepo.localMirror(String): remove 7 unconditional
System.err.println calls that were added for debugging.  They fired on
every test run, polluted CI stderr, and are not guarded by any logging
level. Instead, add a class-level Logger and convert the 7 unconditional
System.err.println calls in localMirror(String) to LOGGER.log(Level.FINE).
The messages are still available when debugging (set the logger to FINE),
but are suppressed in normal CI runs.

GitClientCloneTest: remove 4 System.err.println calls from the two
SHA256-parameterized reference repo tests (wsRefrepoBase/wsRefrepo state
dumps and clone output dumps). Likewise, add a class-level Logger and
convert the 4 System.err.println calls in the two SHA256-parameterized
reference repo tests to LOGGER.log(Level.FINE). Also remove the no-arg
assertAlternateFilePointsToLocalWorkspaceMirror() overload which was
introduced alongside assertAlternateFilePointsToLocalMirror() and is
identical in body; the no-arg form is never called, so it is dead code.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The commented-out early-return block had a note suggesting it might "overload
the flag's meaning" — it was disabled and the note itself acknowledged the idea
was not fully thought through. Remove it to keep the method readable.
@jimklimov

jimklimov commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

@MarkEWaite : as discussed on the Jenkins Contributor Summit(s), I was slowly getting my feet wet with AI. Now finally exploring how it (Claude) would fare with the review comments that you posted in 2022 and requested tests to be added.

It seems to have done a decent job, also finding some bugs in my PR source branch. There were some changes it had done that I am less sure about, so would rather run the updated plugin for some time in practice to see if those are okay or break some smart idea I had years ago and might have forgotten about since.

If all seems OK I will update this PR, maybe even rebase to lose years of "merged from master" changes if that would not be extremely much work. For now the relevant updated code boils at https://github.com/jimklimov/git-client-plugin/tree/refrepo-args-tests (you can probably see a full diff against master at https://github.com/jimklimov/git-client-plugin/pull/new/refrepo-args-tests page, and a diff of just those follow-up fixes since the PR source branch at https://github.com/jimklimov/git-client-plugin/compare/refrepo-args...jimklimov:git-client-plugin:refrepo-args-tests?expand=1).

@MarkEWaite

Copy link
Copy Markdown
Contributor

Now finally exploring how it (Claude) would fare with the review comments that you posted in 2022 and requested tests to be added.

I'd love to see tests added to the pull request. If Claude can do it, that's great.

If all seems OK I will update this PR, maybe even rebase to lose years of "merged from master" changes if that would not be extremely much work

I'd be happy to see the pull request updated. No need to rebase unless that helps you in some way. Once your testing is completed, my review is complete, and my testing is completed, I'll most likely squash merge the pull request so that it will be a single entry in the commit history of the master branch.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces support for parameterized reference repository paths (fan-out layouts) so a single configured ref-repo base can resolve to smaller per-URL (or per-hash) reference repositories, reducing contention on a combined “monolith” refrepo. It also extends the GitClient API to query all remote URLs and to create same-implementation clients for other worktrees.

Changes:

  • Add parameterized refrepo resolution (e.g., ${GIT_URL_BASENAME}, ${GIT_URL_SHA256}, ${GIT_SUBMODULES} + fallback variants) and helpers like getObjectsFile() / URL normalization.
  • Extend GitClient with getRemoteUrls(), getRemotePushUrls(), and newGit(String); implement in CLI/JGit/remoting wrappers.
  • Add / adjust tests and helpers to exercise parameterized refrepo behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
src/test/java/org/jenkinsci/plugins/gitclient/WorkspaceWithRepo.java Test helper updates for configurable mirror directory names and path normalization.
src/test/java/org/jenkinsci/plugins/gitclient/GitClientCloneTest.java New tests for parameterized refrepo modes and refactored alternates-file assertions.
src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java Delegate new GitClient API methods through the remoting proxy.
src/main/java/org/jenkinsci/plugins/gitclient/LegacyCompatibleGitAPIImpl.java Core implementation of parameterized refrepo resolution, URL normalization, and repo-scanning helpers.
src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java Implement new APIs and parameterized refrepo handling for the JGit path.
src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java Define new GitClient interface methods (getRemoteUrls, getRemotePushUrls, newGit).
src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java Implement new APIs and parameterized refrepo handling for the CLI path; improve error message with workdir.
src/main/java/hudson/plugins/git/GitAPI.java Route newGit(String) to CLI or JGit implementation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/test/java/org/jenkinsci/plugins/gitclient/GitClientCloneTest.java Outdated
Comment on lines +707 to 709
final String expectedContent = _testAltDir.replace("\\", "/") + "/objects";
final String actualContent = Files.readString(testGitDir.toPath().resolve(alternates), StandardCharsets.UTF_8);
assertThat("Alternates file content", actualContent, is(expectedContent));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, need to think about it :)

Comment thread src/main/java/org/jenkinsci/plugins/gitclient/LegacyCompatibleGitAPIImpl.java Outdated
Comment thread src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java Outdated
Comment on lines +270 to +287
@Test
void test_clone_reference_parameterized_basename() throws Exception {
testGitClient
.clone_()
.url(workspace.localMirror())
.repositoryName("origin")
.reference(workspace.localMirror() + "/${GIT_URL_BASENAME}")
.execute();
testGitClient.checkout().ref("origin/master").branch("master").execute();
check_remote_url(workspace, testGitClient, "origin");
// Verify JENKINS-46737 expected log message is written
String messages = StringUtils.join(handler.getMessages(), ";");
assertThat(
"Reference repo name-parsing logged in: " + messages,
handler.containsMessageSubstring("Parameterized reference path ")
&& handler.containsMessageSubstring(" replaced with: "),
is(true));
// TODO: Actually construct the local filesystem path to match
Comment thread src/main/java/org/jenkinsci/plugins/gitclient/LegacyCompatibleGitAPIImpl.java Outdated
@jimklimov

Copy link
Copy Markdown
Contributor Author

Thanks @MarkEWaite for the added review points. Most of them did hit what Claude already suggested (so far shelved in the other branch), but a few comments seem new to me. Hope to review them closer later.

@jimklimov

jimklimov commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

The plugin built with recent changes at least was not caught misbehaving in the past week or so, updating this PR source branch with new tests and fixes for older implementation bugs found by AI (Claude, sponsored by dayjob - Provys).

jimklimov and others added 5 commits July 22, 2026 22:55
…il methods

Addresses @MarkEWaite's Sep 2022 review request for test coverage of the new
methods introduced by the PR:

- GitClientTest.java: five new parameterized tests (run for git/jgit/jgitapache)
  covering getRemoteUrls() with one and two remotes, getRemotePushUrls() with
  and without an explicit pushurl configured, and newGit() verifying same-class
  client creation for a different directory.

- LegacyCompatibleGitAPIImplStaticTest.java: new test class for the static
  utility methods that need no git implementation parameter — getObjectsFile()
  for null/missing/normal-workspace/bare-repo/submodule-pointer inputs,
  isParameterizedReferenceRepository() for all six recognized magic suffixes and
  negative cases, and normalizeGitUrl() for trailing-slash stripping, .git
  suffix handling, scheme inference (file://, ssh://), and URL canonicalization.
testNewGit: The assertion comparing getClass() of the gitClient field (which
is a hudson.plugins.git.GitAPI wrapper) to the class returned by newGit()
(which is the unwrapped CliGitAPIImpl) always fails. Replace the class-equality
check with the functionally meaningful assertion: the returned client can
actually init a git repo in the specified directory.

isParameterizedReferenceRepository_fileOverload: On Windows, File.getPath()
uses backslash separators while the parameterized-suffix checks use forward
slashes, so new File("/.../${GIT_URL_BASENAME}").getPath() does not end with
"/${GIT_URL_BASENAME}". Restrict the File-overload test to platform-independent
cases (null → false, non-parameterized → false).
…am; drop redundant public on interface methods

GitClientCloneTest.assertAlternateFilePointsToLocalMirror(): the method
takes a _testGitDir parameter but line 708 read the alternates file using
the class field testGitDir instead, silently ignoring the parameter.
Any caller passing a different directory would assert the wrong file content.
Fix: use new File(_testGitDir).toPath() so the parameter is honoured.

GitClient.java: interface methods are implicitly public; the explicit
'public' modifier on getRemoteUrls() and getRemotePushUrls() was
inconsistent with every other method declaration in the interface.
Remove the redundant modifier.
…t<List<String>>

String[] uses identity-based equals/hashCode, so a LinkedHashSet<String[]>
never actually removes duplicate entries — every new String[] {} is a
distinct object. The comment at the merge point ("Hopefully the Set should
deduplicate entries if something matched twice") was therefore never true.

Change the element type from String[] to List<String> throughout
getSubmodulesUrls() and its callers. List.equals() / List.hashCode() are
content-based, so the LinkedHashSet now correctly deduplicates entries
with the same (dirname, url, urlNormalized, remoteName) tuple.

Mechanical changes:
- LinkedHashSet<String[]> → LinkedHashSet<List<String>> (10 sites)
- new String[] {a,b,c,d} → List.of(a,b,c,d) (4 sites)
- entry[N] → entry.get(N) (9 sites)
- for (String[] ...) → for (List<String> ...) (4 sites)
- Javadoc updated: [N] → get(N)
…lesUrls() and findParameterizedReferenceRepository() [JENKINS-64383]

Signed-off-by: Evgeny Klimov <klimov@provys.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jimklimov

Copy link
Copy Markdown
Contributor Author

Oh those dalmatians! :}

Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
@jimklimov

jimklimov commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Discussion on Gitter reminded me of a loose thread in this world, needed to pull it: by now we have (along with https://github.com/jimklimov/git-refrepo-scripts helper to maintain the refrepo location) a way to store a map of URLs to dirs, speeding up the checkouts and reducing log noise. Finally got around to implementing the missing bits.

Another fast-track improvement is to parse the git config files directly (via jGit) to look for remote definitions and not waste time and resources to spawn numerous git processes for that by default (only do so if such parsing attempt fails).

Thanks again to Claude for heavy-lifting, especially around tests. And code. And javadocs. Going back to sip coffee and test the result...

jimklimov and others added 2 commits July 23, 2026 12:02
…fast-track correct refrepo discovery [JENKINS-64383]

The https://github.com/jimklimov/git-refrepo-scripts helper can
maintain a file which maps Git URLs to refrepo directory names,
and now the plugin can use that file to speed up discovery of
the correct dir with less noise in build log when many projects
are cached and had to all/many be walked until now.

Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…forking git for remote lookups during refrepo walk [JENKINS-64383]

getSubmodulesUrls() previously called GitClient.getRemoteUrls() for every
candidate directory while walking a fanned-out refrepo, which forks a
`git config --local --list` subprocess (CliGit) or opens a full JGit
Repository (JGit) per directory. Add getRemoteUrlsFast(), which first
tries reading each dir's "config" file directly and parsing it in memory
with JGit's own Config/RemoteConfig classes (no subprocess, no Repository
object), falling back to the previous GitClient-based lookup when the
config is missing, unreadable, unparseable, or uses an include/includeIf
directive we do not attempt to resolve here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Jim Klimov <jimklimov+jenkinsci@gmail.com>
@jimklimov

jimklimov commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

The updated code seems to find the correct refrepo faster (at least much less noisily) when there are many fanned-out ref repo dirs to look at - if done pedantically :)

  • new:
[2026-07-25T00:41:01.149Z] Cloning the remote Git repository
[2026-07-25T00:41:01.160Z] Honoring refspec on initial clone
[2026-07-25T00:41:01.656Z] Cloning repository https://github.com/networkupstools/nut.git
[2026-07-25T00:41:02.469Z]  > git init /export/home/abuild/master-worker/workspace/nut_nut_master # timeout=10
[2026-07-25T00:41:02.737Z] [INFO] The git reference repository path '/var/lib/jenkins/gitcache/${GIT_SUBMODULES}' is parameterized, it may take a few git queries logged below to resolve it into a particular directory name
[2026-07-25T00:41:02.742Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-07-25T00:41:03.145Z] [WARNING] Parameterized reference path '/var/lib/jenkins/gitcache/${GIT_SUBMODULES}' replaced with: '/var/lib/jenkins/gitcache/nut.git'
[2026-07-25T00:41:03.151Z] Using reference repository: /var/lib/jenkins/gitcache/nut.git
[2026-07-25T00:41:03.155Z] Fetching upstream changes from https://github.com/networkupstools/nut.git
  • old (recent build, just before installing the updated plugin):
[2026-06-28T13:17:47.567Z] Cloning the remote Git repository
[2026-06-28T13:17:47.570Z] Honoring refspec on initial clone
[2026-06-28T13:17:47.573Z] Cloning repository https://github.com/networkupstools/nut.git
[2026-06-28T13:17:47.575Z]  > git init /export/home/abuild/master-worker/workspace/nut_nut_master@3 # timeout=10
[2026-06-28T13:17:47.648Z] [INFO] The git reference repository path '/var/lib/jenkins/gitcache/${GIT_SUBMODULES}' is parameterized, it may take a few git queries logged below to resolve it into a particular directory name
[2026-06-28T13:17:47.651Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:47.663Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:47.672Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:47.735Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:47.747Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:47.755Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:47.765Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:47.774Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:47.835Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:47.844Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:47.855Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:47.871Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:47.935Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:47.946Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:47.964Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:47.972Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.041Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.051Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.063Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.077Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.145Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.159Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.175Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.236Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.248Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.261Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.278Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.343Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.359Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.367Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.441Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.548Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.557Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.564Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.639Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.662Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.679Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.747Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.758Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.768Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.838Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.852Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:48.879Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:48.963Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.043Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.062Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.081Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.156Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.181Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.250Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.265Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.274Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.343Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.353Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.366Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.374Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.437Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.439Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.460Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.472Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.535Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.550Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.558Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.573Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.639Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.659Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.671Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.684Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.858Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.935Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:49.947Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:49.967Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.039Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.051Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.060Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.073Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.170Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.241Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.272Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.367Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.436Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.451Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.465Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.484Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.572Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.648Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.661Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.675Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.736Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.779Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.842Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.861Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.871Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.936Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.951Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:50.966Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:50.976Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:51.043Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:51.069Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:51.140Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:51.152Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:51.167Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:51.182Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:51.274Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:51.340Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:51.353Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:51.361Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:51.373Z]  > git config --local --list # timeout=10
[2026-06-28T13:17:51.436Z]  > git rev-parse --is-bare-repository # timeout=10
[2026-06-28T13:17:51.452Z] [WARNING] Parameterized reference path '/var/lib/jenkins/gitcache/${GIT_SUBMODULES}' replaced with: '/var/lib/jenkins/gitcache/nut.git'
[2026-06-28T13:17:51.453Z] Using reference repository: /var/lib/jenkins/gitcache/nut.git
[2026-06-28T13:17:51.453Z] Fetching upstream changes from https://github.com/networkupstools/nut.git

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Improvement or new feature ShortTerm Short term improvements tests Automated test addition or improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.