Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Conversation

spencerschrock
Copy link
Member

What kind of change does this PR introduce?

bug fix from #4474

What is the current behavior?

Each call to ListFiles would walk the HEAD commit's tree, but go-git is not thread-safe.
So the old method would lead to race conditions when multiple checks called ListFiles at the same time. This slipped by my development but was reproducible when built with -race:

go build -race && ./scorecard --repo ossf-tests/scorecard-check-license-e2e --file-mode git
...
fatal error: concurrent map read and map write
...
==================
WARNING: DATA RACE
Write at 0x00c000222f30 by goroutine 67:
  runtime.mapassign()
      ~/sdk/go1.24.0/src/internal/runtime/maps/runtime_swiss.go:191 +0x0
< omitted > 
  github.com/go-git/go-git/v5.(*Repository).Head()
      ~/go/pkg/mod/github.com/go-git/go-git/v5@v5.13.2/repository.go:1497 +0x1ee
  github.com/ossf/scorecard/v5/internal/gitfile.(*Handler).ListFiles()
      ~/go/src/github.com/spencerschrock/scorecard.git/internal/gitfile/gitfile.go:103 +0xc6
  github.com/ossf/scorecard/v5/clients/githubrepo.(*Client).ListFiles()
      ~/go/src/github.com/spencerschrock/scorecard.git/clients/githubrepo/client.go:187 +0x87
  github.com/ossf/scorecard/v5/checks/fileparser.OnAllFilesDo()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/fileparser/listing.go:160 +0x6b
  github.com/ossf/scorecard/v5/checks/raw.DependencyUpdateTool()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/raw/dependency_update_tool.go:35 +0x110
  github.com/ossf/scorecard/v5/checks.DependencyUpdateTool()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/dependency_update_tool.go:42 +0x7a
  github.com/ossf/scorecard/v5/checker.(*Runner).Run()
      ~/go/src/github.com/spencerschrock/scorecard.git/checker/check_runner.go:118 +0xe48
  github.com/ossf/scorecard/v5/pkg/scorecard.runEnabledChecks.func1()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:67 +0x257

Previous read at 0x00c000222f30 by goroutine 58:
  runtime.mapaccess1()
      ~/sdk/go1.24.0/src/internal/runtime/maps/runtime_swiss.go:43 +0x0
< omitted > 
  github.com/go-git/go-git/v5.(*Repository).Head()
      ~/go/pkg/mod/github.com/go-git/go-git/v5@v5.13.2/repository.go:1497 +0x1ee
  github.com/ossf/scorecard/v5/internal/gitfile.(*Handler).ListFiles()
      ~/go/src/github.com/spencerschrock/scorecard.git/internal/gitfile/gitfile.go:103 +0xc6
  github.com/ossf/scorecard/v5/clients/githubrepo.(*Client).ListFiles()
      ~/go/src/github.com/spencerschrock/scorecard.git/clients/githubrepo/client.go:187 +0x87
  github.com/ossf/scorecard/v5/checks/raw/github.Packaging()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/raw/github/packaging.go:34 +0x69
  github.com/ossf/scorecard/v5/checks.Packaging()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/packaging.go:64 +0x2aa
  github.com/ossf/scorecard/v5/checker.(*Runner).Run()
      ~/go/src/github.com/spencerschrock/scorecard.git/checker/check_runner.go:118 +0xe48
  github.com/ossf/scorecard/v5/pkg/scorecard.runEnabledChecks.func1()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:67 +0x257

Goroutine 67 (running) created at:
  github.com/ossf/scorecard/v5/pkg/scorecard.runEnabledChecks()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:59 +0xda
  github.com/ossf/scorecard/v5/pkg/scorecard.runScorecard.gowrap2()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:179 +0x8f

Goroutine 58 (running) created at:
  github.com/ossf/scorecard/v5/pkg/scorecard.runEnabledChecks()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:59 +0xda
  github.com/ossf/scorecard/v5/pkg/scorecard.runScorecard.gowrap2()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:179 +0x8f
==================

What is the new behavior (if this is a feature change)?**

The data race no longer occurs because the files are enumerated from the git tree inside a sync.Once, so only one thread.
These files are put in a slice where they can concurrently be read by multiple goroutines later.

go build -race && ./scorecard --repo ossf-tests/scorecard-check-license-e2e --file-mode git
<normal output>
  • Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

NONE

Special notes for your reviewer

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

Fixed a data race when analyzing repositories with `--file-mode git`

go-git is not thread-safe, so the old method would lead to race
conditions when multiple checks called ListFiles at the same time. This
was reproducible when built with `-race`, and now no longer occurs
because it's done inside a sync.Once.

Signed-off-by: Spencer Schrock <sschrock@google.com>
@spencerschrock spencerschrock requested a review from a team as a code owner February 13, 2025 07:03
@spencerschrock spencerschrock requested review from justaugustus and raghavkaul and removed request for a team February 13, 2025 07:03
Copy link

codecov bot commented Feb 13, 2025

Codecov Report

Attention: Patch coverage is 66.66667% with 7 lines in your changes missing coverage. Please review.

Project coverage is 68.19%. Comparing base (353ed60) to head (e4146dc).
Report is 114 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4522      +/-   ##
==========================================
+ Coverage   66.80%   68.19%   +1.39%     
==========================================
  Files         230      247      +17     
  Lines       16602    18635    +2033     
==========================================
+ Hits        11091    12709    +1618     
- Misses       4808     5083     +275     
- Partials      703      843     +140     

@spencerschrock spencerschrock changed the title 🐛 cache files during gitfile handler setup to avoid data race 🐛 save files during gitfile handler setup to avoid data race Feb 13, 2025
@spencerschrock spencerschrock changed the title 🐛 save files during gitfile handler setup to avoid data race 🐛 save file names during gitfile handler setup to avoid data race Feb 13, 2025
Copy link
Member

@justaugustus justaugustus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @spencerschrock!

@justaugustus justaugustus merged commit cd152cb into ossf:main Feb 16, 2025
43 checks passed
@spencerschrock spencerschrock deleted the git-list branch February 17, 2025 16:32
balteravishay pushed a commit to PrinceAsiedu/scorecard that referenced this pull request Feb 17, 2025
go-git is not thread-safe, so the old method would lead to race
conditions when multiple checks called ListFiles at the same time. This
was reproducible when built with `-race`, and now no longer occurs
because it's done inside a sync.Once.

Signed-off-by: Spencer Schrock <sschrock@google.com>
Signed-off-by: balteravishay <avishay.balter@gmail.com>
balteravishay pushed a commit to PrinceAsiedu/scorecard that referenced this pull request Feb 26, 2025
go-git is not thread-safe, so the old method would lead to race
conditions when multiple checks called ListFiles at the same time. This
was reproducible when built with `-race`, and now no longer occurs
because it's done inside a sync.Once.

Signed-off-by: Spencer Schrock <sschrock@google.com>
Signed-off-by: balteravishay <avishay.balter@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.