Description
Description
While running a brand-new image, there are two .dist-info dirs for pip:
$ ls -l /opt/hostedtoolcache/Python/3.12.1/x64/lib/python3.12/site-packages
drwxrwxrwx+ 5 runner runneradmin 4096 Dec 17 22:19 pip
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.2.1.dist-info
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.3.2.dist-infoThe first (pip-23.2.1.dist-info) belongs to the pip version that was initially installed, the second (pip-23.3.2.dist-info) to the updated version. Apparently this is because pip is updated using --ignore-installed, which leaves behind the old pip-23.2.1.dist-info.
Even though the latest version of pip is installed and used, the presence of the two dirs can create issues. For example, tools like safety detect the old version and report it since it has vulnerabilities, causing CI failures:
Unless there is a valid reason to keep the old .dist-info around, I suggest removing the --ignore-installed flag, so that pip-23.2.1.dist-info is automatically removed during the pip update.
Click to see the full analysis of the issue
This initially came up because of a CI failure triggered by safety which detected an old version of pip, even though we were running the latest version. This lead to this issue:
To debug the issue, I created the following test PR:
The output shows that the latest version of pip was installed from the beginning and correctly used by the other commands, but an ls shows 2 .dist-info dirs for pip.
To double-check, I created an empty workflow that only executes the ls, and the two .dist-info are still present:
I looked at the code of this repo to see how Python and pip where installed, and apparently it happens in:
This loop installs all the tools, including Python, from https://github.com/actions/python-versions
The code that actually installs Python and updates pip should be:
Here the --ignore-installed flag is used:
-I, --ignore-installed Ignore the installed packages, overwriting them. This can break your system
if the existing package is of a different version or was installed with a
different package manager!
I'm not sure if/why this is needed, but I verified locally that this flag leaves around the old .dist-info. When --ignore-installed is not used, only a .dist-info dir is present after the upgrade:
$ python3 -m venv venv && source venv/bin/activate
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
$ pip install --upgrade pip
...
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.3.2.dist-info
$ deactivate && rm -rf venvWhen --ignore-installed is used, the old pip-23.2.dist-info dir is left behind after the upgrade:
$ python3 -m venv venv && source venv/bin/activate
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
$ pip install --ignore-installed --upgrade pip
...
$ ls venv/lib64/python3.11/site-packages/ | grep pip
pip
pip-23.2.dist-info
pip-23.3.2.dist-infoRemoving --ignore-installed from nix-setup-template.sh should therefore fix the issue, assuming it is not needed for other reasons.
Also note that the same flag is also used elsewhere, e.g. in install-pypy.sh.
If my analysis is correct, I can prepare a PR (or more) to remove the --ignore-installed flag.
Platforms affected
- Azure DevOps
- GitHub Actions - Standard Runners
- GitHub Actions - Larger Runners
Runner images affected
- Ubuntu 20.04
- Ubuntu 22.04
- macOS 11
- macOS 12
- macOS 13
- macOS 13 Arm64
- Windows Server 2019
- Windows Server 2022
Image version and build link
This was tested on the following image:
- Image: ubuntu-22.04
- Version: 20231217.2.0
It likely affects other (all?) images.
See e.g. https://github.com/ezio-melotti/cherry-picker/actions/runs/7386618932/job/20093604056?pr=2
Is it regression?
No
Expected behavior
There should be only one version of pip installed, and only one .dist-info dir that matches the installed version.
Actual behavior
There are two .dist-info dirs.
Repro steps
Run this workflow to check:
name: Check installed pip versions
on: [pull_request, push, workflow_dispatch]
jobs:
check_pip:
runs-on: ubuntu-latest
steps:
- run: ls -l /opt/hostedtoolcache/Python/3.12.1/x64/lib/python3.12/site-packagesIt will output this:
drwxrwxrwx+ 5 runner runneradmin 4096 Dec 17 22:19 pip
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.2.1.dist-info
drwxrwxrwx+ 2 runner runneradmin 4096 Dec 17 22:19 pip-23.3.2.dist-info