Summary:
- Better handle git references for sdists.
- Better handle direct whl references.
- Add an extra test that turned out to be not needed in the end, but I
  left it to increase the code coverage.

Work towards #2363
Fixes #2828

(cherry picked from commit a79bbfaece3e41f361b7d5baf89aec269184eb4d)