[FIXED] Why does pip upgrade to different versions depending on python version?

Issue

On mac, I pulled 4 images 2.9.0, 2.6.1, 2.4.0, 2.3.1 using docker run tensorflow/tensorflow:2.9.0-gpu python3 -m pip install -U pip and observed the changes in pip version.

• All 4 containers started with pip 20.2.4.
• 2.3.1 and 2.4.0 had python 3.6.9 and their pips were upgraded to 21.3.1
• 2.6.1 and 2.9.0 had python 3.8.10 and their pips were upgraded to 22.2.2

1. Why this difference between 21.3.1 and 22.2.2 when both are using the same python3 -m pip install -U pip?

2. How does python version influence pip version?

3. Can 2.3.1 and 2.4.0 tensorflow images have 22.2.2 pip too? The 21.3.1 pip causes ERROR: No matching distribution found for scipy==1.7.3. It seems to have a smaller selection of available scipy packages to install.

I feel like i'm missing some concepts. Could someone introduce them, and some commands to debug/predict what python version will come with/upgrade to what pip version?

Edit

I just found a new way of debugging how pip works with verbose flag, so tried pip install -v scipy==1.7.3 using pip 21.3.1 and found out that different pips refer to the same index at https://pypi.org/simple/scipy/ which houses versions more than 1.5.4, the largest version that pip 21.3.1 was showing as available during No matching distribution error.

I initially thought different pip version have access to different portions of the index, and pip 21.3.1 had no way to see scipy 1.7.3. Now it looks like any pip version can see the full list of versions for a package, but the versions have to go through python version compatibility checks to be installed.

The last line from the verbose install was Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/4b/30/e15b806597e67057e07a5acdc135216ccbf76a5f1681a324533b61066b0b/pip-22.2.2.tar.gz#sha256=3fd1929db052f056d7a998439176d3333fa1b3f6c1ad881de1885c0717608a4b (from https://pypi.org/simple/pip/) (requires-python:>=3.7)

Not sure why there is a pip-22.2.2.tar.gz in the url. Previous lines have earlier pip versions there too, making me think this whole item by item scan not only requires python version to be satisfied, but also the pip version?

Edit 2

The last line with pip-22.2.2.tar.gz was a separate search (for pip versions) from the search for scipy versions. I thought the logs from searching list of pip versions were produced from searching the scipy list, that's why i got confused. Scrolling the logs more carefully shows that it searches scipy version first

1 location(s) to search for versions of scipy:
 * https://pypi.org/simple/scipy/

, then later does

1 location(s) to search for versions of pip:
 * https://pypi.org/simple/pip/

Solution

The reason is that Pip 22 doesn't not support python 3.6, so you're getting the latest supported version of pip for your python.

Similarly, https://docs.scipy.org/doc/scipy/release.1.7.0.html says "This release requires Python 3.7+".

Answered By - Tom Dalton