Trekking Through Python’s “DLL Hell”: Versions, Packages, Virtual Environments

On my local system (Ubuntu of some flavor), I had Python 2.7. Sage (A nickname ChatGPT chose for itself when I asked) said to leave it alone or else bad things would happen. OK. My system also had Python 3.8, which Sage also said to leave alone. This time, I didn’t listen. I thought, “what could be the harm in going 3.8 to 3.12?” Of course, this is why we try these things out in a non-enterprise scenario… it fried my system: no more terminal windows, no more file explorer, mouse right-click stopped working correctly in my browser. Nutso things. I undid that, promptly, pointing my default OS python from 3.12 back to 3.8, and EVERYTHING returned to normal. Whew, I had simply underestimated to what level Ubuntu was leaning on Python vs C++.

Now, as more seasoned Pythonists would explain, 3.12 deprecated some goodies which 3.8 was still allowing. However, I am sitting here with THREE Python commands “python”, “python3”, “python3.12” and the corresponding pip, pip3 and pip3.12 commands. I bet that is clever, to someone, somewhere. . . who probably is like “mu hahaha I am genius!” and flicks their lights on and off to feel powerful.

I was too far in the weeds to contemplate how this could be done in the most extensible and efficient manner for an enterprise level approach across products/versions/teams. But the immediate fix for my Janet Jackson “Escapade” was a virtual environment (venv). Let’s say you want a project that is on Python 3.12 or whatever version you feel like using, and you don’t want to worry about the extant installs of Python on your system. You just want to deal with your project.  That’s the use case for a venv.

#in a terminal, from your top-level project folder (e.g. ~/my-code/ploopy)
python -m venv ploopy

This creates a virtual environment called “ploopy.” Activating on Windows/Linux is different but go talk to your favorite AI if I am missing your OS.  For linux it is:

source ploopy/bin/activate

and you will know it was successful because in your terminal window you will see the environment listed at the prompt:

(ploopy) me@mycomputer:~/bitbucket/ploopy-project-root

Generally, if you use a requirements.txt file to track your packages (like, Flask), you would run the pip command to restore those:

pip install -r requirements.txt

However, in my case everything was already installed. How is that possible? Oh, dear Python friends, it is because packages can be installed globally, even from your venv. And although this can save you time, generally, I would think (open to debate) this is not a great idea unless it is handled at an architectural level and everyone in the ecosystem either agrees or is otherwise corralled into subscribing to the approach.  In my case, my global stuff is there because it is. Not the right reason.

How did I know my stuff was coming from a globally installed package? Just examine where the package is installed using pip show:

pip show flask

That showed my flask was coming from a location that was outside my ploopy project. Don”t we all want healthy ploopy? Yes, we do.

After reading some fun like this little ditty about Ubuntu ignoring the venv (not sure that claim is wholly accurate, but it resonates with what I was experiencing):
https://askubuntu.com/questions/922789/ubuntu-pip-installs-packages-globally-instead-of-inside-virtualenv

Then reading about venvs in general to see if it addressed global packages (it did not, but still a good read):
https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/

I then stumbled into this little gem:
https://stackoverflow.com/questions/39648189/disable-global-installs-using-pip-allow-only-virtualenvs

Finally, a way to ground pip. Stay in your project, pip!! I ran the command:

pip config set global.require-virtualenv truepip install -r requirements.txt

For me, that did the trick (sans ye little issue in the section to follow) and pip installed all the packages to the venv instead of looking globally (And also polluting my global packages, which I now need to clean up… someday)

OK, ok, yes, this can happen. Because (sigh), the packages might be compatible with the global version of python and NOT compatible with the venv version of Python. For example, I had a package called cffi – it would not compile.  That’s because my venv was using Python 3.12, but the version of cffi in my requirements.txt was for Python 3.8 (where it was globally installed, 3.8 was a thing).  In 3.12, some of the calls were deprecated.

The solution was to manually install those in my venv without a version designation:

pip install cffi

If you try this, remember to remove that package from your current requirements.txt file.  Please understand, in large projects with LOTS of specifically interwoven package versions, my example will be oversimplified and there might be a more clever way to get through this exercise (Anaconda, anyone?! haha). But for my requirements.txt with 30-40 packages, it was no big deal to install a few manually.

Once everything is installed, and you can get through your “pip install -r requirements.txt” without any error messages, you can run and test your project. Then re-freeze a new requirements file:

pip freeze > requirements.txt

Yup, that was exhausting, but you survived. Ergo, you are entitled to a 5 minute break. Might I recommend 60 seconds of “Box Breathing” and a beverage. See you next time!