Don't mix yum/dnf and pip for installation of system-wide Python packages

Posted on Fri 13 January 2017 in Python

Update (January 25, 2017): Updated instructions on preparing the Python virtual environment to work with setuptools 34+ on CentOS/RHEL 7. Also updated the Ansible playbook to skip seperately updating pip and setuptools upon creation of a new virtual environment on Fedora.

Too many times I've seen people using a mix of yum/ dnf and pip for installation of system-wide Python packages This causes all sorts of problems .

TL; DR Install system-wide Python packages with yum/dnf and only use pip inside a virtual environment. For an example, see my Ansible snippet below .

Note

I'll use CentOS 7 for the examples, but the concepts apply to all current Fedora and CentOS/RHEL distributions.

Problems with mixing yum/dnf and pip

People usually install the python2-pip package to boot-strap installation of pip on their systems:

sudo yum -y install epel-release
sudo yum -y install python2-pip

At the time of writing, this will install pip 8.1.2 on the system, while the latest pip version is 9.0.1. A user will soon notice that on each run of the pip command the following message is shown:

You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Naively, he will upgrade pip with:

sudo pip install --upgrade pip

which will replace the previous pip installation in system locations (e.g. /usr/bin, /usr/lib/python2.7/site-packages/pip, ...) with the new pip version.

The problem is that this will leave RPM with no clue as to what is going on. If you run rpm --verify python2-pip it will show that everything is broken:

S.5....T.    /usr/bin/pip
S.5....T.    /usr/bin/pip2
S.5....T.    /usr/bin/pip2.7
missing     /usr/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg-info
missing     /usr/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg-info/PKG-INFO
missing     /usr/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg-info/SOURCES.txt
missing     /usr/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg-info/dependency_links.txt
missing     /usr/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg-info/entry_points.txt
missing     /usr/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg-info/not-zip-safe
missing     /usr/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg-info/requires.txt
missing     /usr/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg-info/top_level.txt
S.5....T.    /usr/lib/python2.7/site-packages/pip/__init__.py
S.5....T.    /usr/lib/python2.7/site-packages/pip/__init__.pyc
missing     /usr/lib/python2.7/site-packages/pip/__init__.pyo
.......T.    /usr/lib/python2.7/site-packages/pip/__main__.py
S.5....T.    /usr/lib/python2.7/site-packages/pip/__main__.pyc
missing     /usr/lib/python2.7/site-packages/pip/__main__.pyo

[ ... output trimmed ... ]

S.5....T.    /usr/lib/python2.7/site-packages/pip/vcs/subversion.py
S.5....T.    /usr/lib/python2.7/site-packages/pip/vcs/subversion.pyc
missing     /usr/lib/python2.7/site-packages/pip/vcs/subversion.pyo
S.5....T.    /usr/lib/python2.7/site-packages/pip/wheel.py
S.5....T.    /usr/lib/python2.7/site-packages/pip/wheel.pyc
missing     /usr/lib/python2.7/site-packages/pip/wheel.pyo

Furthermore, when a new version of the python2-pip becomes available, yum upgrade will complain about not being able to find some files and directories.

For example, on upgrade from python-pip-7.1.0-1.el7 to python2-pip-8.1.2-5.el7 when pip was upgraded with pip in-between, yum upgrade gives the following warnings:

warning: file /usr/lib/python2.7/site-packages/pip/_vendor/_markerlib/markers.pyo: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip/_vendor/_markerlib/markers.pyc: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip/_vendor/_markerlib/markers.py: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip/_vendor/_markerlib/__init__.pyo: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip/_vendor/_markerlib/__init__.pyc: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip/_vendor/_markerlib/__init__.py: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip/_vendor/_markerlib: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info/top_level.txt: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info/requires.txt: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info/pbr.json: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info/not-zip-safe: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info/entry_points.txt: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info/dependency_links.txt: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info/SOURCES.txt: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info/PKG-INFO: remove failed: No such file or directory
warning: file /usr/lib/python2.7/site-packages/pip-7.1.0-py2.7.egg-info: remove failed: No such file or directory

Proper way to to use pip without mixing it with yum/dnf

The solution is to limit pip's use to installation of Python packages inside a Python virtual environment.

First install the python-virtualenv package with yum/dnf:

sudo yum -y install python-virtualenv

Then use it to create and activate a new virtual environment:

virtualenv myvenv
source myvenv/bin/activate

Note

At the time of writing, CentOS/RHEL 7's system-installed virtualenv package is very old (1.10.1) and creates a virtual environment with very old pip (1.4.1) and setuptools (0.9.8) packages. Hence, it is recommended to update them separately, before installation of other things in the virtual environment.

Since setuptools version 34, setuptools package no longer bundles its requirements and relies on installing whell distributions of its requirements. To install these wheels, it needs a newer version pip, hence pip needs to be updated separately before setuptools:

pip install -U pip
pip install -U setuptools

A complete Ansible playbook for proper creation of a Python virtual environment is provided later.

Then install whichever Python package you want inside the Python virtual environment.

Tip: Don't install the system pip package

To avoid mistakenly using pip outside a virtual environment, don't install the pip system package (e.g. python2-pip).

This way, pip executable will only be available after you activate the chosen virtual environment, which limits mistakes with using the system-wide pip to the minimum.

Ansible playbook for installation of packages inside a virtual environment

If you need to automate installation of python-virtualenv system package and creation of a Python virtual environment for installation of project's requirements inside this virtual environment, you can use the following Ansible playbook:

- name: Playbook for installation of packages inside a virtual environment
  hosts: all
  become: true

  vars:
    # adjust these variables to your project's needs
    venv_path: /opt/myvenv/
    requirements:
      - requests
      - simplejson
      - six

  tasks:
    - name: Install python-virtualenv package
      package: name=python-virtualenv state=installed

    - block:

      - name: Create virtual environment with up-to-date pip
        pip:
          virtualenv: "{{ venv_path }}"
          name: pip
          state: latest

      - name: Update virtual environment's setuptools
        pip:
          virtualenv: "{{ venv_path }}"
          name: setuptools
          state: latest

      when:
        - ansible_distribution in ["CentOS", "RedHat"]
        - ansible_distribution_major_version | int == 7

    - name: Install project's requirements in virtual environement
      pip:
        virtualenv: "{{ venv_path }}"
        name: "{{ requirements }}"
        state: latest

Note

Due to a bug in Ansible's pip module which causes Ansible to use system-installed pip2 executable and ignore the pip executable installed inside the virtual environment, you must use at least version 2.2.1 RC3 which fixes the bug.

Note

Fedora comes with virtualenv 14.0.1+ which automatically downloads new releases of pip, setuptools, wheel and their requirements from PyPI, therefore there is no need to update them separately before installing project's requirements in virtual environment.

If your project's requirements are not pure Python packages, but also include packages with C/C++ extensions, they'll need to be built as part of the installation process. Therefore, you'll need to have at least gcc and python-devel packages installed. You can use the following Ansible task to achieve that:

- name: Install project's building prerequisites
  package: name={{ item }} state=installed
  with_items:
    - gcc
    - python-devel
    # add other packages required to build your project's requirements

Put it before the Install project's requirements in virtual environement task.

yum dnf pip python packaging virtualenv fedora centos rhel