Python Virtual Environments Masterclass

TL;DR

This tutorial gives a rapid primer of Python Virtual Environments and how to use them in your effectively in your Python programming endeavours.

By the end you’ll know how to use Virtual Environments in Python 3 and 2, how to install modules, freezing and replaying environments, whether or not to activate an environment, how to package environments for reuse, how to create wheels of installed modules and how you can make use of the awesome pip-tools module for effective management.

Finally, I provide some context on other ways of managing virtual environments, outside of the python core pip package.

Introduction

When working on Python, or teaching Python to others, an area that I am an advocate for is the use of Virtual Environments for projects. Whilst many online training resources skip or leave such context until later on, personally, I feel that it is an area that is best covered at the earliest opportunity. From a simplistic viewpoint, it provides a convenient environment for individuals to experiment with the plethora of modules available in the python ecosystem, and, mitigates the need for root privileges when installing modules on Linux based systems. There are also a variety of other benefits that this post will aim to cover.

Creating Virtual Environments

In Both Python 2 and Python 3, virtual environments are available, however the creation of which varies between the two major languages and in version 3, there is both a preferred and legacy method. Given the future of Python 3 this post emphasises more on this version of Python, however, much of which in the post will apply to both versions.

Creating A Virtual Environment in Python 3 (Preferred Approach)

$ python3 -mvenv python3venv

Breaking this down, the -m option on the command line, tells the interpreter to run the venv module that is provided as part of the core Python3 distribution. The second parameter of ‘python3venv’ is the name of the virtual environment. The execution of this command should give you a virtual environment in the directory of python3venv.

If you’re inquisitive, you can understand more on what is happening behind the scenes by reviewing the actual venv module. A convenient way of identifying the source location, is through the REPL shell -

$ python3
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import venv
>>> print(venv)
<module 'venv' from '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/venv/__init__.py'>

With the code here being at ‘/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/venv/__init__.py’

If you jump to the bottom of the code in question, it has an execution block at the footer of the file allowing it to be executed as a script -

if __name__ == '__main__':
rc = 1
try:
main()
rc = 0
except Exception as e:
print('Error: %s' % e, file=sys.stderr)
sys.exit(rc)

Thus, executing the code directly, executes the main function that provides a list of available arguments to the user -

$ python3 -mvenv
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
[--upgrade] [--without-pip] [--prompt PROMPT]
ENV_DIR [ENV_DIR ...]
venv: error: the following arguments are required: ENV_DIR

Creating A Virtual Environment in Python 3.6 and Below (Deprecated Approach)

Python Version 3.6 and below, have an executable called pyvenv which is essentially a binary wrapper for the venv module. If you execute the binary as it is, you can see similarities in the usage syntax, with, an added deprecation warning in certain versions -

$ pyvenv
WARNING: the pyenv script is deprecated in favour of `python3.6 -m venv`
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear]
[--upgrade] [--without-pip] [--prompt PROMPT]
ENV_DIR [ENV_DIR ...]
venv: error: the following arguments are required: ENV_DIR

Running the executable through the strings command, shows a similar entry point with an implicit import of the venv module -

$ strings /Library/Frameworks/Python.framework/Versions/3.6/bin/pyvenv
#!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6
if __name__ == '__main__':
import sys
import pathlib
executable = pathlib.Path(sys.executable or 'python3').name
print('WARNING: the pyenv script is deprecated in favour of '
f'`{executable} -m venv`', file=sys.stderr)
rc = 1
try:
import venv
venv.main()
rc = 0
except Exception as e:
print('Error: %s' % e, file=sys.stderr)
sys.exit(rc)

Finally, executing the binary gives the same result as the preferred approach, with the added warning shown in the snippet -

$ pyvenv python3venv
WARNING: the pyenv script is deprecated in favour of `python3.6 -m venv`

Creating A Virtual Environment in Python 2.x

On Python 2.x, the virtualenv binary can be used to create the Virtual Environment, on my Mac, this is available as part of the Python 2.7 installation. In other operating systems your milage may vary and subsequently, virtualenv may need to be installed if it’s not already available.

Check for a virtualenv binary -

$ which virtualenv
/Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenv

Create a Python 2.x virtualenv. The syntax is similar to Python 3 with the target virtualenv directory. The Python 2 equivalent however gives a lot more in terms of output -

$ virtualenv python2venv
New python executable in /Users/james/python2venv/bin/python
Installing setuptools, pip, wheel...done.

Inspecting the Virtual Environments

In both cases, a similar structure is created -

$ ls python3venv python2venv/
python2venv/:
bin include lib

python3venv:
bin include lib pyvenv.cfg

The main difference being the pyvenv.cfg file that is visible in the root of the directory. If you’re interested in knowing more about this file PEP-0405 has a lot of context.

Activating the Virtual Environment

A convenient way of using the virtual environment is through activation, however, it is not an implicit requirement (something we’ll look into further). When you activate the virtual environment, the user’s shell prompt (PS1) is updated and subsequently, the command path gives higher priority to the virtual environment. If we check the PATH before activation, it may look something like this -

$ echo $PATH
/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Activate the Virtual Environment, n.b. the change to the prompt

$ source python3venv/bin/activate
(python3venv) $

And checking the PATH, my virtual environment now takes precedence with it being the first entry in the PATH -

(python3venv) $ echo $PATH
/Users/james/python3venv/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Python and pip usage in the venv

If we now execute either python or pip, owing to the PATH precedence, the binary from the virtual environment take priority

(python3venv) $ which python
/Users/james/python3venv/bin/python
(python3venv) $ which pip
/Users/james/python3venv/bin/pip

The Python environment, is automatically configured to search the environments paths for module initialisation, we can verify this with the REPL environment, note the addition of /Users/james/python3venv/lib/python3.6/site-packages to the sys.path list -

(python3venv) $ python
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/james/python3venv/lib/python3.6/site-packages']

Installing packages through pip

With the virtual environment activated, the use of pip will result in requested packages being installed direct to the virtual environment. For example, if we install the awesome datetime module called arrow -

(python3venv) $ pip install arrow
Collecting arrow
Downloading https://files.pythonhosted.org/packages/f4/7f/0360628ba40bb93c10cd89cd289b6a8e9ea87b2db884b8edf32c80ee1c73/arrow-0.13.1-py2.py3-none-any.whl
Collecting python-dateutil (from arrow)
Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
100% |████████████████████████████████| 235kB 3.6MB/s
Collecting six>=1.5 (from python-dateutil->arrow)
Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, arrow
Successfully installed arrow-0.13.1 python-dateutil-2.8.0 six-1.12.0
You are using pip version 9.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

It has installed both arrow, and a dependent module of six and python-dateutil. We can again confirm, that it is actually installed in the virtual environment, through the REPL interface -

(python3venv) $ python
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import arrow
>>> print(arrow)
<module 'arrow' from '/Users/james/python3venv/lib/python3.6/site-packages/arrow/__init__.py'>
>>> ^D

Using pip to upgrade pip

The eagle eyed amongst you, may have noticed the warning that pip provided above, where a new version is available. In the virtual environment, I typically recommend that people upgrade pip as a post action to creating the venv, prior to installing any modules but showed the approach above as an example of the warning that you may see. We can upgrade the environment as follows -

(python3venv) $ pip install --upgrade pip
Cache entry deserialization failed, entry ignored
Collecting pip
Using cached https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.1
Uninstalling pip-9.0.1:
Successfully uninstalled pip-9.0.1
Successfully installed pip-19.0.3

(python3venv) $ pip --version
pip 19.0.3 from /Users/james/python3venv/lib/python3.6/site-packages/pip (python 3.6)

Freezing Virtual Environments

If we install additional modules to make this example more effective, i.e. flask

(python3venv) $ pip install flask
Collecting flask
Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)
100% |████████████████████████████████| 92kB 2.4MB/s
Collecting Jinja2>=2.10 (from flask)
Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
100% |████████████████████████████████| 133kB 7.6MB/s
Collecting Werkzeug>=0.14 (from flask)
Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
100% |████████████████████████████████| 327kB 13.0MB/s
Collecting itsdangerous>=0.24 (from flask)
Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting click>=5.1 (from flask)
Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
100% |████████████████████████████████| 81kB 15.2MB/s
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask)
Downloading https://files.pythonhosted.org/packages/f0/00/a6aea33f5598b080b86d6b6d1214b51afe3ffa6100b902d5aa465080083f/MarkupSafe-1.1.1-cp36-cp36m-macosx_10_6_intel.whl
Installing collected packages: MarkupSafe, Jinja2, Werkzeug, itsdangerous, click, flask
Successfully installed Jinja2-2.10 MarkupSafe-1.1.1 Werkzeug-0.14.1 click-7.0 flask-1.0.2 itsdangerous-1.1.0

We have a significant number of modules, post Flask/Arrow installation. The convenient command of pip freeze, allows us to capture a snapshot of the virtual environment, in this example I’m using tee, to also send the output to a text file called requirements.txt

(python3venv) $ pip freeze | tee /tmp/requirements.txt
arrow==0.13.1
Click==7.0
Flask==1.0.2
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.1
python-dateutil==2.8.0
six==1.12.0
Werkzeug==0.14.1

Replaying a Virtual Environments Requirements

Earlier on, I created a Python 2 virtual environment, let’s use this, to replay the virtual environment configuration. Firstly, we deactivate the existing venv

(python3venv) $ deactivate
$

Then we activate the Python 2 virtual environment -

$ source python2venv/bin/activate
(python2venv) $

Upgrade pip, in most cases you should be able to use ‘pip install –upgrade pip’, however, I encountered an issue and had to perform the following workaround -

$ (python2venv) curl https://bootstrap.pypa.io/get-pip.py | python
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1659k 100 1659k 0 0 2375k 0 --:--:-- --:--:-- --:--:-- 2377k
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting pip
Using cached https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.1
Uninstalling pip-9.0.1:
Successfully uninstalled pip-9.0.1
Successfully installed pip-19.0.3

We can now replay the environment in our Python 3 setup, in the Python 2 environment, using the requirements.txt captured using pip freeze -

$ (python2venv) pip install -r /tmp/requirements.txt
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting arrow==0.13.1 (from -r /tmp/requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/f4/7f/0360628ba40bb93c10cd89cd289b6a8e9ea87b2db884b8edf32c80ee1c73/arrow-0.13.1-py2.py3-none-any.whl
Collecting Click==7.0 (from -r /tmp/requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Collecting Flask==1.0.2 (from -r /tmp/requirements.txt (line 3))
Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl
Collecting itsdangerous==1.1.0 (from -r /tmp/requirements.txt (line 4))
Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting Jinja2==2.10 (from -r /tmp/requirements.txt (line 5))
Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting MarkupSafe==1.1.1 (from -r /tmp/requirements.txt (line 6))
Downloading https://files.pythonhosted.org/packages/6d/d2/0ccd2c0e2cd93b35e765d9b3205cd6602e6b202b522fc7997531353715b3/MarkupSafe-1.1.1-cp27-cp27m-macosx_10_6_intel.whl
Collecting python-dateutil==2.8.0 (from -r /tmp/requirements.txt (line 7))
Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting six==1.12.0 (from -r /tmp/requirements.txt (line 8))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting Werkzeug==0.14.1 (from -r /tmp/requirements.txt (line 9))
Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl
Collecting backports.functools-lru-cache>=1.2.1; python_version == "2.7" (from arrow==0.13.1->-r /tmp/requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/03/8e/2424c0e65c4a066e28f539364deee49b6451f8fcd4f718fefa50cc3dcf48/backports.functools_lru_cache-1.5-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, backports.functools-lru-cache, arrow, Click, MarkupSafe, Jinja2, itsdangerous, Werkzeug, Flask
Successfully installed Click-7.0 Flask-1.0.2 Jinja2-2.10 MarkupSafe-1.1.1 Werkzeug-0.14.1 arrow-0.13.1 backports.functools-lru-cache-1.5 itsdangerous-1.1.0 python-dateutil-2.8.0 six-1.12.0

Pip freeze, will confirm that the Python 2 environment has the same versions installed as the Python 3 environment and is almost identical to what we saw previously. The version 2 equivalent also required a dependency of ‘backports.functools-lru-cache==1.5’ hence an additional module -

(python2venv) $ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
arrow==0.13.1
backports.functools-lru-cache==1.5
Click==7.0
Flask==1.0.2
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.1
python-dateutil==2.8.0
six==1.12.0
Werkzeug==0.14.1

Whilst I don’t necessarily recommend switching virtual environments between Python versions, the example was chosen to show the versitiliy of the pip freeze command and the ability, to replay an environment, albeit into the same version of a virtual environment, or another version whether major, or minor.

Using a Virtual Environment, without activation

Earlier, I made reference that it’s convenient, to activate a virtual environment but, in some cases, it’s also convenient not to activate it. For example, you may wish to call a script or application direct.

The python binary, in the virtual environment can be called directly and existing search paths are honoured. For example, if we call the binary direct without any virtualenv, we can still import the modules we’ve installed without error -

(python2venv) $ deactivate
$ python3venv/bin/python
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import arrow
>>> print(arrow)
<module 'arrow' from '/Users/james/python3venv/lib/python3.6/site-packages/arrow/__init__.py'>
>>> ^D

$ python2venv/bin/python
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 12:01:12)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import arrow
>>> print(arrow)
<module 'arrow' from '/Users/james/python2venv/lib/python2.7/site-packages/arrow/__init__.pyc'>
>>> ^D

Our scripts can also use the #! directive, making direct reference to the python binary, if we create a quick script -

$ cat <<EOF > test.py
> #!/Users/james/python3venv/bin/python
>
> import arrow
> utc = arrow.utcnow()
> print(utc)
> EOF

Set executable, and execute, we can confirm that modules within the venv are honoured -

$ chmod u+x test.py
$ ./test.py
2019-03-03T21:17:22.632130+00:00

Making it highly convenient for a script or application, to run with the specific requirements that are needed.

Packaging and re-use

With a virtual environment configured, it’s possible to package an entire virtual environment, and use it as an execution base providing that the target installation has the same version of Python that was used to create the virtual environment and that the virtual environment is placed in the same location. This is very convenient, when working in environments that have standardised builds. Essentially, the whole environment can be packaged and deployed to another system.

To quickly demonstrate, take a dump of the virtual env

$ tar cf python3venv.tar python3venv

Destroy the venv -

$ rm -rf python3venv

Extract the dump -

$ tar xf python3venv.tar

Test a dependent script -

$ ./test.py
2019-03-03T21:26:47.082092+00:00

Using pip-tools to manage Virtual Environments

Personally, I am a big fan of pip-tools, it provides a great way, of managing and maintaining virtual environments and their dependencies. From the examples above, we saw that the installation of both arrow and flask, resulted in a number of dependent modules. If for example we now remove both arrow and flask, the dependencies, remain in the virtual environment -

(python3venv) $ pip uninstall arrow flask
Uninstalling arrow-0.13.1:
Would remove:
/Users/james/python3venv/lib/python3.6/site-packages/arrow-0.13.1.dist-info/*
/Users/james/python3venv/lib/python3.6/site-packages/arrow/*
Proceed (y/n)? y
Successfully uninstalled arrow-0.13.1
Uninstalling Flask-1.0.2:
Would remove:
/Users/james/python3venv/bin/flask
/Users/james/python3venv/lib/python3.6/site-packages/Flask-1.0.2.dist-info/*
/Users/james/python3venv/lib/python3.6/site-packages/flask/*
Proceed (y/n)? y
Successfully uninstalled Flask-1.0.2
(python3venv) $ pip freeze
Click==7.0
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.1
python-dateutil==2.8.0
six==1.12.0
Werkzeug==0.14.1

Through the use of pip-tools, there is a cleaner way. If we start with a fresh virtual environment and install pip-tools -

(python3venv) $ deactivate
$ rm -rf python3venv
$ python3 -mvenv python3venv
$ source python3venv/bin/activate
(python3venv) $ pip install --upgrade pip
Cache entry deserialization failed, entry ignored
Collecting pip
Using cached https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 9.0.1
Uninstalling pip-9.0.1:
Successfully uninstalled pip-9.0.1
Successfully installed pip-19.0.3

(python3venv) $ pip install pip-tools
Collecting pip-tools
Downloading https://files.pythonhosted.org/packages/58/7a/f93b24807b7ac2d9d0bd6b8a886bcbe67eb39c1b1184b985dd5e0e2eca92/pip_tools-3.4.0-py2.py3-none-any.whl (43kB)
100% |████████████████████████████████| 51kB 2.3MB/s
Collecting six (from pip-tools)
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools)
Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Installing collected packages: six, click, pip-tools
Successfully installed click-7.0 pip-tools-3.4.0 six-1.12.0

We can now use an input file to track requirements. In this case, I’m going to create a requirements.in file, and I will implicitly put the versions of arrow and flask that I am interested in

(python3venv) $ cat <<EOF > requirements.in
> arrow==0.13.1
> Flask==1.0.2
> EOF

Using pip-compile, part of the pip-tools suite, we can create our requirements.txt from the requirements.in

(python3venv) $ pip-compile --generate-hashes --output-file requirements.txt requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes --output-file requirements.txt requirements.in
#
arrow==0.13.1 \
--hash=sha256:3397e5448952e18e1295bf047014659effa5ae8da6a5371d37ff0ddc46fa6872 \
--hash=sha256:6f54d9f016c0b7811fac9fb8c2c7fa7421d80c54dbdd75ffb12913c55db60b8a
click==7.0 \
--hash=sha256:2335065e6395b9e67ca716de5f7526736bfa6ceead690adf616d925bdc622b13 \
--hash=sha256:5b94b49521f6456670fdb30cd82a4eca9412788a93fa6dd6df72c94d5a8ff2d7 \
# via flask
flask==1.0.2 \
--hash=sha256:2271c0070dbcb5275fad4a82e29f23ab92682dc45f9dfbc22c02ba9b9322ce48 \
--hash=sha256:a080b744b7e345ccfcbc77954861cb05b3c63786e93f2b3875e0913d44b43f05
itsdangerous==1.1.0 \
--hash=sha256:321b033d07f2a4136d3ec762eac9f16a10ccd60f53c0c91af90217ace7ba1f19 \
--hash=sha256:b12271b2047cb23eeb98c8b5622e2e5c5e9abd9784a153e9d8ef9cb4dd09d749 \
# via flask
jinja2==2.10 \
--hash=sha256:74c935a1b8bb9a3947c50a54766a969d4846290e1e788ea44c1392163723c3bd \
--hash=sha256:f84be1bb0040caca4cea721fcbbbbd61f9be9464ca236387158b0feea01914a4 \
# via flask
markupsafe==1.1.1 \
--hash=sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473 \
--hash=sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161 \
--hash=sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235 \
--hash=sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5 \
--hash=sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff \
--hash=sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b \
--hash=sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1 \
--hash=sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e \
--hash=sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183 \
--hash=sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66 \
--hash=sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1 \
--hash=sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1 \
--hash=sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e \
--hash=sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b \
--hash=sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905 \
--hash=sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735 \
--hash=sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d \
--hash=sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e \
--hash=sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d \
--hash=sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c \
--hash=sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21 \
--hash=sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2 \
--hash=sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5 \
--hash=sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b \
--hash=sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6 \
--hash=sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f \
--hash=sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f \
--hash=sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7 \
# via jinja2
python-dateutil==2.8.0 \
--hash=sha256:7e6584c74aeed623791615e26efd690f29817a27c73085b78e4bad02493df2fb \
--hash=sha256:c89805f6f4d64db21ed966fda138f8a5ed7a4fdbc1a8ee329ce1b74e3c74da9e \
# via arrow
six==1.12.0 \
--hash=sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c \
--hash=sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73 \
# via python-dateutil
werkzeug==0.14.1 \
--hash=sha256:c3fd7a7d41976d9f44db327260e263132466836cef6f91512889ed60ad26557c \
--hash=sha256:d5da73735293558eb1651ee2fddc4d0dedcfa06538b8813a2e20011583c9e49b \
# via flask

The clear benefit of this approach, is that the dependencies also have hashes to ensure consistency of the desired modules.

Running pip-sync, again, part of the pip-tools suite will synchronise our virtual environment with that of the requirements.txt file

(python3venv) $ pip-sync
Collecting arrow==0.13.1 (from -r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 1))
Using cached https://files.pythonhosted.org/packages/f4/7f/0360628ba40bb93c10cd89cd289b6a8e9ea87b2db884b8edf32c80ee1c73/arrow-0.13.1-py2.py3-none-any.whl
Collecting flask==1.0.2 (from -r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 4))
Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl
Collecting itsdangerous==1.1.0 (from -r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 7))
Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting jinja2==2.10 (from -r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 10))
Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting markupsafe==1.1.1 (from -r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 13))
Using cached https://files.pythonhosted.org/packages/f0/00/a6aea33f5598b080b86d6b6d1214b51afe3ffa6100b902d5aa465080083f/MarkupSafe-1.1.1-cp36-cp36m-macosx_10_6_intel.whl
Collecting python-dateutil==2.8.0 (from -r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 42))
Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting werkzeug==0.14.1 (from -r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 45))
Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl
Requirement already satisfied: click>=5.1 in ./python3venv/lib/python3.6/site-packages (from flask==1.0.2->-r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 4)) (7.0)
Requirement already satisfied: six>=1.5 in ./python3venv/lib/python3.6/site-packages (from python-dateutil==2.8.0->-r /var/folders/xk/p46swdnn3b56rbd01qwdjhdh0000gn/T/tmpfwkk4j83 (line 42)) (1.12.0)
Installing collected packages: python-dateutil, arrow, werkzeug, itsdangerous, markupsafe, jinja2, flask
Successfully installed arrow-0.13.1 flask-1.0.2 itsdangerous-1.1.0 jinja2-2.10 markupsafe-1.1.1 python-dateutil-2.8.0 werkzeug-0.14.1
(python3venv) $ pip freeze
arrow==0.13.1
Click==7.0
Flask==1.0.2
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.1
pip-tools==3.4.0
python-dateutil==2.8.0
six==1.12.0
Werkzeug==0.14.1

Should however we wish to remove a component, for example Flask, we can update our requirements.in. When we repeat the process with pip-sync, it will remove Flask as well as Flask’s dependencies -

(python3venv) $ cat <<EOF > requirements.in
> arrow==0.13.1
> EOF
(python3venv) $ pip-compile --generate-hashes --output-file requirements.txt requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes --output-file requirements.txt requirements.in
#
arrow==0.13.1 \
--hash=sha256:3397e5448952e18e1295bf047014659effa5ae8da6a5371d37ff0ddc46fa6872 \
--hash=sha256:6f54d9f016c0b7811fac9fb8c2c7fa7421d80c54dbdd75ffb12913c55db60b8a
python-dateutil==2.8.0 \
--hash=sha256:7e6584c74aeed623791615e26efd690f29817a27c73085b78e4bad02493df2fb \
--hash=sha256:c89805f6f4d64db21ed966fda138f8a5ed7a4fdbc1a8ee329ce1b74e3c74da9e \
# via arrow
six==1.12.0 \
--hash=sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c \
--hash=sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73 \
# via python-dateutil
(python3venv) $ pip-sync
Uninstalling Flask-1.0.2:
Successfully uninstalled Flask-1.0.2
Uninstalling itsdangerous-1.1.0:
Successfully uninstalled itsdangerous-1.1.0
Uninstalling Jinja2-2.10:
Successfully uninstalled Jinja2-2.10
Uninstalling MarkupSafe-1.1.1:
Successfully uninstalled MarkupSafe-1.1.1
Uninstalling Werkzeug-0.14.1:
Successfully uninstalled Werkzeug-0.14.1
(python3venv) $ pip freeze
arrow==0.13.1
Click==7.0
pip-tools==3.4.0
python-dateutil==2.8.0
six==1.12.0

Using your Virtual Environment with a Specific PyPi Repository

In some circumstances, you may have a requirement to use a specific PyPi repository for installation. Should this be the case, the creation of a file called pip.conf, in the root of the Virtual Environment is enough to satisfy this dependency. An example pip.conf looks like the following, change hostname.spurin.com as required -

[global]
disable-pip-version-check = True
trusted-host = hostname.spurin.com
index = https://hostname.spurin.com/repository/pypi-all/pypi
index-url = https://hostname.spurin.com/repository/pypi-all/simple

Isolating Module Versions

Virtual Environments are a great way of isolating, specific versions of modules. Over time you may have a project, that has dependencies on a specific version whilst another, may benefit from newer functionality.

A particular module where this isolation is effective in Ansible and in the course that I created, Mastering Ansible, my preferred approach for installing Ansible is via a Virtual Environment. You can be specific with the version of Ansible that you install, can choose whether or not you leverage Python 2 or Python 3 (historically, Python 2 is a more reliable choice with Ansible) and you can have multiple versions of Ansible on the same system. The following diagram from the course demonstrates the effectiveness, of Virtual Environments -

Using Multiple Virtual Environments

Creating Wheels from your Virtual Environment

Wheel can be used to capture Python modules into zip like files that can be conveniently used for installations. They even handle complicated Python packages that require compilation (C extensions). Using such an approach, allows you to facilitate installations from one machine that has a compiler to another that may not.

(python3venv) $ pip wheel --wheel-dir=/tmp/wheel-dir -r requirements.txt
Collecting arrow==0.13.1 (from -r requirements.txt (line 7))
Using cached https://files.pythonhosted.org/packages/f4/7f/0360628ba40bb93c10cd89cd289b6a8e9ea87b2db884b8edf32c80ee1c73/arrow-0.13.1-py2.py3-none-any.whl
Saved /private/tmp/wheel-dir/arrow-0.13.1-py2.py3-none-any.whl
Collecting python-dateutil==2.8.0 (from -r requirements.txt (line 10))
Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Saved /private/tmp/wheel-dir/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting six==1.12.0 (from -r requirements.txt (line 14))
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Saved /private/tmp/wheel-dir/six-1.12.0-py2.py3-none-any.whl
(python3venv) $ ls /tmp/wheel-dir
arrow-0.13.1-py2.py3-none-any.whl python_dateutil-2.8.0-py2.py3-none-any.whl six-1.12.0-py2.py3-none-any.whl
(python3venv) $ pip uninstall arrow python_dateutil six
Uninstalling arrow-0.13.1:
Would remove:
/Users/james/python3venv/lib/python3.6/site-packages/arrow-0.13.1.dist-info/*
/Users/james/python3venv/lib/python3.6/site-packages/arrow/*
Proceed (y/n)? y
Successfully uninstalled arrow-0.13.1
Uninstalling python-dateutil-2.8.0:
Would remove:
/Users/james/python3venv/lib/python3.6/site-packages/dateutil/*
/Users/james/python3venv/lib/python3.6/site-packages/python_dateutil-2.8.0.dist-info/*
Proceed (y/n)? y
Successfully uninstalled python-dateutil-2.8.0
Uninstalling six-1.12.0:
Would remove:
/Users/james/python3venv/lib/python3.6/site-packages/six-1.12.0.dist-info/*
/Users/james/python3venv/lib/python3.6/site-packages/six.py
Proceed (y/n)? y
Successfully uninstalled six-1.12.0
(python3venv) $ pip install --find-links=/tmp/wheel-dir -r requirements.txt
Looking in links: /tmp/wheel-dir
Collecting arrow==0.13.1 (from -r requirements.txt (line 7))
Collecting python-dateutil==2.8.0 (from -r requirements.txt (line 10))
Collecting six==1.12.0 (from -r requirements.txt (line 14))
Installing collected packages: six, python-dateutil, arrow
Successfully installed arrow-0.13.1 python-dateutil-2.8.0 six-1.12.0

Closing remarks

This post, covers a lot of the use cases for how I personally use virtual environment and the traditional approach for setup and use. There are as to be expected with a topic such as this, a multitude of a ways of managing and maintaining your own virtual environments.

A popular module at the moment is Kenneth Reitz pipenv. It is an excellent module and as per his other endeavours, it has been met with significant praise. There is however a flip side to it’s use and both the article by Chris Warrick and the subsequent discussion in the hacker news thread provide another viewpoint. I recommend both trying the module for yourself and reading the counter arguments.

Personally, I think of pipenv as a fantastic tool for personal development and if your development cycle is more aligned to that approach then take a look. When working in a shared environment, having been a long time pip user prior to the release of pipenv, I’m personally happier with the approaches that I’ve outlined above. YMMV :-)

Share