Dependency Management¶
Warning
Experimental Feature - PyScaffold support for virtual environment management is experimental and might change in the future.
The greatest advantage in packaging Python code (when compared to other forms
of distributing programs and libraries) is that packages allow us to stand on
the shoulders of giants: you don’t need to implement everything by yourself,
you can just declare dependencies on third-party packages and setuptools
,
pip
, PyPI and their friends will do the heavy lifting for you.
Of course, with great power comes great responsibility. Package authors must be careful when declaring the versions of the packages they depend on, so the people consuming the final work can do reliable installations, without facing dependency hell. In the opensource community, two main strategies have emerged in the last few years:
- the first one is called abstract and consists of having permissive, minimal and generic dependencies, with versions specified by ranges, so anyone can install the package without many conflicts, sharing and reusing as much as possible dependencies that are already installed or are also required by other packages
- the second, called concrete, consists of having strict dependencies, with pinned versions, so all the users will have repeatable installations
Both approaches have advantages and disadvantages, and usually are used together in different phases of a project. As a rule of thumb, libraries tend to emphasize abstract dependencies (but can still have concrete dependencies for the development environment), while applications tend to rely on concrete dependencies (but can still have abstract dependencies specially if they are intended to be distributed via PyPI, e.g. command line tools and auxiliary WSGI apps/middleware to be mounted inside other domain-centric apps). For more information about this topic check Donald Stufft post.
Since PyScaffold aims the development of Python projects that can be easily
packaged and distributed using the standard PyPI and pip
flow, we adopt the
specification of abstract dependencies using setuptools
’ install_requires
. This
basically means that if PyScaffold generated projects specify dependencies
inside the setup.cfg
file (using general version ranges), everything will
work as expected.
Test Dependencies¶
While specifying the final dependencies for packages is pretty much
straightforward (you just have to use install_requires
inside
setup.cfg
), dependencies for running the tests can be a little bit trick.
Historically, setuptools
provides a tests_require
field that follows
the same convention as install_requires
, however this field is not strictly
enforced, and setuptools
doesn’t really do much to enforce the packages
listed will be installed before the test suite runs.
PyScaffold’s recommendation is to create a testing
field (actually you can
name it whatever you want, but let’s be explicit!) inside the
[options.extras_require]
section of setup.cfg
. This way multiple test
runners can have a centralised configuration and authors can avoid double
bookkeeping.
If run py.test
runner, you will have to install those dependencies
manually, or do a editable install of your package with
pip install -e .[testing]
.
If you use tox
, you can list testing
under the the extras
configuration field option
(PyScaffold template for tox.ini
already takes care of this
configuration for you).
Note
If you prefer to use just tox
and keep everything inside
tox.ini
, please go ahead and move your test dependencies.
Every should work just fine :)
Warning
PyScaffold strongly advocates the use of test runners to guarantee
your project is correctly packaged/works in isolated environments.
A good start is to create a new project passing the --tox
option to
putup
and try running tox
in your project root.
Development Environment¶
As previously mentioned, PyScaffold will get you covered when specifying the
abstract or test dependencies of your package. We provide sensible
configurations for setuptools
and tox
out-of-the-box.
In most of the cases this is enough, since developers in the
Python community are used to rely on tools like virtualenv
and have a
workflow that take advantage of such configurations. As an example, someone
could do:
$ pip install pyscaffold
$ putup myproj --tox
$ cd myproj
$ python -m venv .venv
$ source .venv/bin/activate
# ... edit setup.cfg to add dependencies ...
$ pip install -e .
$ pip install tox
$ tox
However, someone could argue that this process is pretty manual and laborious to maintain specially when the developer changes the abstract dependencies. Moreover, it is desirable to keep track of the version of each item in the dependency graph, so the developer can have environment reproducibility when trying to use another machine or discuss bugs with colleagues.
In the following sections, we describe how to use two popular command line tools, supported by PyScaffold, to tackle these issues.
How to integrate Pipenv¶
We can think in Pipenv as a virtual environment manager. It creates
per-project virtualenvs and generates a Pipfile.lock
file that contains a
precise description of the dependency tree and enables re-creating the exact
same environment elsewhere.
Pipenv supports two different sets of dependencies: the default one, and the dev set. The default set is meant to store runtime dependencies while the dev set is meant to store dependencies that are used only during development.
This separation can be directly mapped to PyScaffold strategy: basically the
default set should mimic the install_requires
option in setup.cfg
,
while the dev set should contain things like tox
, sphinx
,
pre-commit
, ptpython
or any other tool the developer uses while
developing.
Note
Test dependencies are internally managed by the test runner, so we don’t have to tell Pipenv about them.
The easiest way of doing so is to add a -e .
dependency (in resemblance
with the non-automated workflow) in the default set, and all the other ones in
the dev set. After using Pipenv, you should add both Pipfile
and
Pipfile.lock
to your git repository to achieve reproducibility (maintaining
a single Pipfile.lock
shared by all the developers in the same project can
save you some hours of sleep).
In a nutshell, PyScaffold+Pipenv workflow looks like:
$ pip install pyscaffold pipenv
$ putup myproj --tox
$ cd myproj
# ... edit setup.cfg to add dependencies ...
$ pipenv install
$ pipenv install -e . # proxy setup.cfg install_requires
$ pipenv install --dev tox sphinx # etc
$ pipenv run tox # use `pipenv run` to access tools inside env
$ pipenv lock # to generate Pipfile.lock
$ git add Pipfile Pipfile.lock
After adding dependencies in setup.cfg
, you can run pipenv update
to
add them to your virtual environment.
Warning
Experimental Feature - Pipenv is still a young project that is moving very fast. Changes in the way developers can use it are expected in the near future, and therefore PyScaffold support might change as well.