One non-trivial thing about Python is managing its versions, environments, dependencies – and thus shipping projects to other people. In this post we will have a look at the packaging and dependency manager poetry – which – in the author’s humble opinion – is the best way of doing said things.
Before getting started, let us define some terms and clear up a bit of confusion, which also I have fallen victim to initially: in addition to poetry, we will use pyenv – a Python installation manager. pyenv lets us manage different Python versions on our system, s.t. we can for example use Python 2.7 for project A, and 3.10 for project B – in addition preventing a pollution of our Python installation, if we were to simply install all needed packages to our system Python via pip install ...
.
On top of this, there are different packaging and dependency managers – such as poetry, but for example also pipenv, venv, or conda. In this post we will highlight poetry, which – as I mentioned earlier – I personally find the best of these solutions.
Installing Pyenv
We first install pyenv (assuming you are on a Linux machine, for all others operating systems please follow the installation guide):
curl https://pyenv.run | bash
Then, in pyenv, we install the Python version we want to use for our project, e.g.:
pyenv install -v 3.10.0
Setting up Poetry
With Python installed as described, we switch our focus to poetry.
Install via:
curl -sSL https://install.python-poetry.org | python3 -
Then, change into your Python project folder, and point poetry to the Python version you want to use – in our example case this is done via (you might need a pyproject.toml
file, see below):
pyenv local 3.10.0
poetry env use $(which python)
Working with Poetry
Poetry functions using two files: pyproject.toml
and poetry.lock
.
In the pyproject.toml
, we describe needed dependencies / packages, for example torch
. The developer, after adding all needed dependencies, then runs poetry update
.
This triggers the creation of the poetry.lock
file, which "captures" and describes the exact environment consisting of all packets from the pyproject.toml
. When now a user or another developer downloads the repo / code, they simply run poetry install
, and this will install all dependencies – resulting in them obtaining exactly the same Python Environment as intended by the first developer.
This is a huge advantage, and a must – as opposed to everybody just installing their own dependencies – not doing this just causes too much trouble.
To use your environment (run scripts, …), run poetry shell
. This activates your environment, and in it you can work as usual – e.g. run Python programs via python file.py
.
Sample Project
To conclude, let us put this all together in a sample project – and for the first time also unveil contents of a sample pyproject.toml
file.
Our project will consist of three files: main.py
, pyproject.toml
, and the auto-generated poetry.lock
file.
main.py
has the following content:
import matplotlib.pyplot as plt
import numpy as np
def plot():
x = np.linspace(0, 10, 50)
y = np.sin(x)
plt.plot(x, y)
plt.savefig("plot.png")
if __name__ == "__main__":
plot()
As we can see, we use numpy
and matplotlib
to generate a sinus curve and plot it. These modules thus need to be installed, which – ofc – we do with poetry.
Let us take a look at the contents of the pyproject.toml
:
The first section contains some metadata about the project:
[tool.poetry]
name = "myproject"
version = "0.1.0"
description = "..."
authors = ["hermanmichaels <[email protected]>"]
After this, it is time to define needed dependencies. You can (which is advised) also define a specific version for each package, to make the poetry update
process deterministic. Further, here we also define which Python version is expected. The following snippet installs matplotlib
andnumpy
, as well as some other packets I like to use for convenience (more on that in a future post!):
[tool.poetry.dependencies]
python = "3.10"
matplotlib = "3.5.1"
mypy = "0.910"
numpy = "1.22.3"
black = "22.3.0"
Putting it together:
[tool.poetry]
name = "myproject"
version = "0.1.0"
description = "..."
authors = ["hermanmichaels <[email protected]>"]
[tool.poetry.dependencies]
python = "3.10"
matplotlib = "3.5.1"
mypy = "0.910"
numpy = "1.22.3"
black = "22.3.0"
As explained above, to generate the poetry.lock
file, we run poetry update
. Then, another person using this project, can simply run poetry install
to obtain all our dependencies. You or them can then execute the programm by running:
poetry shell
python main.py
Summary
Let’s wrap up with a quick summary: poetry is a (the best, in my opinion) packaging and dependency manager. It builds upon a working Python installation, which I recommend to manage via the Python installation manager pyenv.
After following the installation and setup steps above, create a new project, and add a pyproject.toml file
. In this, define all needed dependencies.
Then, run:
poetry update
to generate the lock file (or update it, when having added new dependencies), which will be used by others to obtain the same set of packetspoetry install
to install all packets from the lock file (e.g., when you downloaded a new poetry project and want to install the dependencies – or your colleagues pushed a new version containing new packets)poetry shell
to enter the environment and run any Python script
This wraps up this introduction to poetry. Hope you find this useful in your future work -feel free to come back for more!