Poetry Configuration
Configure a Poetry environment that runs in PyCharm
A quick and easy guide for dependency management and packaging in Python
Poetry is a wonderful open-source tool that can ease the pain of setting up your environment and configuring its dependencies.
In this blog, I will give you a tiny glance at Poetry’s capabilities and the power that lies within. I will also guide you through the technical process of configuring Poetry’s environment inside PyCharm.
Poetry has several advantages over a few existing tools for dependency management and packaging (for more detail refer to Poetry’s introduction and A Poetic Apology). Perhaps the key reason for Poetry’s growing popularity can be found in the quote below from Poetry’s introduction:
poetryis a tool to handle dependency installation as well as building and packaging of Python packages. It only needs one file to do all of that: the new, standardizedpyproject.toml.In other words, poetry uses
pyproject.tomlto replacesetup.py,requirements.txt,setup.cfg,MANIFEST.inandPipfile.
There are numerous different configuration options for using Poetry to manage Python packaging and dependencies. The one that I will present in this blog is the configuration that I found to be the most convenient and satisfies all my needs.
Installing Poetry
Follow Poetry’s documentation to install Poetry In accordance with your system environment.
Installing the Poetry Plugin for PyCharm
For PyCharm Versions >=2021.3, Poetry is already integrated without a manual installation of the plugin.
For PyCharm Versions < 2021.3, we will install the required plugin for PyCharm. You can find the list of Poetry versions for the different JetBrains IDEs below.
After clicking the link above, a popup will appear in the IDE with instructions on how to install the plugin.
Creating a new Poetry project
We are now ready to create our new Poetry environment! In this example, I’ll create a package called reviews that will be used for both training and serving our text classification model that extracts information pertaining to quality-of-life in residential complexes from tenant reviews (Haber and Waks, 2021).
poetry new reviewsThe output of this step is the pyproject.toml file:
[tool.poetry]
name = "reviews"
version = "0.1.0"
description = ""
authors = ["hai.rozencwajg <EMAIL>"]
[tool.poetry.dependencies]
python = "^3.7"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"My installed Python version is 3.7.9, and the Python version under the [tool.poetry.dependencies] section is ^3.7 , i.e., Python versions which are equal to or greater than 3.7and smaller than 4 version.
To read more about Poetry’s symbols such as the caret symbol (^) and additional version constraints, see Poetry’s dependency-specification guide.
Creating a Poetry Virtual Environment
Next, we will create a Poetry environment for this project by running the following command:
poetry installThis will create a local virtual environment that will be used for running the project based on the configurations listed in thepyproject.toml file.
The local environment was created in the default path for the Poetry local environments ~/Library/Caches/pypoetry/virtualenvs.
Adding the virtual environment as the interpreter
After clicking the Add Interpreter button in PyCharm, the existing interpreter was found automatically, and the Poetry virtual environment was automatically activated for this project.
Installing Python packages
Similar to the famous pip install command, Poetry also supports the installation of Python packages easily using the Poetry add command.
poetry add google-cloud-storageAfter running the above, a new line was added to thepyproject.toml file that documents the version for the google-cloud-storage package.
[tool.poetry]
name = "reviews"
version = "0.1.0"
description = ""
authors = ["hai.rozencwajg <EMAIL>"][tool.poetry.dependencies]
python = "^3.7"
google-cloud-storage = "^2.4.0"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
In addition, a poetry.lock file was created. This file keeps track of both the installed versions of packages and their dependencies, as listed in the log below:
Using version ^2.4.0 for google-cloud-storageUpdating dependencies
Resolving dependencies... (9.7s)Writing lock filePackage operations: 18 installs, 0 updates, 0 removals • Installing pyasn1 (0.4.8)
• Installing cachetools (5.2.0)
• Installing certifi (2022.6.15)
• Installing charset-normalizer (2.1.0)
• Installing idna (3.3)
• Installing protobuf (4.21.2)
• Installing pyasn1-modules (0.2.8)
• Installing rsa (4.8)
• Installing six (1.16.0)
• Installing urllib3 (1.26.10)
• Installing google-auth (2.9.0)
• Installing googleapis-common-protos (1.56.3)
• Installing requests (2.28.1)
• Installing google-api-core (2.8.2)
• Installing google-crc32c (1.3.0)
• Installing google-cloud-core (2.3.1)
• Installing google-resumable-media (2.3.3)
• Installing google-cloud-storage (2.4.0)
Using Poetry’s power to resolve compatibility issues
Next, I wanted to add the google-cloud-bigquery package to Poetry, so I ran the Poetry add command:
poetry add google-cloud-bigqueryBut this time, I got an error that demonstrates the power of Poetry:
Using version ^3.2.0 for google-cloud-bigqueryUpdating dependencies
Resolving dependencies... (0.2s)SolverProblemErrorThe current project's Python requirement (>=3.7,<4.0) is not compatible with some of the required packages Python requirement:
- google-cloud-bigquery requires Python >=3.6, <3.11, so it will not be satisfied for Python >=3.11,<4.0
Because google-cloud-bigquery (3.2.0) requires Python >=3.6, <3.11
and no versions of google-cloud-bigquery match >3.2.0,<4.0.0, google-cloud-bigquery is forbidden.
So, because reviews depends on google-cloud-bigquery (^3.2.0), version solving failed.at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
237│ packages = result.packages
238│ except OverrideNeeded as e:
239│ return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
240│ except SolveFailure as e:
→ 241│ raise SolverProblemError(e)
242│
243│ results = dict(
244│ depth_first_search(
245│ PackageNode(self._package, packages), aggregate_package_nodes• Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
For google-cloud-bigquery, a possible solution would be to set the `python` property to ">=3.7,<3.11"https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
https://python-poetry.org/docs/dependency-specification/#using-environment-markers
It turns out that there is a compatibility issue between the newest version of google-cloud-bigquery==3.2.0 and the possible Python versions listed in the pyproject.toml file.
Lucky for us, Poetry also provides us with a possible solution:
For google-cloud-bigquery, a possible solution would be to set the `python` property to ">=3.7,<3.11"
As we can see above, Poetry instructs us to be stricter about the Python version by limiting to >=3.7,<3.11 instead of the current setup ^3.7, so I changed the Python version in pyproject.toml file as suggested.
[tool.poetry]
name = "reviews"
version = "0.1.0"
description = ""
authors = ["hai.rozencwajg <EMAIL>"]
[tool.poetry.dependencies]
python = ">=3.7,<3.11"
google-cloud-storage = "^2.4.0"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"Now, after re-running the Poetry add command for thegoogle-cloud-bigquery package, the installation ended successfully.
Using version ^3.2.0 for google-cloud-bigqueryUpdating dependencies
Resolving dependencies... (91.8s)Writing lock filePackage operations: 8 installs, 1 update, 0 removals • Updating protobuf (4.21.2 -> 3.20.1)
• Installing grpcio (1.47.0)
• Installing grpcio-status (1.47.0)
• Installing numpy (1.21.6)
• Installing proto-plus (1.20.6)
• Installing google-cloud-bigquery-storage (2.14.0)
• Installing pyarrow (8.0.0)
• Installing python-dateutil (2.8.2)
• Installing google-cloud-bigquery (3.2.0)
We can also see that theprotobuf version was downgraded (automatically) to support all our version compatibility constraints.
The new pyproject.toml file now looks like this:
[tool.poetry]
name = "reviews"
version = "0.1.0"
description = ""
authors = ["hai.rozencwajg <EMAIL>"]
[tool.poetry.dependencies]
python = ">=3.7,<3.11"
google-cloud-storage = "^2.4.0"
google-cloud-bigquery = "^3.2.0"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"Other goodies
Installation of a specific package version
Similar to pip, Poetry supports the installation of specific package versions:
poetry add pandas~=1.3Installation of packages only for development environments
It is fun to say goodbye to multiple requirements.txt files. Poetry supports the management of both production and development environments under one file. Below I install pandas-profiling for data exploration use solely in the development environment.
poetry add pandas-profiling --devThe pandas-profilingpackage will be listed under the[tool.poetry.dev-dependencies] section instead of under[tool.poetry.dependencies].
Uninstalling packages
Uninstalling is as easy as adding packages:
poetry remove google-cloud-bigqueryOpen a new shell and activate it with a Poetry environment
poetry shellUpdate the Poetry lock
poetry lockConclusion
Poetry is a great tool with many capabilities that enable you as a developer to manage your dependencies in a clear, easy, and transparent way. Using Poetry facilitates the environment setting and helps resolve compatibility issues.
I would definitely continue to use Poetry for my future projects and I hope you also think so.
Hai Rozencwajg is a Lead Data Scientist at Skyline AI, a JLL company and part of JLL Technologies. We use large and diverse information alongside cutting-edge technology in order to provide real estate professionals with perspectives that generate unique business value.
