Setting up Python Projects: Part III

Whether you’re a seasoned developer or just getting started with 🐍 Python, it’s important to know how to build robust and maintainable projects. This tutorial will guide you through the process of setting up a Python project using some of the most popular and effective tools in the industry. You will learn how to use GitHub and GitHub Actions for version control and continuous integration, as well as other tools for testing, documentation, packaging and distribution. The tutorial is inspired by resources such as Hypermodern Python and Best Practices for a new Python project. However, this is not the only way to do things and you might have different preferences or opinions. The tutorial is intended to be beginner-friendly but also cover some advanced topics. In each section, you will automate some tasks and add badges to your project to show your progress and achievements.

The repository for this series can be found at github.com/johschmidt42/python-project-johannes

Requirements

• OS: Linux, Unix, macOS, Windows (WSL2 with e.g. Ubuntu 20.04 LTS)
• Tools: python3.10, bash, git, tree
• Version Control System (VCS) Host: GitHub
• Continuous Integration (CI) Tool: GitHub Actions

It is expected that you are familiar with the versioning control system (VCS) git. If not, here’s a refresher for you: Introduction to Git

Commits will be based on best practices for git commits & Conventional commits. There is the conventional commit plugin for PyCharm or a VSCode Extension that help you to write commits in this format.

Overview

• Part I (GitHub, IDE)
• Part II (Formatting, Linting, CI)
• Part III (Testing, CI)
• Part IV (Documentation, CI/CD)
• Part V (Versioning & Releases, CI/CD)
• Part VI (Containerisation, Docker, CI/CD)

Structure

• Testing framework (pytest)
• Pytest configuration (_pytest.inioptions)
• Testing the application (Fastapi, httpx)
• Coverage (pytest-coverage)
• Coverage configuration (coverage.report)
• CI (test.yml)
• Badge (Testing)
• Bonus (Report coverage in README.md)

Testing framework

Testing your code is a vital part of software development. It helps you ensure that your code works as expected. You can test your code or application manually or use a testing framework to automate the process. Automated tests can be of different types, such as unit tests, integration tests, end-to-end tests, penetration tests, etc. In this tutorial, we will focus on writing a simple unit test for our single function in our project. This will demonstrate that our codebase is well tested and reliable, which is a basic requirement for any proper project.

Python has some testing frameworks to choose from, such as the built-in standard library unittest. However, this module has some drawbacks, such as requiring boilerplate code, class-based tests and specific assert methods. A better alternative is pytest, which is a popular and powerful testing framework with many plugins. If you are not familiar with pytest, you should read this introductory tutorial before you continue, because we will write a simple test without explaining much of the basics.

So let’s get started by creating a new branch: feat/unit-tests

In our app src/example_app we only have two files that can be tested: __init__.py and app.py . The __init__ file contains just the version and the app.py our fastAPI application and the GET pokemon endpoint. We don’t need to test the __init__.py file because it only contains the version and it will be executed when we import app.py or any other file from our app.

We can create a tests folder in the project’s root and add the test file test_app.py so that it looks like this:

.
...
├── src
│   └── example_app
│       ├── __init__.py
│       └── app.py
└── tests
    └── test_app.py

Before we add a test function with pytest, we need to install the testing framework first and add some configurations to make our lives a little easier:

Because the default visual output in the terminal leaves some room for improvement, I like to use the plugin pytest-sugar. This is completely optional, but if you like the visuals, give it a try. We install these dependencies to a new group that we call test. Again, as explained in the last part (part II), this is to separate app and dev dependencies.

Pytest configuration

Because pytest might not know where our tests are located, we can add this information to the pyproject.toml:

# pyproject.toml
...

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-p no:cacheprovider" # deactivating pytest caching.

Where addopts stands for "add options" or "additional options" and the value -p no:cacheprovider tells pytest to not cache runs. Alternatively, we can create a pytest.ini and add these lines there.

Testing the application

Let’s continue with adding a test to the fastAPI endpoint that we created in app.py. Because we use httpx, we need to mock the response from the HTTP call (https://pokeapi.co/api). We could use monkeypatch or unittest.mock to change the behaviour of some functions or classes in httpx but there already exists a plugin that we can use: respx

Mock HTTPX with awesome request patterns and response side effects.

Furthermore, because fastAPI is an ASGI and not a WSGI, we need to write an async test, for which we can use the pytest plugin: [pytest-asyncio](https://pypi.org/project/pytest-asyncio/) together with [trio](https://pypi.org/project/trio/) . Don’t worry if these are new to you, they are just libraries for async Python and you don’t need to understand what they do.

> poetry add --group test respx pytest-asyncio trio

Let’s create our test in the _testapp.py:

I won’t go into the details of how to create unit-tests with pytest, because this topic could cover a whole series of tutorials! But to summarise, I created an async test called test_get_pokemon in which the response will be the expected_response because we are using the respx_mock library. The endpoint of our fastAPI application is called and the result is compared to the expected result. If you want to find more about how to test with fastAPI and httpx, check out the official documentation: Testing in fastAPI

And if you have async functions, and don’t know how to deal with them, take a look at: Testing with async functions in fastAPI

Assuming that you installed your application with poetry install we now can run pytest with

> pytest

and pytest knows in which directory it needs to look for test files!

To make our linters happy, we should also run them on the newly created file. For this, we need to modify the command lint-mypy so that mypy also covers files in the tests directory (previously only src):

# Makefile

...

lint-mypy:
   @mypy .

...

At last, we can now run our formatters and linters before committing:

> make format
> make lint

Coverage

The code coverage in a project is a good indicator of how much of the code is covered by unit tests. Hence, code coverage is a good metric (not always) to check if a particular codebase is well tested and reliable.

We can check our code coverage with the coverage module. It creates a coverage report and gives information about the lines that we missed with our unit-tests. We can install it via a pytest plugin pytest-cov:

> poetry add --group test pytest-cov

We can run the coverage module through pytest:

> pytest --cov=src --cov-report term-missing --cov-report=html

To only check the coverage for the src directory we add the flag --cov=src . We want the report to be displayed in the terminal --cov-report term-missing and stored in a html file with --cov-report html

We see that a coverage HTML report has been created in the directory htmlcov in which we find an index.html.

.
...
├── index.html
├── keybd_closed.png
├── keybd_open.png
├── status.json
└── style.css

Opening it in a browser allows us to visually see the lines that we covered with our tests:

Clicking on the link _src/exampleapp/app.py we see a detailed view of what our unit-tests covered in the file and more importantly which lines they missed:

Coverage configuration

We notice that the code under the if __name__ == "main": line is included in our coverage report. We can exclude this by setting the correct flag when running pytest, or better, add this configuration in our pyproject.toml:

# pyproject.toml
...

[tool.coverage.report]
exclude_lines = [
    'if __name__ == "__main__":'
]

The lines after the if __name__=="__main__" are now excluded*.

*It probably makes sense to include other common lines such as

• def __repr__
• def __str__
• raise NotImplementedError
• …

If we run pytest with the coverage module again

> pytest --cov=src --cov-report term-missing --cov-report=html

the last line is not excluded as expected.

We have covered the basics of the coverage module, but there are more features that you can explore. You can read the official documentation to learn more about the options.

Let’s add these commands (pytest, coverage) to our Makefile, the same way we did in Part II, so that we don’t have to remember them. Additionally we add a command that uses the --cov-fail-under=80 flag. This signals pytest to fail if the total coverage is lower than 80 %. We will use this later in the CI part of this tutorial. Because the coverage report creates some files and directories within the project, we should also add a command that removes these for us (clean-up):

# Makefile

unit-tests:
   @pytest

unit-tests-cov:
   @pytest --cov=src --cov-report term-missing --cov-report=html

unit-tests-cov-fail:
   @pytest --cov=src --cov-report term-missing --cov-report=html --cov-fail-under=80

clean-cov:
   @rm -rf .coverage
   @rm -rf htmlcov

...

And now we can invoke these with

> make unit-tests
> make unit-tests-cov

and clean up the created files with

> make clean-cov

Continuous Integration (CI)

Once again, we use the software development practice CI to make sure that nothing is broken every time we commit to our default branch main.

Up until now, we were able to run our tests locally. So let us create our second workflow that will run on a server from Github! We have the option of using codecov.io in combination with the codecov-action OR we can create the report in the Pull Request (PR) itself with a pytest-comment action. I will choose the second option for simplicity.

We can either create a new workflow that runs parallel to our linter lint.yml (faster) or have one workflow that runs the linters first and then the testing job (more efficient). This is a design choice that depends on the project’s needs. Both options have pros and cons. For this tutorial, I will create a separate workflow (test.yml). But before we do that, we need to update our command in the Makefile, so that we create a pytest.xml and a pytest-coverage.txt, which are needed for the pytest-comment action:

# Makefile

...

unit-tests-cov-fail:
   @Pytest --cov=src --cov-report term-missing --cov-report=html --cov-fail-under=80 --junitxml=pytest.xml | tee pytest-coverage.txt

clean-cov:
   @rm -rf .coverage
   @rm -rf htmlcov
   @rm -rf pytest.xml
   @rm -rf pytest-coverage.txt

...

Now we can write our workflow test.yml:

Let’s break it down to make sure we understand each part. GitHub action workflows must be created in the .github/workflows directory of the repository in the format of .yaml or .yml files. If you’re seeing these for the first time, you can check them out here to better understand them. In the upper part of the file, we give the workflow a name name: Testing and define on which signals/events, this workflow should be started: on: ... . Here, we want that it runs when new commits come into a PullRequest targeting the main branch or commits go the main branch directly. The job runs in an ubuntu-latest (runs-on) environment and executes the following steps:

• checkout the repository using the branch name that is stored in the default environment variable ${{ github.head_ref }} . GitHub action: checkout@v3
• install Poetry with pipx because it’s pre-installed on all GitHub runners. If you have a self-hosted runner in e.g. Azure, you’d need to install it yourself or use an existing GitHub action that does it for you.
• Setup the Python environment and caching the virtualenv based on the content in the poetry.lock file. GitHub action: setup-python@v4
• Install the application & its requirements together with the test dependencies that are needed to run the tests with pytest: poetry install --with test
• Running the tests with the make command: [poetry run](https://python-poetry.org/docs/cli/#run) make unit-tests-cov-vail Please note, that running the tools is only possible in the virtualenv, which we can access through poetry run.
• We use a GitHub action that allows us to automatically create a comment in the PR with the coverage report. GitHub action: pytest-coverage-comment@main

When we open a PR targeting the main branch, the CI pipeline will run and we will see a comment like this in our PR:

It created a small badge with the total coverage percentage (81%) and has linked the tested files with URLs. With another commit in the same feature branch (PR), the same comment for the coverage report is overwritten by default.

Badge

To display the status of our new CI pipeline on the homepage of our repository, we can add a badge to the README.md file.

We can retrieve the badge when we click on a workflow run:

and select the main branch. The badge markdown can be copied and added to the README.md:

Our landing page of the GitHub now looks like this ❤:

If you are curious about how this badge reflects the latest status of the pipeline run in the main branch, you can check out the statuses API on GitHub.

---

Conclusion

Congratulations! You’ve completed the tutorial on how to use the testing framework pytest and the testing reporting tool coverage. You learned how to run them locally and how to set up a CI pipeline that checks that your code has a coverage percentage of more than 80%. You also learned how to show off your test results with a badge in your README.md file that indicates the status of the last testing pipeline in the main branch.

If you enjoyed this tutorial and found it helpful, please share it with your friends and colleagues. And if you have any questions or feedback, feel free to leave a comment below. I’d love to hear from you.

Thank you for reading and happy coding!

The next parts will cover: Documentation, Badges, Releases, Containerization, CI/CD and much more!

---

Join Medium with my referral link – Johannes Schmidt

BONUS:

Adding a job summary report

GitHub Actions allows us to generate custom markdown content on the run summary as explained in their blog post or in the documention. We can make use of this by outputting the content of the pytest-coverage.txt line by line to the environment variable called $GITHUB_STEP_SUMMARY. Any content added to this variable will then be displayed on the actions run summary page.

For this, we need to update the test.yml file:

In a linux shell, the >> operator is used to redirect the output of a command to a file, appending the output to the end of the file. The content is wrapped in a `` block and thecat` command is used to print the content to the standard output line by line.

Alternatively, we can add the content of the coverage report (markdown file) from the GitHub action we use to add a PR comment to the step summary so that we have a nice visual coverage summary:

The changes in the test.yml file are the following:

The GitHub Action MishaKav/pytest-coverage-comment stores some variables that are defined on the GitHub repository. One of these variables is the coverageHtml , that contains the html content with links to files of missing lines. Another option would be the summaryReport . We can access the variable’s content from another step by following the syntax of

steps.<step-id>.outputs.<variable-name>

We could also add the content to a existing Markdown file (i.e. README.md) using place-holders, but this would require us to give the runner (it’s identity) the permission to commit to the branch. If the branch is protected, by requiring a PR review, then it would require us to add these permissions accordingly. Permissions will be important in Part V, where we automatically create releases from commit messages.