Code Coverage (Python / DevOps)

Testing is important, especially in Continuous Integration / Continuous Deployment (CI/CD) environments, as it gives some reassurance that a code commit has not broken anything. But what happens if you are unsure how much of your project is covered by tests?

Code Coverage

Code coverage monitors tests as they run and checks which parts of the code the tests run against. This helps to give an impression of how much code is, or is not, covered by tests. Having lots of tests written looks impressive, but if they test only a small amount of the code (i.e. a low coverage) then those tests may be giving a false reassurance.

There are a variety of code coverage tools available for various programming languages, in this blog post I will be looking at coverage for Python and running it against my Flask app project.

Installing coverage

pip install coverage

Pip can be used to install coverage. I’d recommend installing it into the same virtual environment as your project.

Running coverage

I have tests for the project automatically running via a CI pipeline when new commits are pushed to the repository, however I can also run them manually on my local computer using the command:

flask test

This shows the tests running and (hopefully) passing, but it does not show how much of the code is covered by the tests. For this I need to add run coverage and have coverage run Flask (via a -m option):

coverage run -m flask test

The same tests have now run via coverage and the code coverage has been recorded.

Viewing Coverage Results

coverage report -m

coverage report -m outputs the results from the coverage run to show the results with the amount of cover for each file involved.

The results are outputted to the terminal, which does the job but coverage also has options for other outputs such as HTML.

coverage html

How Much Coverage Is Needed?

The level of code coverage needed is a consideration for every project as there are trade-offs for reaching higher levels of code coverage such as designing and building tests for each bit of missing coverage. More tests also means longer running tests suites.

Atlassian advises to aim for around 80%.