Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Measure and report test coverage #6649

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sigmavirus24
Copy link
Contributor

This runs pytest within coverage, enables pytest-xdist to improve test run times, and reports the coverage on GitHub Actions for us.

This also ensures that the parallel mode coverage files are ignored and adds things to pytest configuration to make tox command simpler

@sigmavirus24 sigmavirus24 force-pushed the enable-coverage branch 4 times, most recently from 0af0ee6 to e3751d9 Compare February 24, 2024 15:32
@nateprewitt
Copy link
Member

I haven't had a chance to dig into the failure yet, but it's showing 26% coverage which seems extremely low. I tried to go back to find older examples of when codecov was running and am seeing closer to 89% coverage. Do we possibly have something misconfigured?

@sigmavirus24
Copy link
Contributor Author

Running in tox I get north of 80% as well but we're not running in tox on GitHub actions so that may be part of the problem

@nateprewitt
Copy link
Member

It looks like we're possibly hitting an issue with coverage combine similar to what's noted in nedbat/coveragepy#1036.

We're generating 21 files for the 21 tests across 7 interpreters. From Ned's response it sounds like those combinations have subtle variances that may result in false flags for missing coverage? When running the tests locally without parallel=True and generating a single file it shows 87% like I'd expect. If I use parallel = True and don't use combine, we get the same results we're seeing in CI as if we're missing coverage data.

Have you used this workflow in other projects? If not, we might consider dividing coverage by interpreter or another partitioning strategy.

@sigmavirus24
Copy link
Contributor Author

sigmavirus24 commented Mar 2, 2024

I mostly stole this from urllib3 and https://hynek.me/articles/ditch-codecov-python/ but maybe I missed something important

This runs pytest within coverage, enables pytest-xdist to improve test
run times, and reports the coverage on GitHub Actions for us.

This also ensures that the parallel mode coverage files are ignored and
adds things to pytest configuration to make tox command simpler
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants