Hook up Codecov for coverage reporting on C code

[brettcannon]

See https://github.com/brettcannon/cpython-ci-test/blob/master/codecov.yml for an initial configuration (only differs from the default thanks to the new config for comments).

[tiran]

I have been running lcov regularly. gcov / lcov coverage of C code is a bit tricky for some modules. Some tests require a real terminal (ncurses), X server (tkinter, idle), root privileges (spwd) or special system configuration (yp for nis). The make coverage-lcov target ignores coverage of 3rd party code, too.

        @ # remove 3rd party modules, system headers and internal files with
        @ # debug, test or dummy functions.
        @lcov --remove $(COVERAGE_INFO) \
            '*/Modules/_blake2/impl/*' \
            '*/Modules/_ctypes/libffi*/*' \
            '*/Modules/_decimal/libmpdec/*' \
            '*/Modules/_sha3/kcp/*' \
            '*/Modules/expat/*' \
            '*/Modules/zlib/*' \
            '*/Include/*' \
            '*/Modules/xx*.c' \
            '*/Parser/listnode.c' \
            '*/Python/pyfpe.c' \
            '*/Python/pystrcmp.c' \
            '/usr/include/*' \
            '/usr/local/include/*' \
            '/usr/lib/gcc/*' \
            --output-file $(COVERAGE_INFO)

[brettcannon]

Would we run lcov as a separate coverage step from coverage.py? Or somehow run them simultaneously? I don't know what kind of slowdown we're looking at here if we had both on and if we might run afoul of Travis' free tier limitations.

[tiran]

On my machine it takes almost 10 minutes to run Python's test suite with a gcov instrumented build of Python. My laptop is pretty fast (8 cores i7-4900MQ CPU @ 2.80GHz, 16 GB RAM). I'm able to utilze all cores with 10 parallel test process. It's going to be hard enough to get lcov finish on Travis without coverage.py.

$ make coverage
$ time make test
...
real    9m40.226s
user    56m49.298s
sys     1m17.350s

[brettcannon]

OK, then we can look at running it as a separate test. If you want to try and get this working now then send a PR at https://github.com/brettcannon/cpython-ci-test as that's where I plan to pull the initial config files for Travis and Codecov.

[tiran]

I have to look into it. I'd like to use a service that can use lcov database so we can re-use out-of-tree build and lcov excludes from make coverage-lcov.

[brettcannon]

It looks like lcov is supported by Codecov: https://github.com/codecov/codecov-bash/blob/master/codecov#L822

[brettcannon]

Did you want me to leave this open for you, @tiran to look into lcov?

[abalkin]

Please take a look at https://github.com/abalkin/cpython/tree/ci or specifically at https://codecov.io/gh/abalkin/cpython/branch/ci. @brettcannon is that the "appropriate merging" that you are looking for?

[brettcannon]

@abalkin unfortunately no as there's no coverage for the Lib/ directory. If you include coverage runs for both lcov and coverage.py do the results show up in Codecov as if they were run together?

[abalkin]

Right. I thought I've seen both, but clearly my focus was on C coverage.

We are evaluating codecov's commercial offering in my firm at the moment, so I should have a working setup "soon".

[brettcannon]

Tweaked the title in hopes that someone is able to look into getting C code coverage working.

[ammaraskar]

Looked into this. Got it working on my fork here: https://codecov.io/gh/ammaraskar/cpython

Codecov makes this incredibly simple, they support auto merging of multiple reports, so both the python and C parts show up in the coverage report.

Some disadvantages, the coverage instrumented build takes longer to test on travis than without coverage. My build ran for ~40 minutes compared to ~20 minutes currently. This isn't too big of a concern since the coverage build is optional and does not block the travis status but its something to consider.

[ammaraskar]

and here it is as part of the pull request: https://codecov.io/gh/python/cpython/tree/f2148586600ea3a2c4a04550105f32d561f100aa

[brettcannon]

Great work! Hopefully one of us will be able to review python/cpython#7773 shortly.

[brettcannon]

This is now merged!