Enable test coverage

[PokhodenkoSA]

Ask dpNP team.

[diptorupd]

I spent some time exploring how this is done. For the C-API, we can use llvm-cov (https://clang.llvm.org/docs/SourceBasedCodeCoverage.html) and for the Python API coverage should work (https://developer.ibm.com/recipes/tutorials/testing-and-code-coverage-with-python/)

[vlad-perevezentsev]

@diptorupd, @PokhodenkoSA The dpNP team uses codecov.io and they are ready to help us. What do you say ?

[diptorupd]

@vlad-perevezentsev does Intel have an account with codecov.io? Is it not a paid service?

From a quick read of codecov.io page suggests that we will need to generate the coverage reports ourselves. We can surely use codecov.io services to generate reports. Also, it will be good to discuss with @xaleryb if code coverage generation using coverage.py or llvm-cov can be part of our internal CI. Maybe we can rely on a public CI instead?

In any case, we should definitely take help from the good folks at DPNP 👍

[diptorupd]

I played around with llvm-cov and was able to generate the summary and details reports using the tool. I will open a PR adding targets to our CMake scripts to automate the generation of the coverage reports.

@PokhodenkoSA question is where should these reports be stored?

E.g. for current master the C APi coverage is as follows:

Filename                              Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dpctl_utils.cpp                             2                 0   100.00%           2                 0   100.00%           6                 0   100.00%
dpctl_sycl_queue_manager.cpp              121                37    69.42%          21                 0   100.00%         315               103    67.30%
dpctl_sycl_queue_interface.cpp             77                50    35.06%          14                 4    71.43%         248               153    38.31%
dpctl_sycl_device_interface.cpp            71                24    66.20%          19                 3    84.21%         155                44    71.61%
dpctl_sycl_event_interface.cpp              2                 1    50.00%           2                 1    50.00%           7                 4    42.86%
dpctl_sycl_context_interface.cpp           17                 8    52.94%           4                 1    75.00%          32                14    56.25%
dpctl_sycl_usm_interface.cpp               14                 2    85.71%          10                 1    90.00%          64                 7    89.06%
dpctl_sycl_platform_interface.cpp          35                 5    85.71%           6                 0   100.00%         106                 9    91.51%
dpctl_sycl_kernel_interface.cpp            12                 3    75.00%           3                 0   100.00%          27                 5    81.48%
dpctl_sycl_program_interface.cpp           62                21    66.13%           7                 0   100.00%         175                62    64.57%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                     413               151    63.44%          88                10    88.64%        1135               401    64.67%

[PokhodenkoSA]

@PokhodenkoSA question is where should these reports be stored?

I think output to terminal during CI is ok. We could consider to upload report to 3rd party service if llvm-cov could generate report in some common and appropriate format like xml.

[diptorupd]

@vlad-perevezentsev @PokhodenkoSA I added the bits for the C API coverage in PR #242. I will be nice if we can generate the html report for both Python and C and upload them to our gh-pages site.

[PokhodenkoSA]

gh-pages are for documentation. I do not think we need to push temporary information like coverage related to particular PR into the documentation.
The best way is to upload coverage info to 3rd party service like https://about.codecov.io/ (good example for integration). Also printing coverage to output is ok for beginning.

[vlad-perevezentsev]

Part of the work done is contained in the PR #238
We implemented python,cython,and c++ file coverage using coverage and llvm-cov on Linux.
You need to install additional packages to cover
conda install coverage
conda install conda-forge::lcov
conda install conda-forge::gtest

Before installing the package, you need to add an environment variable
export CODE_COVERAGE=ON
It collects the debug version of dpctl and includes line tracing, which allows you to analyze the strings to create a coverage report.
The environment variable has been added for the convenience of configuring CI in the future.

Next, we install the package using the command
python setup.py develop

The dpctl-c-api-coverage folder will appear in the root folder after installation
The folder contains a report on the coverage of c++ files in html format.

You need to run tests to cover the cython and python files
coverage run -m unittest dpctl.tests
The required flags for the command coverage run are contained in the file .coveragerc
To show a report on the coverage results.
coverage report -m
To create an annotated HTML listings with coverage results.
coverage html
The htmlcov folder will appear in the root folder of the project. It contains reports on the coverage of python and cython files in html format.

Useful links:
https://coverage.readthedocs.io/en/coverage-5.0/cmd.html
http://blog.behnel.de/posts/coverage-analysis-for-cython-modules.html
https://alastairs-place.net/blog/2016/05/20/code-coverage-from-the-command-line-with-clang/
https://clang.llvm.org/docs/SourceBasedCodeCoverage.html
https://github.com/mapbox/cpp/blob/master/docs/coverage.md