-
Notifications
You must be signed in to change notification settings - Fork 769
[UR] Move conformance tests over to lit #17998
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
8678110
to
1c859bd
Compare
1c859bd
to
0e7250e
Compare
0e7250e
to
4bdb2e5
Compare
4bdb2e5
to
9573682
Compare
9573682
to
8b7f5fa
Compare
8b7f5fa
to
be110df
Compare
be110df
to
185b853
Compare
185b853
to
ae9cb42
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please format the python to 80 columns and consider my final round of suggestions.
Benie can correct me if I'm wrong, but I don't think we have an established column limit for Python code? Black has a column limit of 88, but sometimes goes over it, especially with comments and strings. |
IMO, if formatter doesn't complain then formatting is OK. |
I hate this - especially as automated formatting is meant to give consistency; the rest of llvm has a strict 80 column limit which clang-format works extremely hard to maintain; and PEP-8 says 79(!). But since upstream LLVM have somehow decided darker is the formatter of choice, I concede. |
Just found this: https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style/68257/33 I'll take my grievances to discourse |
Conformance tests are now ran by LLVM's lit framework rather than ctest. A new `check-unified-runtime-conformance` target is available, which builds and runs all conformance tests. The `test` target will still run them. Some additional housekeeping was done: * `add_conformance_test_with_x_environment` was renamed to `add_conformance_x_test`. * Conformance tests now explicitly respect `ONEAPI_DEVICE_SELECTOR`. * If unspecified, `ONEAPI_DEVICE_SELECTOR` will now be populated with the list of built targets. Notably, this will exclude adapters which exist in the build directory but were not specified using a `BUILD_ADAPTER` cmake option (e.g, from a previous build before a reconfigure). * References to `UR_CTS_ADAPTER_PLATFORM` and `UR_CTS_BACKEND` are removed (they never did anything anyway). * Github build jobs have been updated. * The generated gtest binaries have been moved from `build/bin` to `build/test/conformance` * `requirements_testing.txt` now imports psutil for using timeouts. * Output format has been changed to show a progress bar and print the test output in real time. * Testing now respects the "device" filters from `ONEAPI_DEVICE_SELECTOR` (previously, the filter was only used to select the platform. `opencl:cpu` now works as expected. * Some CUDA tests are marked as failing since they are failing after this change. As a resut of these changes, a build with both the v1 and v2 versions of the level zero adapter will only run the v2 adapter. Building with only one version of the level zero adapter (as the Github CI currently does) works as expected and only runs that version.
@intel/dpcpp-devops-reviewers Just need a quick look at the workflow files - it's just a tweak to run the conformance testing in the new way. |
@@ -184,7 +171,8 @@ jobs: | |||
- name: Test adapters | |||
env: | |||
ZE_ENABLE_LOADER_DEBUG_TRACE: 1 | |||
run: env UR_CTS_ADAPTER_PLATFORM="${{matrix.adapter.platform}}" ctest -C ${{matrix.build_type}} --test-dir build --output-on-failure -L "conformance" --timeout 600 -VV | |||
LIT_OPTS: " --timeout 120" | |||
run: cmake --build build -j $(nproc) -- check-unified-runtime-conformance |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: if you use ninja you don't need to set -j
@intel/llvm-gatekeepers I think this should be ready to merge. @sarnex We don't use Ninja, I'm afraid. @igchor With the "building V1 and V2 at the same time" thing, I know this is a change in behaviour, but since it doesn't affect our CI I think it makes sense to merge this as is. If running both v1 and v2 in the same build is desired, I can create a followup MR to add a new cmake target to allow that. |
Conformance tests are now ran by LLVM's lit framework rather than ctest.
A new
check-unified-runtime-conformance
target is available, whichbuilds and runs all conformance tests. The
test
target will still runthem.
Some additional housekeeping was done:
add_conformance_test_with_x_environment
was renamed toadd_conformance_x_test
.ONEAPI_DEVICE_SELECTOR
.ONEAPI_DEVICE_SELECTOR
will now be populated withthe list of built targets. Notably, this will exclude adapters which
exist in the build directory but were not specified using a
BUILD_ADAPTER
cmake option (e.g, from a previous build before areconfigure).
UR_CTS_ADAPTER_PLATFORM
andUR_CTS_BACKEND
areremoved (they never did anything anyway).
build/bin
tobuild/test/conformance
requirements_testing.txt
now imports psutil for using timeouts.the test output in real time.
ONEAPI_DEVICE_SELECTOR
(previously, the filter was only used toselect the platform.
opencl:cpu
now works as expected.after this change.
As a resut of these changes, a build with both the v1 and v2
versions of the level zero adapter will only run the v2 adapter.
Building with only one version of the level zero adapter (as the
Github CI currently does) works as expected and only runs that
version.