Linux Kernel needs

[epage]

I had a discussion with Miguel Ojeda (@ojeda) today about the needs of the Linux kernel for testing

Historically Linux's testing strategy has been decentralized.

• Some have taken the form of kernel syscalls doing end-to-end tests.
• More recently, there has been the kunit framework for in-kernel unit tests
• Miguel experimented with user-mode Rust unit tests but the harnessing was complex due to their special build requirements and they are very limited in what they test

Overall, our focus in those post will be on kunit. This is a Linux kernel framework that manages of running all tests, whether C or Rust. This is effectively taking the place of cargo test.

The first area we focused on was doc tests. They have a hack today where they run rustdoc with a custom runner (using an unstable CLI, see rust-lang/rust#102981) with a custom runner that just dumps all of the tests to a file. They then merge all of these files and register it with kunit.

See also https://lore.kernel.org/rust-for-linux/[email protected]/

The main issues are

• This is unstable
• The runner is given a fully formed test rather than a test body and metadata (source file and line), requiring them to find a symbol in the function and parse that data out (hacks all the way). The really just want an extraction tool

I think this aligns with my interest in trying to make all test runners appear the same and to try to speed up doc tests. I'll likely want that same metadata as Miguel dumped all at once for me to do with as I desire (and report it back as desired).

The second area was on kunit's special requirements on tests

All tests must report failures back up through a context pointer. The first way this hits is with assert_eq. They have a kunit_tests suite (mod) macro that imports their custom assert_eq. The second is access the pointer. Currently, they hack that in behind the scenes. One thought I had for this is if an assert reported errors back up through Result then the macro-generated caller can take that and register it with the context pointer.

They need to register tests. kunit_tests suite (mod) macro enumerates all tests with a #[test] macro and rewrites them. Distributed slice work could be a big help (#40).

They need test markers (fast, slow, stress). This could just be handled with a custom test macro. Unsure if reusing our work on test harnesses would help or not.

I asked about runtime detection support (like hardware) and ignoring of tests and there wasn't much interest, at least at this time.

---

We also got side tracked talking abut cargo. It sounds like they have similar needs to application developers of something around cargo.

When they did try to use cargo

• They were used to a single file per driver. Tracking Issue for cargo-script RFC 3424 rust-lang/cargo#12207 could help
• They had interleaved output between their build system and cargo
• They had issues with scaling of features and just want to use cfg

They are building std manually and if build-std became the only way to build it, then that could push them to cargo again.

[ojeda]

Thanks @epage -- I was writing a summary of what the kernel is looking for, which you may find useful, so I paste it here.

---

For rustdoc, the kernel would like a "proper" way to scrape/extract the tests (e.g. as a JSON output or similar, rather than directly "running" the tests), and then take it over from there.

We would essentially need the location of the test (path/file, line and "test number"), plus the content/text of the test. This will be written (by custom steps in the kernel build system) into a generated file, built and eventually run at some point within the kernel. Currently we figure those out (i.e. path etc.) via a hack on top of the --test-builder option.

We would also like to have the assert*! line numbers to print the location of a failing one (rather than the first line of the test). Currently we have an "anchor" to know where the start of the current test is (in the generated file), and then use that to compute back where the assert was in the original file. If a new approach to assert!s is provided by Rust, that returns Results, perhaps the error type/value could carry the line number?

There are also other bits of information that we may want to annotate the tests with and that rustdoc could pass. For instance, we may want to handle ignore or compile_fail on our own (rather than e.g. let rustdoc prefilter the ignored ones), or define custom categories/tags/... too.

For #[test]-like tests, we are looking into an approach like https://lore.kernel.org/rust-for-linux/[email protected]/, which uses a custom proc macro on a Rust mod so that it looks close to the usual Rust #[test]s. If doctests and normal tests are going to somehow be more integrated, or share features, perhaps the kernel may be able to take advantage of that too.

[ojeda]

I asked about runtime detection support (like hardware) and ignoring of tests and there wasn't much interest, at least at this time.

I think we want to eventually expose extra KUnit features to Rust kernel tests (such as expectations, skipping of tests, mocking, and so on). However, it remains to be seen how that will look like and how what Rust provides fits with that.

However, "static" annotations like ignore and compile_fail in tests (or categories/tags) would be interesting already for us (please see above).

[ojeda]

Miguel experimented with user-mode Rust unit tests but the harnessing was complex due to their special build requirements and they are very limited in what they test

To be clear, we currently support this within the kernel via the rusttest target. However, as you say, it is indeed quite involved for what it gives us (since those tests run in userspace so they cannot call kernel APIs etc.), but it was our first foray into using tests in the form of #[test]. We may just drop it from the kernel now that we have better integrations.