file_test_runner

cargo test in Rust is an excellent tool, but sometimes writing Rust code isn't the best way to maintain certain types of tests.

Case: Deno

Take Deno's codebase. The pattern for writing an integration test with the Deno binary has required writing code like the following in Rust:

// ~/tests/integration/run_tests.rs
itest!(002_hello {
  // these are relative from ~/tests/testdata
  args: "run --quiet --reload run/002_hello.ts",
  output: "run/002_hello.ts.out",
});

This macro expands to a #[test] function, which launches the Deno binary with the provided arguments and asserts its output against the provided file.

Problems:

1. Requires recompiling the test binary when adding/changing/deleting tests.
2. Test definition is in a different folder than the files being run or asserted—lots of tests in a single file using lots of data files in another folder.
   • Changing folders felt like context switching because of how far away they were.
   • It was hard to associate what testdata files were for what test.
     • These files were often not deleted when the test definition was deleted.
   • It didn't encourage developers to write a lot of tests.

Ideally a single test or group of related tests should be co-located in the same folder as the testdata files and not be defined in Rust.

Case: dprint's formatters

In dprint's formatter codebases, I've long stored tests in text files. For example, here's an example in dprint-plugin-json (JSON/JSONC formatter) at tests/specs/strings/Strings_All.txt:

~~ lineWidth: 80 ~~
== should support single quote strings ==
'te\'st'

[expect]
"te'st"

== should support double quote strings ==
"test\"test"

[expect]
"test\"test"

As you can see, it's groups of related tests stored in the same file.

Problems:

1. Had custom filtering.
2. Had its own custom infrastructure for running these tests.

Benefits:

1. Didn't require recompiling after updates.
2. Test file was tailored to the situation being tested.

Goals for writing a new test runner

I wanted a test runner that:

1. Allows storing the test definition close to the files used in the tests and the expected output.
2. Doesn't require recompiling Rust code when adding, changing, or deleting tests.
3. Is non-opinionated to allow structuring the tests according to the needs of the project.
4. Runs the tests in parallel.
5. Allows filtering via cargo test <test_name>.

Solution: file_test_runner

The solution I've settled on is file_test_runner.

This does two main steps:

1. Collects tests in any format on the file system.
2. Runs each test using custom provided code.

The basic setup is as follows:

1. Add a [[test]] section to the project's Cargo.toml with the default test harness disabled:

   [[test]]
   name = "specs"
   path = "tests/spec_test.rs"
   harness = false
   

2. Create a tests/spec_test.rs file to run the code:

   use file_test_runner::collect_and_run_tests;
   use file_test_runner::collection::CollectedTest;
   use file_test_runner::collection::CollectOptions;
   use file_test_runner::RunOptions;
   use file_test_runner::TestResult;
   
   fn main() {
     collect_and_run_tests(
       CollectOptions {
         base: "tests/specs".into(),
         strategy: Box::new(..omitted..),
         filter_override: None,
       },
       RunOptions {
         parallel: true,
       },
       // custom function to run the test...
       |test| {
         // do something like this, or do some checks yourself and
         // return a value like TestResult::Passed
         TestResult::from_maybe_panic(AssertUnwindSafe(|| {
           // run the test here
         }))
       }
     )
   }
   

3. Add test files or directories in any format to the tests/specs/ folder as specified above and update the code above to handle it.

Collecting tests

Tests can be collected from the file system using several strategies (note the strategy property under CollectOptions above).

For example, by a file in a directory:

// goes recursively through each directory under the base
// ("tests/specs") and finds the directories with a
// `__test__.jsonc` file
strategy: Box::new(TestPerDirectoryCollectionStrategy {
  file_name: "__test__.jsonc".into(),
})

Or by all descendant files:

// goes recursively through each directory under the base
// ("test/specs") excluding readme.md files and collects
// a test per file
strategy: Box::new(TestPerFileCollectionStrategy {
  file_pattern: None,
})

If you need more flexibility than that, you can implement your own file_test_runner::collection::strategies::TestCollectionStrategy:

pub trait TestCollectionStrategy<TData = ()> {
  fn collect_tests(
    &self,
    base: &Path
  ) -> Result<CollectedTestCategory<TData>, CollectTestsError>;
}

This is extremely flexible and even allows collecting multiple tests within the same file (the file_test_runner::collection::strategies::FileTestMapperStrategy is helpful for that).

Running tests

After tests are collected, file_test_runner will go through each category of tests running them in parallel on different threads, providing each test to the closure to run a test:

// custom function to run the test...
|test| {
  // do something like this, or do some checks yourself and
  // return a value like TestResult::Passed
  TestResult::from_maybe_panic(AssertUnwindSafe(|| {
    // Properties:
    // * `test.name` - Fully resolved name of the test
    // * `test.path` - Path to the test file this test is associated with
    // * `test.data` - Data associated with the test that may have been
    //                 set by the collection strategy
  }))
}

Benefits

1. file_test_runner handles collecting tests, orchestrating tests, and reporting test results to the console.
2. Tests can be added/modified/deleted without recompiling the Rust test binary—it's very fast to re-run a test.
3. Test reporter output looks very similar to cargo test's default, but also shows how long each test takes to run.
4. Tests can be filtered using cargo test <test_name>.
5. Tests can be structured in whatever makes most sense for what's being tested since it's non-opinionated. Stuff like snapshot testing can be implemented within the run test function.
6. Tests are run in parallel.

You can see the documentation for the implementation we ended up using in Deno here (we're still in the process of migrating all the itests to it).

Future

Right now there's not any support for cargo test -- --nocapture. I'm not entirely sure how to handle it (especially when a test uses multiple threads), but at the moment it doesn't capture any output within a test and a test implementation needs to handle that itself.

Overall I'm quite satisfied with this crate and it's made adding these kind of tests in several of Deno's repos much easier.