After three months of intensive work on Kyua's executor Git branch, I am happy to announce that the new execution engine, whose crown feature is the ability to run test cases in parallel, has just landed in master and passes all self-tests!
You can head over to the commit message for more details on the merge, read the NEWS entries, and skim throught the history of the executor branch to understand how this feature has been built.
One caveat: the history will look surprisingly short for a project that has spanned over three months. The reason is that, in the executor branch, I have routinely been using git rebase -i master to build a reduced set of changes that tell the story behind the implementation without distracting commits of the kind "Fix this little bug I forgot about" here and there. An unfortunate side-effect of this is that the temporal history of the commits makes no sense, and also that all the work I've been doing is not properly accounted for in GitHub's nice activity graphs; oh well, I consider a sane semantical history more important than these tiny details.
Why is this work important? Why is running tests in parallel such a big deal?
First, because the computing industry has fully moved into multiprocessor systems and thus taking advantage of multiple cores is something that any piece of modern software should do. As a little fun fact, Kyua is now able to stress my Mac Mini when running tests, spinning its fans to the maximum; this did not happen at all before.
But second, and I would say more importantly, because many tests are not resource-hungry or the system resources they stress do not overlap the resources used by other tests. For example: a significant number of tests are penalized by disk and time delays in them, which in turn cause the whole test suite to run for much longer than it would otherwise. Parallelization allows these long-running but not-heavy tests to run without blocking forward progress.
Let me also add that a secondary goal of this work is to optimize the inner workings of Kyua by reducing the system call overhead. In particular, eliminating one fork(2) call from every test case has been an explicit design goal. This especially helps Kyua when running on OS X, as fork is particularly expensive on the Darwin kernel (yes, citation needed).
As a very unscientific example: running the Kyua, ATF, Lutok, and shtk test suites with the old sequential execution engine takes about 2 minutes and 15 seconds in my dual-core FreeBSD virtual machine running on a dual-core Mac Mini. With the new implementation, the total run time goes down to 1 minute and 3 seconds using a parallelism setting of 4. Pretty cool I would say, but your mileage may (will) vary!
Are we done yet?
The merge of the executor branch marks the beginning of a major restructuring of Kyua's internals. As things are today, only the kyua test command has been switched to using the new execution engine, and not fully: only the execution of the test's body and cleanup routines happen through the executor; listing of test cases still happens as it did before. Similarly, both kyua list and kyua debug still use the out-of-process, testers-based, sequential implementation.
Therefore, there is a bunch of tricky work left to be done: the old test case execution engine (the runner plus the out-of-process testers) need to be fully removed, which in turn means that their functionality has to first be integrated into the new executor; there is a need for a feature to explicitly mark test programs as "exclusive", which is a prerequisite for tests that modify system-wide settings (as is done in the FreeBSD test suite); and an important regression needs to be fixed.
So... if we are not done, why merge now?
Because the current code is complete enough as a first piece of the whole puzzle. Even if the implementation does not yet meet my personal quality standards, the behavior of the code is already 80% of the way to my goal of fully switching to the new execution backend. You, as an end user, care about the behavior (not so much about the implementation), so by doing the merge now you can already start taking advantage of the new parallel execution functionality.
Also, because I am tired of managing a relatively large set of commits with git rebase -i. At this point, the set of commits that build the executor provide a good foundation for the code and its design. From now on, any other improvements to this codebase, such as the addition of new features or the correction of the existing regressions, should be properly tracked in the Git history.
And lastly because, at the beginning of 2015, I set myself the personal goal of getting this code merged by the end of February... so I just made the deadline! Which reminds me I gotta plan how the year's timeline looks like to reach Kyua 1.0.
Can I try it?
Of course! Please do!
There is no release available yet, but you can obviously fetch the code from the GitHub project page and and build it on your own! If you do that, do not forget to set parallelism=4 (or some other value greater than 1) in your ~/.kyua/kyua.conf file to enable the new behavior.
In fact, I am not going to cut a new release just yet because some of the issues mentioned above are of the "release-blocking" severity and thus must be resolved first. What I am going to do, though, is file bugs for each known issue so that they can be properly tracked.
Have fun and please share any feedback you may have!
