The FreeBSD devsummit that just passed by gave me enough insight into Jenkins to question the long-term plans for Kyua. Uh, WHAT?! Let me explain.
In the beginning...
One of the original but unstated goals of Kyua was to fix the "mess" that is the NetBSD releng test run logs site: if you pay close attention, you will notice that various individuals have reinvented the wheel over and over again in an attempt to automate release builds and test suite runs. In other words: different parties have implemented independent continuous integration systems several times with more or less success.
Ending up with such duplication of infrastructure was never an intended result of ATF as ATF should have had this functionality on its own. Unfortunately, my lack of experience on continuous integration when I started ATF seven years ago made ATF's plans and design not cover the truly end goal of having a system up and running for all the platforms we care about.
In other words: even if this was never published on the Kyua website, my long-term plan was to turn Kyua into a continuous integration platform. This would involve providing some form of automation to execute builds along the execution of tests, a mechanism to distribute build and test tasks to more than one machine, a comprehensive web interface to interact with the system, a monitoring system to ensure the platform keeps up and running, and much more.
As you can imagine, implementing all of these features is a humongous amount of work and Kyua is currently far from providing any of these. (Note: it is relatively simple to come up with a simple do-one-thing implementation, but productizing and productionizing the result is a completely different story — and I am not here to ship toy-quality software!)
Enter Jenkins
During the May 2014 FreeBSD devsummit, I got to learn about Jenkins: An extendable open source continuous integration server. Thanks to Craig Rodrigues, I had a local instance up and running in less than 20 minutes and was able to hack Kyua to output Jenkins-compatible reports in less than an hour of hands-on work. (Sorry, changes still not pushed to Github.)
As it turns out, Jenkins is no joke. It is a hugely popular piece of software used by high-profile projects and companies. Just think on holding a conference on such a niche tool and getting 400+ attendees to understand this.
My original reaction to the UI was to dismiss it as ugly (it indeed is), but a bit of poking revealed a large amount of hidden-but-easily-reachable functionality. And after visiting the FreeBSD Jenkins deployment, I quickly realized the power of this system once I saw it distributing jobs across a cluster and monitoring their status in real time and in a relatively nice way.
What a discovery. Jenkins does all I ever wanted Kyua to do and more, much more. "Fortunately," Jenkins lacks the pieces to actually define and execute the FreeBSD/NetBSD test suite, and this is exactly what Kyua offers today — which means that Kyua is not irrelevant.
Not enough time
So we have Jenkins out there that does most of what I ever wanted Kyua to do and we have Kyua which currently does not do any of it but fulfills the one missing piece in Jenkins. Hmm... sound like an opportunity to rethink Kyua's goals maybe. But why? It all revolves around time and spending it wisely in the things that will have the most impact with the fewer effort.
The time I can devote to Kyua is minimal these days, so at my current pace I —or, rather, we— will never reach those grandiose goals of having native continuous integration in Kyua. What's worse: the fact that I had such plans in mind but no time for them made me feel bad about Kyua overall (I'm pretty sure there is a name for this apathetic feeling but couldn't find it).
Learning about Jenkins has been a relief. "What if... what if I could reshape Kyua's goals to be less ambitious in scope? What if Kyua did one thing only, which would be to define and execute a test suite in a single configuration under a single machine, and excelled at it? Then you'd be free to plug Kyua into whichever continuous integration system that best suit your needs and Jenkins would be a reasonable choice." Those are the kinds of questions that are floating my mind since last week.
No downsides?
The main problem with Jenkins and the context in which we want to use it in (BSD systems) is, probably, the fact that Jenkins is written in Java. Java support on the BSDs has never been great, but things generally work OK under amd64 and i386. This actually is a serious issue considering all the platforms that FreeBSD and NetBSD support. Without going too far, I now would like to run Jenkins at home to offer continuous integration for Kyua itself... but my two servers are PowerPC-based machines — so no Jenkins for them yet, at least under FreeBSD.
Let us not get distracted from the core idea of splitting continuous integration out of Kyua just because Jenkins may not fit some obscure use case out there. Jenkins may be a very good solution to this, but it need not be the only one!
TL;DR
The more I think about it, the more I become convinced that Kyua's goals need to be simplified. Let Kyua focus on what it already is good at: defining a complex test suite and making sure that it can be run easily and deterministically. Let the remaining scaffolding for continuous integration to the big players out there. Jenkins, in this case, would be just one of the possible options for the continuous integration framework; we'd just need to make sure that Kyua plays well with at least one other system to ensure the design is generic.
If we went this route, Kyua would not need to be as complex as it already is. As a result, there are several things in the current design that could be yanked and/or streamlined to offer a simpler and more reliable piece of software.
It really all boils down to a very simple idea:
Optimize my little and precious time to have the greatest impact on the project and other people's work.
The alternative to this is to remain in the status quo, which is to naively wait for years until Kyua gets all of the necessary continuous integration features and makes Jenkins and other custom scripts unnecessary. And I cannot promise any of this will ever happen, nor I can say it makes sense at all.
I haven't made a decision yet as I would like these thoughts to solidly settle before announcing any changes widely, and I would also like to think about any possible simplifications to Kyua's design.
In the meantime: what are your thoughts?
