spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <>
Subject Unit tests in < 5 minutes
Date Fri, 08 Aug 2014 16:01:57 GMT

Do we think it's both feasible and worthwhile to invest in getting our unit
tests to finish in under 5 minutes (or something similarly brief) when run
by Jenkins?

Unit tests currently seem to take anywhere from 30 min to 2 hours. As
people add more tests, I imagine this time will only grow. I think it would
be better for both contributors and reviewers if they didn't have to wait
so long for test results; PR reviews would be shorter, if nothing else.

I don't know how how this is normally done, but maybe it wouldn't be too
much work to get a test cycle to feel lighter.

Most unit tests are independent and can be run concurrently, right? Would
it make sense to build a given patch on many servers at once and send
disjoint sets of unit tests to each?

I'd be interested in working on something like that if possible (and


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message