crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: getting crunch running on hadoop 2.0.0
Date Wed, 27 Jun 2012 04:43:56 GMT
None, but I'm not particularly knowledgable w/Eclipse's handling of
unit tests and how they get cleaned up. I rarely run more than one
unit test at a time in Eclipse. I'd be surprised about the conflicts,
since I think that all of the other unit tests run in local mode, and
don't make use of a mini cluster.

There was awhile there when the sbt test cycle of Scrunch was giving
me fits b/c I didn't realize that it was running all of the tests
simultaneously and they were conflicting with each other, but I doubt
that is the case here.

On Tue, Jun 26, 2012 at 9:39 PM, Gabriel Reid <> wrote:
> Something somewhat related: yesterday I had an issue when running the unit tests (or
a subset of the tests) from within Eclipse. I noticed that if the WordCountHbaseTest was run,
a lot of tests after it would fail. Not running the WordCountHbaseTest (ie setting @Ignore
on it) seemed to resolve the problem, an all tests would run through fine.
> This was only an issue when running in Eclipse; running the tests from maven seemed to
work fine.
> Any ideas on this?
> On 27 Jun 2012, at 06:26, Josh Wills <> wrote:
>> Hey all,
>> That last commit from me "fixes" (in some loose sense of the word) the
>> WordCountHBaseTest, which was the last test that we needed to get
>> working in order to have all of the Crunch unit tests passing under
>> Hadoop-2.0.0.
>> The complexity stems from the fact that the WordCountHBaseTest starts
>> up a mini hbase/mapreduce cluster in order to run, and I had to do
>> some work to make sure that all of the proper dependencies and class
>> definitions that were needed for the test case to pass were available
>> to the mini YARN instance that runs w/HBaseTestingUtility under hadoop
>> 2.0.0.
>> Vinod, if you get a chance to take a look at the test case and see if
>> there is anything I could have done to make it cleaner, I would be
>> much obliged.
>> J

Director of Data Science
Twitter: @josh_wills

View raw message