crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Reid <>
Subject Re: Exceptions in integration test cases
Date Tue, 02 Apr 2013 19:45:51 GMT

On 02 Apr 2013, at 21:39, Josh Wills <> wrote:

> In the context of that test (PCollectionGetSizeIT), the EOFException is
> expected-- there isn't any data to read b/c the file is empty. We could
> change up the impl to throw a CrunchRuntimeException instead.
> For the cogroup test, that sounds suspiciously like a deep copy issue.
> Gabriel is the most knowledgable person to ask, he always fixes my deep
> copy errors. :)

Well that's a bizarre coincidence -- I just encountered this object reuse bug in CoGroup earlier
today at work, and was going over mails from today while running the integration tests after
having written a patch for it :-)

In any case, yes, there is a deep copy issue with CoGroup -- I'm actually very surprised that
this is still in there, as I was sure that I had gone over CoGroup in the past to make sure
that object reuse wasn't an issue.

In any case, I'll commit the fix shortly.

- Gabriel

> On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich <> wrote:
>> Hi,
>> I'm a bit worried about the IO exceptions we get in our integration
>> test (see [1], grep for EOFException). Since we don't validate the
>> results, there may be data loss, at least in local mode.
>> I have stumbled upon a test case [2] where Crunch seems to reliably
>> lose some data when using Writables, while the same code works fine
>> with Avros. This time, I was able to reproduce this on two machines
>> (with slightly different error messages, ie. [3]).
>> Has anyone seen this before? Is anyone able to reproduce the error
>> or is it just me? :)
>> Thanks,
>>  Matthias
>> [1]
>> [2]
>> [3]

View raw message