From dev-return-2356-apmail-crunch-dev-archive=crunch.apache.org@crunch.apache.org Wed Apr 3 06:38:45 2013 Return-Path: X-Original-To: apmail-crunch-dev-archive@www.apache.org Delivered-To: apmail-crunch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9A9DBF477 for ; Wed, 3 Apr 2013 06:38:45 +0000 (UTC) Received: (qmail 77369 invoked by uid 500); 3 Apr 2013 06:38:45 -0000 Delivered-To: apmail-crunch-dev-archive@crunch.apache.org Received: (qmail 77268 invoked by uid 500); 3 Apr 2013 06:38:43 -0000 Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@crunch.apache.org Delivered-To: mailing list dev@crunch.apache.org Received: (qmail 77234 invoked by uid 99); 3 Apr 2013 06:38:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Apr 2013 06:38:42 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [212.227.126.186] (HELO moutng.kundenserver.de) (212.227.126.186) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Apr 2013 06:38:35 +0000 Received: from mafr.de (et-1-41.gw-nat.bs.ka.oneandone.net [212.227.66.11]) by mrelayeu.kundenserver.de (node=mrbap2) with ESMTP (Nemesis) id 0MF9xL-1UOhBM0PO7-00G3Gz; Wed, 03 Apr 2013 08:38:15 +0200 Date: Wed, 3 Apr 2013 08:38:13 +0200 From: Matthias Friedrich To: dev@crunch.apache.org Subject: Re: Exceptions in integration test cases Message-ID: <20130403063813.GA898@mafr.de> Mail-Followup-To: dev@crunch.apache.org References: <20130402173309.GA22302@mafr.de> <6B7903D1-A373-49BA-9334-81E9EEBCB2B9@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6B7903D1-A373-49BA-9334-81E9EEBCB2B9@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Provags-ID: V02:K0:zB95QyQHg5kypjGhIpJBw865SoiZTgqT1pnLeEltds1 qINyOmboKx7cb0/46agXNmHAYwMGjOanCkVx0ASbs9fiX6icQm UhHkUM2d1HJr0R95Pl9JkLL+de32M95haxz7Ys7kwCCjr7fhjG SCkC0PvceWF02W5nbOS7iEtkosDQzAdHLmZsNsDz/LM/zfQE6h BJzFPx+t0POp0902IvwBO/G/omfc3oUz+eo2FUk49eokVwMIxI YBz6bA8KRfZHPwPZHfg/aipPR1Nk77DmbXnwwK4+8okd34SojG u1ulPrIDQtwPv7URRuLA1yOnEt+cEidFDmcENSx459ymMCsMQ= = X-Virus-Checked: Checked by ClamAV on apache.org Fantastic, Gabriel. Thanks a lot! Regards, Matthias On Tuesday, 2013-04-02, Gabriel Reid wrote: > > On 02 Apr 2013, at 21:39, Josh Wills wrote: > > > In the context of that test (PCollectionGetSizeIT), the EOFException is > > expected-- there isn't any data to read b/c the file is empty. We could > > change up the impl to throw a CrunchRuntimeException instead. > > > > For the cogroup test, that sounds suspiciously like a deep copy issue. > > Gabriel is the most knowledgable person to ask, he always fixes my deep > > copy errors. :) > > Well that's a bizarre coincidence -- I just encountered this object reuse bug in CoGroup earlier today at work, and was going over mails from today while running the integration tests after having written a patch for it :-) > > In any case, yes, there is a deep copy issue with CoGroup -- I'm actually very surprised that this is still in there, as I was sure that I had gone over CoGroup in the past to make sure that object reuse wasn't an issue. > > In any case, I'll commit the fix shortly. > > - Gabriel > > > > On Tue, Apr 2, 2013 at 10:33 AM, Matthias Friedrich wrote: > > > >> Hi, > >> > >> I'm a bit worried about the IO exceptions we get in our integration > >> test (see [1], grep for EOFException). Since we don't validate the > >> results, there may be data loss, at least in local mode. > >> > >> I have stumbled upon a test case [2] where Crunch seems to reliably > >> lose some data when using Writables, while the same code works fine > >> with Avros. This time, I was able to reproduce this on two machines > >> (with slightly different error messages, ie. [3]). > >> > >> Has anyone seen this before? Is anyone able to reproduce the error > >> or is it just me? :) > >> > >> Thanks, > >> Matthias > >> > >> [1] > >> https://builds.apache.org/job/Crunch-master-integration/244/consoleFull > >> [2] http://users.mafr.de/~matthias/crunch/CogroupIT.patch > >> [3] http://users.mafr.de/~matthias/crunch/CogroupIT-error.txt > >> >