From user-return-35631-apmail-spark-user-archive=spark.apache.org@spark.apache.org Tue Jun 16 16:20:01 2015 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 92EC518AC4 for ; Tue, 16 Jun 2015 16:20:01 +0000 (UTC) Received: (qmail 54325 invoked by uid 500); 16 Jun 2015 16:19:58 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 54274 invoked by uid 500); 16 Jun 2015 16:19:58 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 54264 invoked by uid 99); 16 Jun 2015 16:19:57 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Jun 2015 16:19:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 81EAA1A56BA for ; Tue, 16 Jun 2015 16:19:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.999 X-Spam-Level: ** X-Spam-Status: No, score=2.999 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id b8u1eC8gwAK9 for ; Tue, 16 Jun 2015 16:19:49 +0000 (UTC) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0055.outbound.protection.outlook.com [157.56.110.55]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 3783B47BE6 for ; Tue, 16 Jun 2015 16:19:49 +0000 (UTC) Received: from BN1PR0101MB0852.prod.exchangelabs.com (10.160.168.151) by BN1PR0101MB0852.prod.exchangelabs.com (10.160.168.151) with Microsoft SMTP Server (TLS) id 15.1.190.14; Tue, 16 Jun 2015 16:19:40 +0000 Received: from BN1PR0101MB0852.prod.exchangelabs.com ([10.160.168.151]) by BN1PR0101MB0852.prod.exchangelabs.com ([10.160.168.151]) with mapi id 15.01.0190.013; Tue, 16 Jun 2015 16:19:40 +0000 From: Mark Tse To: "user@spark.apache.org" Subject: Unit Testing Spark Transformations/Actions Thread-Topic: Unit Testing Spark Transformations/Actions Thread-Index: AdCoT8D6xjLjg5OQR5CSkqNptCcYAQ== Date: Tue, 16 Jun 2015 16:19:40 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spark.apache.org; dkim=none (message not signed) header.d=none; x-originating-ip: [216.16.228.6] x-microsoft-exchange-diagnostics: 1;BN1PR0101MB0852;3:S5xqquIhg0Q++NZmGd/yuIzZEs/orekXFSGSxm2w4YUlq5xmY5OMKWKRBjgMks47jSNxjl29loWjdDqeGSRfaXDnmj23dXHZlxSawlNqiha+2JrGesF4ppWBomeKvN6+Dn7Bq5m6wL9wl8Pdf9t7Ew==;10:VavZPwLIHJx5t1pnHTAhbBU8Qpwbf1pNYDsNxBLPs4/Azf3uS3NUZzFLcRi+HL5if630pS4tZp8guEdaH2s3RMpLDK5GuWMGgzuu7ZFqtzA=;6:l+9hzCYn7li4jAz80bpZ5sUktz9heTH/DrYgFsRVueR5005DLoRbS4IUrR1oKt7b x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN1PR0101MB0852; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(520003)(5005006)(3002001);SRVR:BN1PR0101MB0852;BCL:0;PCL:0;RULEID:;SRVR:BN1PR0101MB0852; x-forefront-prvs: 06098A2863 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(164054003)(2351001)(19617315012)(74316001)(54356999)(229853001)(5003600100002)(2501003)(19300405004)(102836002)(15975445007)(2900100001)(5001920100001)(15395725005)(5001960100002)(110136002)(189998001)(107886002)(46102003)(40100003)(92566002)(450100001)(77156002)(62966003)(5002640100001)(66066001)(16236675004)(122556002)(86362001)(77096005)(2656002)(33656002)(87936001)(50986999)(19625215002)(19580395003);DIR:OUT;SFP:1101;SCL:1;SRVR:BN1PR0101MB0852;H:BN1PR0101MB0852.prod.exchangelabs.com;FPR:;SPF:None;MLV:sfv;LANG:en; Content-Type: multipart/alternative; boundary="_000_BN1PR0101MB085276E10A26036F962EC48CD6A70BN1PR0101MB0852_" MIME-Version: 1.0 X-OriginatorOrg: d2l.com X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jun 2015 16:19:40.3162 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 74bbca6d-410b-45b3-9b51-2a6aa6477079 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN1PR0101MB0852 --_000_BN1PR0101MB085276E10A26036F962EC48CD6A70BN1PR0101MB0852_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi there, I am looking to use Mockito to mock out some functionality while unit testi= ng a Spark application. I currently have code that happily runs on a cluster, but fails when I try = to run unit tests against it, throwing a "SparkException": org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 i= n stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0= (TID 1, localhost): java.lang.ClassCastException: cannot assign instance o= f java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.Java= RDDLike$$anonfun$foreach$1.f$14 of type org.apache.spark.api.java.function.= VoidFunction in instance of org.apache.spark.api.java.JavaRDDLike$$anonfun$= foreach$1 at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStr= eamClass.java:2089) (Full error/stacktrace and description on SO: http://stackoverflow.com/q/30= 871109/2687324). Has anyone experienced this error before while unit testing? Thanks, Mark --_000_BN1PR0101MB085276E10A26036F962EC48CD6A70BN1PR0101MB0852_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi there,


I am looking to use Mockito to mock out some functionality while unit testi= ng a Spark application.

 

I currently have code that happily runs on a cluster= , but fails when I try to run unit tests against it, throwing a “Spar= kException”:

 

org.apache.spark.SparkException: Job aborted due to = stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Los= t task 0.0 in stage 1.0 (TID 1, localhost): java.lang.ClassCastException: c= annot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.f$14 of = type org.apache.spark.api.java.function.VoidFunction in instance of org.apa= che.spark.api.java.JavaRDDLike$$anonfun$foreach$1

    at java.io.ObjectStreamClass$Fiel= dReflector.setObjFieldValues(ObjectStreamClass.java:2089)

 

(Full error/stacktrace and description on SO: http://stackoverflow.com/q/30871109/2687324).

 

Has anyone experienced this error before while unit = testing?

 

Thanks,

Mark

--_000_BN1PR0101MB085276E10A26036F962EC48CD6A70BN1PR0101MB0852_--