From user-return-60744-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed Aug 3 17:07:18 2016 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DD00C19F01 for ; Wed, 3 Aug 2016 17:07:18 +0000 (UTC) Received: (qmail 90450 invoked by uid 500); 3 Aug 2016 17:07:12 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 90277 invoked by uid 500); 3 Aug 2016 17:07:12 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 90266 invoked by uid 99); 3 Aug 2016 17:07:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Aug 2016 17:07:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id B13D91A047C for ; Wed, 3 Aug 2016 17:07:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.279 X-Spam-Level: * X-Spam-Status: No, score=1.279 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=databricks-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id ioYSeFfmuqiJ for ; Wed, 3 Aug 2016 17:07:09 +0000 (UTC) Received: from mail-lf0-f52.google.com (mail-lf0-f52.google.com [209.85.215.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 673B55FDC7 for ; Wed, 3 Aug 2016 17:07:09 +0000 (UTC) Received: by mail-lf0-f52.google.com with SMTP id g62so165700403lfe.3 for ; Wed, 03 Aug 2016 10:07:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=databricks-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=RnK7/gZbxudB4VCMx6QjQsmkM+SgwgG1opAffbDRfdQ=; b=KGAx+U0G/oCQHFeRsLputK7ac5nCi3vvNfFR57tjm6wKDuKm5SQ8V7QAK/cwj46oDH gP2xO2USvBv4pGwuBWLcJrrbBBUrhuDUGK++qKUc/WADKT915v3SRb999Ufoukgi2Qiy RDKxJMShtzmB9Pj8pHzH2kMYzrljjyzZgu3UvW7lycGA2EcuIt7OV2nD3PaR2Nc3QXty 3fR+ZyKx1YizRuOHkfPIXvHgrWUXApxxsXPJGqV37zE81OlCAnD8weos/HnuefAh4lcx zvsP46reE7XUTfGq41CoE66pgeCT2vMTq8esBpa4VmS89b8AOQ8q7+3i7LpPUr7PjXij drLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=RnK7/gZbxudB4VCMx6QjQsmkM+SgwgG1opAffbDRfdQ=; b=HFpKigrP4bMnvAwPUmpcY4ztJ93+cTiz0uSCyk3ilRbnbAPG386Hy9T5K6jPfxzrpo vpfG93Ii7/ZKzsgT92E1Wcb99pnKBkJTj6kqhyYRZseA1B5d/+X8RI4jdJqdeof3yiGP Obh/Y2wtG9/29oD5aq5H0t+MjIdTB9ciRgKD2zjfxhzSo0TKfVUYnI3khnDVfD/bMMuY be8s2Z6pVsz20288YwBhpW11lEwHeYOspE0wXpT64mF2c6zahFvq1RPg9NcZcgm/NiIz Ii/vT+0mMUNfNAMpmADLoOPj9ynJ0jg6ezzL4WGrfhOvmZC1jeoV0QHpaTo9KMyc569C JR5A== X-Gm-Message-State: AEkoouuGFeSL9MfDJExFbtbRduExs1Un2+YcJqkZNDNg8ILUHACYO5euegYMIrUvdY1Ri96a3CEsPnEVrObI5jYM X-Received: by 10.25.158.140 with SMTP id h134mr18593637lfe.0.1470244027546; Wed, 03 Aug 2016 10:07:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.216.129 with HTTP; Wed, 3 Aug 2016 10:06:46 -0700 (PDT) In-Reply-To: References: From: Michael Armbrust Date: Wed, 3 Aug 2016 10:06:46 -0700 Message-ID: Subject: Re: issue with coalesce in Spark 2.0.0 To: =?UTF-8?B?6ZmI5a6H6Iiq?= Cc: user Content-Type: multipart/alternative; boundary=001a114102a4fed19005392ddcb6 --001a114102a4fed19005392ddcb6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Spark 2.0 is not binary compatible with Spark 1.x, you'll need to recompile your jar. On Tue, Aug 2, 2016 at 2:57 AM, =E9=99=88=E5=AE=87=E8=88=AA wrote: > Hi all. > > > I'm testing on Spark 2.0.0 and found an issue when using coalesce in > my code. > > The procedure is simple doing a coalesce for a RDD[Stirng], and this > happened: > > * java.lang.NoSuchMethodError: > org.apache.spark.rdd.RDD.coalesce(IZLscala/math/Ordering;)Lorg/apache/spa= rk/rdd/RDD; > * > > The environment is *Spark 2.0.0* with *Scala 2.11.8*, while the > application jar is compiled with * Spark 1.6.2 * and * Scala 2.10.6 *. > > BTW, it works OK with *Spark 2.0.0* and *Scala 2.11.8*. So I presume > it=E2=80=99s a compatibility issue? > > > __ > > Best regards. > --001a114102a4fed19005392ddcb6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Spark 2.0 is not binary compatible with Spark 1.x, you'= ;ll need to recompile your jar.

On Tue, Aug 2, 2016 at 2:57 AM, =E9=99=88=E5=AE=87=E8= =88=AA <yuhang.chen@foxmail.com> wrote:
=09 =09 =09

Hi all.


=C2=A0 =C2=A0 I'm testing on Spark 2.0.0 and found an issue when usi= ng coalesce in my code.

=C2=A0 =C2=A0 The procedure is simple doing a coalesc= e for a RDD[Stirng], and this= happened:

=C2=A0 =C2=A0 =C2=A0 =C2=A0java.lang.NoSuchMe= thodError: org.apache.spark.rdd.RDD.coalesce(IZLscala/math/Ordering;)Lorg/a= pache/spark/rdd/RDD;

=C2=A0 =C2=A0 The environment is Spark 2.0.0 with Scala 2.11.8, while the application jar is compiled with Spark 1.6.2 and Scala 2.10.6 .=C2=A0

= =C2=A0 =C2=A0 BTW, it works OK with Spark 2.0.0 and Scala 2.11.8. So I presume it=E2=80=99s a compatibility issue?


__

Best regards.


--001a114102a4fed19005392ddcb6--