From user-return-77265-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed Feb 6 17:51:16 2019 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1B6A61897F for ; Wed, 6 Feb 2019 17:51:16 +0000 (UTC) Received: (qmail 90230 invoked by uid 500); 6 Feb 2019 17:51:08 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 90102 invoked by uid 500); 6 Feb 2019 17:51:08 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 90092 invoked by uid 99); 6 Feb 2019 17:51:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Feb 2019 17:51:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 123DFCAC0B for ; Wed, 6 Feb 2019 17:51:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.799 X-Spam-Level: X-Spam-Status: No, score=0.799 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id TGNRI6pjqXpB for ; Wed, 6 Feb 2019 17:51:06 +0000 (UTC) Received: from mail.norsys.fr (mail.norsys.fr [109.3.144.163]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 6E907623C2 for ; Wed, 6 Feb 2019 17:43:40 +0000 (UTC) Received: from SRVEXCH1.exchange.norsys.fr ([fe80::4db:a744:f2e4:6f61]) by SRVEXCH2.exchange.norsys.fr ([fe80::9da5:aa18:150b:eae9%20]) with mapi id 14.01.0438.000; Wed, 6 Feb 2019 18:43:34 +0100 From: Denis DEBARBIEUX To: Artur Sukhenko , Russell Spitzer CC: "user@spark.apache.org" Subject: RE : 3 equalTo "3.15" = true Thread-Topic: 3 equalTo "3.15" = true Thread-Index: AQHUvjfDc60jYk3/ck2yERLzsH0tBaXS5fyAgAAkWx8= Date: Wed, 6 Feb 2019 17:43:33 +0000 Message-ID: <904C740867261B499D9BA2352A9B9751020DD16EDD@SRVEXCH1.exchange.norsys.fr> References: , In-Reply-To: Accept-Language: fr-FR, en-US Content-Language: fr-FR X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.20.16.51] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 I am confused since the two column have the same name. ________________________________________ De : Artur Sukhenko [artur.sukhenko@gmail.com] Date d'envoi : mercredi 6 f=E9vrier 2019 17:32 =C0 : Russell Spitzer Cc : user@spark.apache.org Objet : Re: 3 equalTo "3.15" =3D true scala> df.select(colString, colShort, colShort.equalTo(colString)).explain =3D=3D Physical Plan =3D=3D LocalTableScan [tier_id#3, tier_id#56, (CAST(tier_id AS SMALLINT) =3D tier_= id)#50] On Wed, Feb 6, 2019 at 6:19 PM Russell Spitzer > wrote: Run an "explain" instead of show, i'm betting it's casting tier_id to a sma= ll_int to do the comparison On Wed, Feb 6, 2019 at 9:31 AM Artur Sukhenko > wrote: Hello guys, I am migrating from Spark 1.6 to 2.2 and have this issue: I am casting string to short and comparing them with equal . Original code is: ... when(col(fieldName).equalTo(castedValueCol), castedValueCol). otherwise(defaultErrorValueCol) Reproduce (version 2.3.0.cloudera4): scala> val df =3D Seq("3.15").toDF("tier_id") df: org.apache.spark.sql.DataFrame =3D [tier_id: string] scala> val colShort =3D col("tier_id").cast(ShortType) colShort: org.apache.spark.sql.Column =3D CAST(tier_id AS SMALLINT) scala> val colString =3D col("tier_id") colString: org.apache.spark.sql.Column =3D tier_id scala> res4.select(colString, colShort, colShort.equalTo(colString)).show +-------+-------+-------------------------------------+ |tier_id|tier_id|(CAST(tier_id AS SMALLINT) =3D tier_id)| +-------+-------+-------------------------------------+ | 3.15| 3| true| +-------+-------+-------------------------------------+ scala> Why is this? -- -- Artur Sukhenko -- -- Artur Sukhenko --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscribe@spark.apache.org