From user-return-61500-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed Aug 17 18:04:04 2016 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0B07419B62 for ; Wed, 17 Aug 2016 18:04:04 +0000 (UTC) Received: (qmail 83979 invoked by uid 500); 17 Aug 2016 18:03:58 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 83860 invoked by uid 500); 17 Aug 2016 18:03:58 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 83850 invoked by uid 99); 17 Aug 2016 18:03:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Aug 2016 18:03:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0E3F3C869E for ; Wed, 17 Aug 2016 18:03:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.449 X-Spam-Level: * X-Spam-Status: No, score=1.449 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 0rF3GDlspLwn for ; Wed, 17 Aug 2016 18:03:56 +0000 (UTC) Received: from mail-io0-f172.google.com (mail-io0-f172.google.com [209.85.223.172]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 94A3E5FBD7 for ; Wed, 17 Aug 2016 18:03:55 +0000 (UTC) Received: by mail-io0-f172.google.com with SMTP id 38so142484827iol.0 for ; Wed, 17 Aug 2016 11:03:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=fB9/jemqG9g6uHEdqwXH++cEd03l5pwegKv0V/TQnI8=; b=E674jGr1ue3aeH2xwZulIWlvZDxPU/rfQgEPhd1wAuQZvv0Cwt9xzVLqf7DU6TTDv1 QCasZFJY7sKKKUBqeO11/hy/xquhy2CbhObcVJ3YsQZTWvQYhwBq1ij/VhJCT9YjVPFE i8wKt5f3Ukb6mkO4tl1cRoVdvh1wrAr9+U8MyIXGsoHKorw5JrYsNpa5XVcvb3LQ3UTL ZStqypO5rnsMDCDvL+vFBPh2UQZxEbOtHmKum78GtdFIHDfscukrV04nYfK8DgsixXJw WVmAfmkJ7ZTbdoQ9hvpjzucV2yPmyW9h6qI9Pl7IVNFFdBYY7gNl2lMpCZsv5HedBdf9 JySg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=fB9/jemqG9g6uHEdqwXH++cEd03l5pwegKv0V/TQnI8=; b=dvYmDBaw8L1Oa5P7J7HIYyhNnBkhhe06lclmPMFrbavEBkgiQC2jq0YwYg+sRN3Tb9 MIJWDAq0ofSqNQy+8Vf3TCMAXqwmFD6lVfw6QyBqnLuebnBcZhmUGJpJZuXYcDUCLM+U n76bLYK92s/DAw/yePtksOkv89tUvpKrbLcOTZzADHMjxvaiibwSPb6ZDaj0c1BKKAKc p+dnKxMx9yApLZkHc+8QyvnSymmj9j0N8+bws8li9wYVHQXxzTBmxG9mqZk/hE86Ps7l x1YJD1mk7NT54ui9rzt2Z+aOa4Z6Wa/Oh3WV0X9kQR0mC/7S8aI6O72Z7290QAeVUga4 Dlmg== X-Gm-Message-State: AEkoouvg2O/d6UspUbDj0S1osOybnMejwSJ9aTxCZDRTHWFQgeIFIFnsCvNIIRLWgU4xMp5VlloYLW9yUYRkDw== X-Received: by 10.107.55.70 with SMTP id e67mr49529972ioa.51.1471457034509; Wed, 17 Aug 2016 11:03:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.16.22 with HTTP; Wed, 17 Aug 2016 11:03:54 -0700 (PDT) In-Reply-To: References: From: vr spark Date: Wed, 17 Aug 2016 11:03:54 -0700 Message-ID: Subject: Re: Undefined function json_array_to_map To: Ted Yu , user Content-Type: multipart/alternative; boundary=001a114a6d76d829c1053a4849b7 --001a114a6d76d829c1053a4849b7 Content-Type: text/plain; charset=UTF-8 Hi Ted/All, i did below to get fullstack and see below, not able to understand root cause.. except Exception as error: traceback.print_exc() and this what i get... File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/context.py", line 580, in sql return DataFrame(self._ssql_ctx.sql(sqlQuery), self) File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 51, in deco raise AnalysisException(s.split(': ', 1)[1], stackTrace) AnalysisException: u'undefined function json_array_to_map; line 28 pos 73' On Wed, Aug 17, 2016 at 8:59 AM, vr spark wrote: > spark 1.6.1 > python > > I0817 08:51:59.099356 15189 detector.cpp:481] A new leading master (UPID= > master@10.224.167.25:5050) is detected > I0817 08:51:59.099735 15188 sched.cpp:262] New master detected at > master@x.y.17.25:4550 > I0817 08:51:59.100888 15188 sched.cpp:272] No credentials provided. > Attempting to register without authentication > I0817 08:51:59.326017 15190 sched.cpp:641] Framework registered with > b859f266-9984-482d-8c0d-35bd88c1ad0a-6996 > 16/08/17 08:52:06 WARN ObjectStore: Version information not found in > metastore. hive.metastore.schema.verification is not enabled so recording > the schema version 1.2.0 > 16/08/17 08:52:06 WARN ObjectStore: Failed to get database default, > returning NoSuchObjectException > Traceback (most recent call last): > File "/data1/home/vttrich/spk/orig_qryhubb.py", line 17, in > res=sqlcont.sql("select parti_date FROM log_data WHERE parti_date >= > 408910 limit 10") > File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/context.py", > line 580, in sql > File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", > line 813, in __call__ > File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", > line 51, in deco > pyspark.sql.utils.AnalysisException: u'undefined function > json_array_to_map; line 28 pos 73' > I0817 08:52:12.840224 15600 sched.cpp:1771] Asked to stop the driver > I0817 08:52:12.841198 15189 sched.cpp:1040] Stopping framework > 'b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6326' > > > On Wed, Aug 17, 2016 at 8:50 AM, Ted Yu wrote: > >> Can you show the complete stack trace ? >> >> Which version of Spark are you using ? >> >> Thanks >> >> On Wed, Aug 17, 2016 at 8:46 AM, vr spark wrote: >> >>> Hi, >>> I am getting error on below scenario. Please suggest. >>> >>> i have a virtual view in hive >>> >>> view name log_data >>> it has 2 columns >>> >>> query_map map >>> >>> parti_date int >>> >>> >>> Here is my snippet for the spark data frame >>> >>> my dataframe >>> >>> res=sqlcont.sql("select parti_date FROM log_data WHERE parti_date >= >>> 408910 limit 10") >>> >>> df=res.collect() >>> >>> print 'after collect' >>> >>> print df >>> >>> >>> * File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", >>> line 51, in deco* >>> >>> *pyspark.sql.utils.AnalysisException: u'undefined function >>> json_array_to_map; line 28 pos 73'* >>> >>> >>> >>> >>> >> > --001a114a6d76d829c1053a4849b7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Ted/All,
i did below to get fullstack and see below= , not able to understand root cause..

except = Exception as error:

=C2=A0 =C2=A0 traceback.print_exc()

and this what i get...


=C2=A0File "/usr/local/spark/python/lib= /pyspark.zip/pyspark/sql/context.py", line 580, in sql

=C2=A0 =C2=A0 return DataFrame(self._ssql_ct= x.sql(sqlQuery), self)

=C2=A0 File "/usr/local/spark/python/li= b/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__=

=C2=A0 =C2=A0 answer, self.gateway_client, s= elf.target_id, self.name)

=C2=A0 File "/usr/local/spark/python/li= b/pyspark.zip/pyspark/sql/utils.py", line 51, in deco

=C2=A0 =C2=A0 raise AnalysisException(s.spli= t(': ', 1)[1], stackTrace)

AnalysisException: u'undefined function = json_array_to_map; line 28 pos 73'


On Wed, Aug 17, 2016 at 8= :59 AM, vr spark <vrspark123@gmail.com> wrote:
spark 1.6.1
python
<= br>
I0817 08:51:59.099356 15189 detector.cpp:481] A new lead= ing master (UPID=3Dmaster@10.224.167.25:5050) is detected
I0817 08:5= 1:59.099735 15188 sched.cpp:262] New master detected at master@x.y.17.25:45= 50
I0817 08:51:59.100888 15188 sched.cpp:272] No credentials prov= ided. Attempting to register without authentication
I0817 08:51:5= 9.326017 15190 sched.cpp:641] Framework registered with b859f266-9984-482d-= 8c0d-35bd88c1ad0a-6996
16/08/17 08:52:06 WARN ObjectStore: V= ersion information not found in metastore. hive.metastore.schema.verif= ication is not enabled so recording the schema version 1.2.0
16/0= 8/17 08:52:06 WARN ObjectStore: Failed to get database default, returning N= oSuchObjectException
Traceback (most recent call last):
=C2=A0 File "/data1/home/vttrich/spk/orig_qryhubb.py", line= 17, in <module>
=C2=A0 =C2=A0 res=3Dsqlco= nt.sql("select parti_date FROM log_data WHERE parti_date =C2=A0>=3D= 408910 limit 10")
=C2=A0 File "/usr/local/spark= /python/lib/pyspark.zip/pyspark/sql/context.py", line 580, i= n sql
=C2=A0 File "/usr/local/spark/python/lib/py4j-0.9= -src.zip/py4j/java_gateway.py", line 813, in __call__
=C2=A0 File "/usr/local/spark/python/lib/pyspark.= zip/pyspark/sql/utils.py", line 51, in deco
pyspark.sql= .utils.AnalysisException: u'undefined function json_array_to_map; = line 28 pos 73'
I0817 08:52:12.840224 15600 sched.cpp:= 1771] Asked to stop the driver
I0817 08:52:12.841198 15189 sched.= cpp:1040] Stopping framework 'b859f2f3-7484-482d-8c0d-35bd91c1ad0a= -6326'


On Wed, A= ug 17, 2016 at 8:50 AM, Ted Yu <yuzhihong@gmail.com> wrote= :
Can you show the compl= ete stack trace ?

Which version of Spark are you using ?=

Thanks

On Wed, Aug 17, 2016 at 8:46 AM, vr sp= ark <vrspark123@gmail.com> wrote:
Hi,
I am getting error on below = scenario. Please suggest.=C2=A0

i have =C2=A0a virtual vi= ew in hive

view name log_data
it has 2 c= olumns

query_map =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0map&l= t;string,string>=C2=A0

parti_date =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 int


Here is my snippet for the spark data frame=

my dataframe=C2=A0

res=3Dsqlcont.sql("select=C2=A0parti_<= wbr>date=C2=A0FROM log_data=C2=A0WHERE=C2=A0parti_date=C2=A0 >=3D 408910= =C2=A0limit 10")

df=3Dres.collect()

print '= after collect'

print df


=C2=A0File "/usr/local/spark/python= /lib/pyspark.zip/pyspark/sql/utils.py", line 51, in deco=

pyspark.sql.util= s.AnalysisException: u'undefined function json_array_to_map; line = 28 pos 73'




=



--001a114a6d76d829c1053a4849b7--