Hi Ted/All,
i did below to get fullstack and see below, not able to understand root cause..

except Exception as error:

    traceback.print_exc()

and this what i get...


 File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/context.py", line 580, in sql

    return DataFrame(self._ssql_ctx.sql(sqlQuery), self)

  File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__

    answer, self.gateway_client, self.target_id, self.name)

  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 51, in deco

    raise AnalysisException(s.split(': ', 1)[1], stackTrace)

AnalysisException: u'undefined function json_array_to_map; line 28 pos 73'


On Wed, Aug 17, 2016 at 8:59 AM, vr spark <vrspark123@gmail.com> wrote:
spark 1.6.1
python

I0817 08:51:59.099356 15189 detector.cpp:481] A new leading master (UPID=master@10.224.167.25:5050) is detected
I0817 08:51:59.099735 15188 sched.cpp:262] New master detected at master@x.y.17.25:4550
I0817 08:51:59.100888 15188 sched.cpp:272] No credentials provided. Attempting to register without authentication
I0817 08:51:59.326017 15190 sched.cpp:641] Framework registered with b859f266-9984-482d-8c0d-35bd88c1ad0a-6996
16/08/17 08:52:06 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/08/17 08:52:06 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
Traceback (most recent call last):
  File "/data1/home/vttrich/spk/orig_qryhubb.py", line 17, in <module>
    res=sqlcont.sql("select parti_date FROM log_data WHERE parti_date  >= 408910 limit 10")
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/context.py", line 580, in sql
  File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
  File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 51, in deco
pyspark.sql.utils.AnalysisException: u'undefined function json_array_to_map; line 28 pos 73'
I0817 08:52:12.840224 15600 sched.cpp:1771] Asked to stop the driver
I0817 08:52:12.841198 15189 sched.cpp:1040] Stopping framework 'b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6326'


On Wed, Aug 17, 2016 at 8:50 AM, Ted Yu <yuzhihong@gmail.com> wrote:
Can you show the complete stack trace ?

Which version of Spark are you using ?

Thanks

On Wed, Aug 17, 2016 at 8:46 AM, vr spark <vrspark123@gmail.com> wrote:
Hi,
I am getting error on below scenario. Please suggest. 

i have  a virtual view in hive

view name log_data
it has 2 columns

query_map           map<string,string> 

parti_date             int


Here is my snippet for the spark data frame

my dataframe 

res=sqlcont.sql("select parti_date FROM log_data WHERE parti_date  >= 408910 limit 10")

df=res.collect()

print 'after collect'

print df


 File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 51, in deco

pyspark.sql.utils.AnalysisException: u'undefined function json_array_to_map; line 28 pos 73'