spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruslan Dautkhanov <dautkha...@gmail.com>
Subject Re: Does spark supports the Hive function posexplode function?
Date Sun, 12 Jul 2015 18:42:02 GMT
You can see what Spark SQL functions are supported in Spark by doing the
following in a notebook:
%sql show functions

https://forums.databricks.com/questions/665/is-hive-coalesce-function-supported-in-sparksql.html

I think Spark SQL support is currently around Hive ~0.11?



-- 
Ruslan Dautkhanov

On Tue, Jul 7, 2015 at 3:10 PM, Jeff J Li <liji@us.ibm.com> wrote:

> I am trying to use the posexplode function in the HiveContext to
> auto-generate a sequence number. This feature is supposed to be available
> Hive 0.13.0.
>
> SELECT name, phone FROM contact LATERAL VIEW
> posexplode(phoneList.phoneNumber) phoneTable AS pos, phone
>
> My test program failed with the following
>
>         java.lang.ClassNotFoundException: posexplode
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:665)
>         at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:942)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:851)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:827)
>         at
> org.apache.spark.sql.hive.HiveFunctionWrapper.createFunction(Shim13.scala:147)
>         at
> org.apache.spark.sql.hive.HiveGenericUdtf.function$lzycompute(hiveUdfs.scala:274)
>         at
> org.apache.spark.sql.hive.HiveGenericUdtf.function(hiveUdfs.scala:274)
>
> Does spark support this Hive function posexplode? If not, how to patch it
> to support this? I am on Spark 1.3.1
>
> Thanks,
> Jeff Li
>
>
>

Mime
View raw message