spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Divya Gehlot <divya.htco...@gmail.com>
Subject Re: Window Functions with SQLContext
Date Thu, 01 Sep 2016 06:02:42 GMT
Hi Saurabh,

Even I am using Spark 1.6+ version ..and when I didnt create hiveContext it
threw the same error .
So have to  create HiveContext to access windows function

Thanks,
Divya

On 1 September 2016 at 13:16, saurabh3d <saurabh.s.dubey@oracle.com> wrote:

> Hi All,
>
> As per  SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001>
>  ,
> Window functions should be supported by SQLContext. But when i try to run
>
> SQLContext sqlContext = new SQLContext(jsc);
> WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
> DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
> df_2.show(false);
>
> it fails with:
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Could
> not
> resolve window function 'row_number'. Note that, using window functions
> currently requires a HiveContext;
>
> This code runs fine with HiveContext.
> Any idea what’s going on?  Is this a known issue and is there a workaround
> to make Window function work without HiveContext.
>
> Thanks,
> Saurabh
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message