spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sri <kali.tumm...@gmail.com>
Subject Re: sql to spark scala rdd
Date Mon, 01 Aug 2016 06:04:54 GMT
Hi ,

Just wondering how spark SQL works behind the scenes does it not convert SQL to some Scala
RDD ? Or Scala ?

How to write below SQL in Scala or Scala RDD

>>>>>>>> SELECT DATE,balance,
>>>>>>>> SUM(balance) OVER (ORDER BY DATE ROWS BETWEEN UNBOUNDED PRECEDING
>>>>>>>> AND
>>>>>>>> CURRENT ROW) daily_balance
>>>>>>>> FROM  table

Thanks
Sri
Sent from my iPhone

> On 31 Jul 2016, at 13:21, Jacek Laskowski <jacek@japila.pl> wrote:
> 
> Hi,
> 
> Impossible - see
> http://www.scala-lang.org/api/current/index.html#scala.collection.Seq@sliding(size:Int,step:Int):Iterator[Repr].
> 
> I tried to show you why you ended up with "non-empty iterator" after
> println. You should really start with
> http://www.scala-lang.org/documentation/
> 
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
> 
> 
> On Sun, Jul 31, 2016 at 8:49 PM, sri hari kali charan Tummala
> <kali.tummala@gmail.com> wrote:
>> Tuple
>> 
>> [Lscala.Tuple2;@65e4cb84
>> 
>>> On Sun, Jul 31, 2016 at 1:00 AM, Jacek Laskowski <jacek@japila.pl> wrote:
>>> 
>>> Hi,
>>> 
>>> What's the result type of sliding(2,1)?
>>> 
>>> Pozdrawiam,
>>> Jacek Laskowski
>>> ----
>>> https://medium.com/@jaceklaskowski/
>>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>>> Follow me at https://twitter.com/jaceklaskowski
>>> 
>>> 
>>> On Sun, Jul 31, 2016 at 9:23 AM, sri hari kali charan Tummala
>>> <kali.tummala@gmail.com> wrote:
>>>> tried this no luck, wht is non-empty iterator here ?
>>>> 
>>>> OP:-
>>>> (-987,non-empty iterator)
>>>> (-987,non-empty iterator)
>>>> (-987,non-empty iterator)
>>>> (-987,non-empty iterator)
>>>> (-987,non-empty iterator)
>>>> 
>>>> 
>>>> sc.textFile(file).keyBy(x => x.split("\\~") (0))
>>>>  .map(x => x._2.split("\\~"))
>>>>  .map(x => (x(0),x(2)))
>>>>    .map { case (key,value) =>
>>>> (key,value.toArray.toSeq.sliding(2,1).map(x
>>>> => x.sum/x.size))}.foreach(println)
>>>> 
>>>> 
>>>> On Sun, Jul 31, 2016 at 12:03 AM, sri hari kali charan Tummala
>>>> <kali.tummala@gmail.com> wrote:
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> I managed to write using sliding function but can it get key as well
in
>>>>> my
>>>>> output ?
>>>>> 
>>>>> sc.textFile(file).keyBy(x => x.split("\\~") (0))
>>>>>      .map(x => x._2.split("\\~"))
>>>>>      .map(x => (x(2).toDouble)).toArray().sliding(2,1).map(x =>
>>>>> (x,x.size)).foreach(println)
>>>>> 
>>>>> 
>>>>> at the moment my output:-
>>>>> 
>>>>> 75.0
>>>>> -25.0
>>>>> 50.0
>>>>> -50.0
>>>>> -100.0
>>>>> 
>>>>> I want with key how to get moving average output based on key ?
>>>>> 
>>>>> 
>>>>> 987,75.0
>>>>> 987,-25
>>>>> 987,50.0
>>>>> 
>>>>> Thanks
>>>>> Sri
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Sat, Jul 30, 2016 at 11:40 AM, sri hari kali charan Tummala
>>>>> <kali.tummala@gmail.com> wrote:
>>>>>> 
>>>>>> for knowledge just wondering how to write it up in scala or spark
RDD.
>>>>>> 
>>>>>> Thanks
>>>>>> Sri
>>>>>> 
>>>>>> On Sat, Jul 30, 2016 at 11:24 AM, Jacek Laskowski <jacek@japila.pl>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Why?
>>>>>>> 
>>>>>>> Pozdrawiam,
>>>>>>> Jacek Laskowski
>>>>>>> ----
>>>>>>> https://medium.com/@jaceklaskowski/
>>>>>>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>>>>>>> Follow me at https://twitter.com/jaceklaskowski
>>>>>>> 
>>>>>>> 
>>>>>>> On Sat, Jul 30, 2016 at 4:42 AM, Kali.tummala@gmail.com
>>>>>>> <Kali.tummala@gmail.com> wrote:
>>>>>>>> Hi All,
>>>>>>>> 
>>>>>>>> I managed to write business requirement in spark-sql and
hive I am
>>>>>>>> still
>>>>>>>> learning scala how this below sql be written using spark
RDD not
>>>>>>>> spark
>>>>>>>> data
>>>>>>>> frames.
>>>>>>>> 
>>>>>>>> SELECT DATE,balance,
>>>>>>>> SUM(balance) OVER (ORDER BY DATE ROWS BETWEEN UNBOUNDED PRECEDING
>>>>>>>> AND
>>>>>>>> CURRENT ROW) daily_balance
>>>>>>>> FROM  table
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> 
>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/sql-to-spark-scala-rdd-tp27433.html
>>>>>>>> Sent from the Apache Spark User List mailing list archive
at
>>>>>>>> Nabble.com.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Thanks & Regards
>>>>>> Sri Tummala
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Thanks & Regards
>>>>> Sri Tummala
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Thanks & Regards
>>>> Sri Tummala
>> 
>> 
>> 
>> 
>> --
>> Thanks & Regards
>> Sri Tummala
>> 

Mime
View raw message