spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nitinkak001 <nitinkak...@gmail.com>
Subject Re: Window comparison matching using the sliding window functionality: feasibility
Date Sat, 11 Oct 2014 02:42:02 GMT
Thanks @category_theory, the post was of great help!!

I had to learn a few thing before I could understand it completely.
However, I am facing the issue of partitioning the data (using partitionBy)
without providing a hardcoded value for number of partitions. The
partitions need to be driven by data(segmentation key I am using) in my
case.

So my question is say if

the number of partitions generated by my segmentation key = 1000
the number given to the partitioner = 2000

In this case, would there be 2000 partitions created(which will break the
partition boundary of the segmentation key)? If so then sliding window will
roll over multiple partitions and computation would generate wrong results.

Thanks again for the response!!

On Tue, Sep 30, 2014 at 11:51 AM, category_theory [via Apache Spark User
List] <ml-node+s1001560n15407h9@n3.nabble.com> wrote:

> Not sure if this is what you are after but its based on a moving average
> within spark...  I was building an ARIMA model on top of spark and this
> helped me out a lot:
>
> http://stackoverflow.com/questions/23402303/apache-spark-moving-average
> ᐧ
>
>
>
>
> *JIMMY MCERLAIN*
>
> DATA SCIENTIST (NERD)
>
> *. . . . . . . . . . . . . . . . . .*
>
>
> *IF WE CAN’T DOUBLE YOUR SALES,*
>
>
>
> *ONE OF US IS IN THE WRONG BUSINESS.*
>
> *E*: [hidden email] <http://user/SendEmail.jtp?type=node&node=15407&i=0>
>
>
> *M*: *510.303.7751 <510.303.7751>*
>
> On Tue, Sep 30, 2014 at 8:19 AM, nitinkak001 <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=15407&i=1>> wrote:
>
>> Any ideas guys?
>>
>> Trying to find some information online. Not much luck so far.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Window-comparison-matching-using-the-sliding-window-functionality-feasibility-tp15352p15404.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=15407&i=2>
>> For additional commands, e-mail: [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=15407&i=3>
>>
>>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Window-comparison-matching-using-the-sliding-window-functionality-feasibility-tp15352p15407.html
>  To unsubscribe from Window comparison matching using the sliding window
> functionality: feasibility, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=15352&code=bml0aW5rYWswMDFAZ21haWwuY29tfDE1MzUyfDEyMjcwMjA2NQ==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Window-comparison-matching-using-the-sliding-window-functionality-feasibility-tp15352p16201.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message