spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nitinkak001 <nitinkak...@gmail.com>
Subject Re: Window comparison matching using the sliding window functionality: feasibility
Date Mon, 02 Feb 2015 15:26:23 GMT
Mine was not really a moving average problem. It was more like partitioning
on some keys and sorting(on different keys) and then running a sliding
window through the partition. I reverted back to map-reduce for that(I
needed secondary sort, which is not very mature in Spark right now).

But, as far as I understand your problem, you should be able to handle it
by converting your RDD to key-value RDDs which I think will be
automatically partitioned on the key and then use *mapPartitions *to run
your logic.

On Mon, Feb 2, 2015 at 1:20 AM, ashu [via Apache Spark User List] <
ml-node+s1001560n21458h90@n3.nabble.com> wrote:

> Hi,
> I want to know about your moving avg implementation. I am also doing some
> time-series analysis about CPU performance. So I tried simple regression
> but result is not good. rmse is 10 but when I extrapolate it just shoot up
> linearly. I think I should first smoothed out the data then try regression
> to forecast.
> i am thinking of moving avg as an option,tried it out according to this
> http://stackoverflow.com/questions/23402303/apache-spark-moving-average
>
> but "partitionBy" is giving me error, I am building with Spark 1.2.0.
> Can you share your ARIMA implementation if it is open source, else can you
> give me hints about it
>
> Will really appreciate the help.
> Thanks
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Window-comparison-matching-using-the-sliding-window-functionality-feasibility-tp15352p21458.html
>  To unsubscribe from Window comparison matching using the sliding window
> functionality: feasibility, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=15352&code=bml0aW5rYWswMDFAZ21haWwuY29tfDE1MzUyfDEyMjcwMjA2NQ==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Window-comparison-matching-using-the-sliding-window-functionality-feasibility-tp15352p21467.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message