spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nitinkak001 <>
Subject Re: Window comparison matching using the sliding window functionality: feasibility
Date Mon, 02 Feb 2015 15:26:23 GMT
Mine was not really a moving average problem. It was more like partitioning
on some keys and sorting(on different keys) and then running a sliding
window through the partition. I reverted back to map-reduce for that(I
needed secondary sort, which is not very mature in Spark right now).

But, as far as I understand your problem, you should be able to handle it
by converting your RDD to key-value RDDs which I think will be
automatically partitioned on the key and then use *mapPartitions *to run
your logic.

On Mon, Feb 2, 2015 at 1:20 AM, ashu [via Apache Spark User List] <> wrote:

> Hi,
> I want to know about your moving avg implementation. I am also doing some
> time-series analysis about CPU performance. So I tried simple regression
> but result is not good. rmse is 10 but when I extrapolate it just shoot up
> linearly. I think I should first smoothed out the data then try regression
> to forecast.
> i am thinking of moving avg as an option,tried it out according to this
> but "partitionBy" is giving me error, I am building with Spark 1.2.0.
> Can you share your ARIMA implementation if it is open source, else can you
> give me hints about it
> Will really appreciate the help.
> Thanks
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>  To unsubscribe from Window comparison matching using the sliding window
> functionality: feasibility, click here
> <>
> .
> <>

View this message in context:
Sent from the Apache Spark User List mailing list archive at
View raw message