I have similar requirememt,take top N by key. right now I use groupByKey,but one key would group more than half data in some dataset. 

Yours, Xuefeng Wu 吴雪峰 敬上

On 2014年12月4日, at 上午7:26, Nathan Kronenfeld <nkronenfeld@oculusinfo.com> wrote:

I think it would depend on the type and amount of information you're collecting.

If you're just trying to collect small numbers for each window, and don't have an overwhelming number of windows, you might consider using accumulators.  Just make one per value per time window, and for each data point, add it to the accumulators for the time windows in which it belongs.  We've found this approach a lot faster than anything involving a shuffle.  This should work fine for stuff like max(), min(), and mean()

If you're collecting enough data that accumulators are impractical, I think I would try multiple passes.  Cache your data, and for each pass, filter to that window, and perform all your operations on the filtered RDD.  Because of the caching, it won't be significantly slower than processing it all at once - in fact, it will probably be a lot faster, because the shuffles are shuffling less information.  This is similar to what you're suggesting about partitioning your rdd, but probably simpler and easier.

That being said, your restriction 3 seems to be in contradiction to the rest of your request - if your aggregation needs to be able to look at all the data at once, then that seems contradictory to viewing the data through an RDD.  Could you explain a bit more what you mean by that?


On Wed, Dec 3, 2014 at 4:26 PM, ameyc <ambrood@gmail.com> wrote:

So my Spark app needs to run a sliding window through a time series dataset
(I'm not using Spark streaming). And then run different types on
aggregations on per window basis. Right now I'm using a groupByKey() which
gives me Iterables for each window. There are a few concerns I have with
this approach:

1. groupByKey() could potentially fail for a key not fitting in the memory.
2. I'd like to run aggregations like max(), mean() on each of the groups,
it'd be nice to have the RDD functionality at this point instead of the
3. I can't use reduceByKey() or aggregateByKey() are some of my aggregations
need to have a view of the entire window.

Only other way I could think of is partitioning my RDDs into multiple RDDs
with each RDD representing a window. Is this a sensible approach? Or is
there any other way of going about this?

View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Alternatives-to-groupByKey-tp20293.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Nathan Kronenfeld
Senior Visualization Developer
Oculus Info Inc
2 Berkeley Street, Suite 600,
Toronto, Ontario M5A 4J5
Phone:  +1-416-203-3003 x 238
Email:  nkronenfeld@oculusinfo.com