spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Tanase <atan...@adobe.com>
Subject Re: "dynamically" sort a large collection?
Date Mon, 12 Oct 2015 10:19:29 GMT
I think you’re looking for the flatMap (or flatMapValues) operator – you can do something
like

sortedRdd.flatMapValues( v =>
If (v % 2 == 0) {
Some(v / 2)
} else {
None
}
)

Then you need to sort again.

-adrian

From: Yifan LI
Date: Monday, October 12, 2015 at 1:03 PM
To: spark users
Subject: "dynamically" sort a large collection?

Hey,

I need to scan a large "key-value" collection as below:

1) sort it on an attribute of “value”
2) scan it one by one, from element with largest value
2.1) if the current element matches a pre-defined condition, its value will be reduced and
the element will be inserted back to collection.
if not, this current element should be removed from collection.


In my previous program, the 1) step can be easily conducted in Spark(RDD operation), but I
am not sure how to do 2.1) step, esp. the “put/inserted back” operation on a sorted RDD.
I have tried to make a new RDD at every-time an element was found to inserted, but it is very
costly due to a re-sorting…


Is there anyone having some ideas?

Thanks so much!

******************
an example:

the sorted result of initial collection C(on bold value), sortedC:
(1, (71, “aaa"))
(2, (60, “bbb"))
(3, (53.5, “ccc”))
(4, (48, “ddd”))
(5, (29, “eee"))
…

pre-condition: its_value%2 == 0
if pre-condition is matched, its value will be reduce on half.

Thus:

#1:
71 is not matched, so this element is removed.
(1, (71, “aaa”)) —> removed!
(2, (60, “bbb"))
(3, (53.5, “ccc”))
(4, (48, “ddd”))
(5, (29, “eee"))
…

#2:
60 is matched! 60/2 = 30, the collection right now should be as:
(3, (53.5, “ccc”))
(4, (48, “ddd”))
(2, (30, “bbb”)) <— inserted back here
(5, (29, “eee"))
…






Best,
Yifan LI





Mime
View raw message