spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Lewis <lordjoe2...@gmail.com>
Subject Reproducing the function of a Hadoop Reducer
Date Fri, 19 Sep 2014 20:37:56 GMT
 I am struggling to reproduce the functionality of a Hadoop reducer on
Spark (in Java)

in Hadoop I have a function
public void doReduce(K key, Iterator<V> values)
in Hadoop there is also a consumer (context write) which can be seen as
consume(key,value)

In my code
1) knowing the key is important to the function
2) there is neither one output tuple2 per key nor one output tuple2 per
value
3) the number of values per key might be large enough that storing them in
memory is impractical
4) keys must appear in sorted order

one good example would run through a large document using a similarity
function to look at the last 200 lines and output any of those with a
similarity of more than 0.3 (do not suggest output all and filter - the
real problem is more complex) the critical concern is an uncertain number
of tuples per key.

my questions
1) how can this be done - ideally a consumer would be a JavaPairRDD but I
don't see how to create one and add items later

2) how do I handle the entire partition, sort, process (involving calls to
doReduce) process

Mime
View raw message