spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lonely Feb <lonely8...@gmail.com>
Subject terasort on spark
Date Fri, 16 Jan 2015 10:24:12 GMT
Hi all , i tried to run a terasort benchmark on my spark cluster, but i
found it is hard to find a standard spark terasort program except a PR from
rxin and ewan higgs:

https://github.com/apache/spark/pull/1242
https://github.com/ehiggs/spark/tree/terasort

The example which rxin provided without a validate test so i tried higgs's
example, but i sadly found a always get an error when validate:

assertion failed: current partition min < last partition max

It seems that it requires the min array in partition 2 must bigger than max
array in partion 1, but the code here is confusing:

        println(s"lastMax" + lastMax.toSeq.map(x => if (x < 0) 256 + x else
x))
        println(s"min " + min.toSeq.map(x => if (x < 0) 256 + x else x))
        println(s"max " + max.toSeq.map(x => if (x < 0) 256 + x else x))

Anyone ever run the terasort example successfully? Or where can i get a
standard terasort application?

Mime
View raw message