spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Xiao <xiaotao.cs....@gmail.com>
Subject How to provide a custom Comparator to sortByKey?
Date Sat, 01 Mar 2014 02:19:06 GMT
I am using Spark 0.9
I have an array of tuples, and I want to sort these tuples using the *sortByKey
*API as follows in Spark shell:

val A:Array[(String, String)] = Array(("1", "One"), ("9", "Nine"), ("3",
"three"), ("5", "five"), ("4", "four"))
val P = sc.parallelize(A)

// MyComparator is an example, maybe I have more complex implementation
class MyComparator extends java.util.Comparator[String] {
def compare(s1:String, s2:String):Int = {
s1.compareTo(s2)
}
}

val comp = new MyComparator()
P.sortByKey(comp, true)


When I invoked P.sortByKey(comp, true),  spark shell complained that there
was a type mismatch, to be specific, *sortByKey *requires *Boolean *but I
provided a *Comparator*.

How should I provide my custom comparator to *sortByKey *?

Mime
View raw message