spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Perrigo <>
Subject Help with Initial Cluster Configuration / Tuning
Date Mon, 21 Oct 2013 20:05:38 GMT
Hi everyone,
I am very new to Spark, so as a learning exercise I've set up a small
cluster consisting of 4 EC2 m1.large instances (1 master, 3 slaves), which
I'm hoping to use to calculate ngram frequencies from text files of various
sizes (I'm not doing anything with them; I just thought this would be
slightly more interesting than the usual 'word count' example).  Currently,
I'm trying to work with a 1GB text file, but running into memory issues.
 I'm wondering what parameters I should be setting (in in
order to properly utilize the cluster.  Right now, I'd be happy just to
have the process complete successfully with the 1 gig file, so I'd really
appreciate any suggestions you all might have.

Here's a summary of the code I'm running through the spark shell on the

def ngrams(s: String, n: Int = 3): Seq[String] = {
  (s.split("\\s+").sliding(n)).filter(_.length == n).map(_.mkString("

val text = sc.textFile("s3n://my-bucket/my-1gb-text-file")

val mapped = text.filter(_.trim.length > 0).flatMap(ngrams(_, 3))

So far so good; the problems come during the reduce phase.  With small
files, I was able to issue the following to calculate the most frequently
occurring trigram:

val topNgram = (mapped countByValue) reduce((a:(String, Long), b:(String,
Long)) => if (a._2 > b._2) a else b)

With the 1 gig file, though, I've been running into OutOfMemory errors, so
I decided to split the reduction to several steps, starting with simply
issuing countByValue of my "mapped" RDD, but I have yet to get it to
complete successfully.

SPARK_MEM is currently set to 6154m.  I also bumped up the
spark.akka.framesize setting to 500 (though at this point, I was grasping
at straws; I'm not sure what a "proper" value would be).  What properties
should I be setting for a job of this size on a cluster of 3 m1.large
slaves? (The cluster was initially configured using the spark-ec2 scripts).
 Also, programmatically, what should I be doing differently?  (For example,
should I be setting the minimum number of splits when reading the text
file?  If so, what would be a good default?).

I apologize for what I'm sure are very naive questions.  I think Spark is a
fantastic project and have enjoyed working with it, but I'm still very much
a newbie and would appreciate any help you all can provide (as well as any
'rules-of-thumb' or best practices I should be following).

Tim Perrigo

View raw message