spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hongbin Liu <Hongbin....@theice.com>
Subject RE: stackoverflow error
Date Tue, 04 Nov 2014 23:17:42 GMT
Sorry, I have to change rank/lambda/iteration to the following

    val ranks = List(100, 200, 400)
    val lambdas = List(1.0, 2.0, 4.0)
    val numIters = List(50, 100, 150)


From: Hongbin Liu
Sent: Tuesday, November 04, 2014 6:14 PM
To: 'user@spark.apache.org'
Cc: Gregory Campbell
Subject: stackoverflow error

Hi, can you help with the following? We are new to spark.

Error stack:

14/11/04 18:08:03 INFO SparkContext: Job finished: count at ALS.scala:314, took 480.318100288
s
Exception in thread "main" java.lang.StackOverflowError
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.rdd.RDD.doCheckpoint(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.rdd.RDD.doCheckpoint(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.rdd.RDD.doCheckpoint(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.rdd.RDD.doCheckpoint(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.rdd.RDD.doCheckpoint(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.rdd.RDD.doCheckpoint(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.rdd.RDD.doCheckpoint(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.rdd.RDD.doCheckpoint(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at org.apache.spark.rdd.RDD$$anonfun$doCheckpoint$1.apply(RDD.scala:1270)
        at scala.collection.immutable.List.foreach(List.scala:318)
....

The code that generates the error I think is

    val ranks = List(100, 120)
    val lambdas = List(1.0, 10.0)
    val numIters = List(10, 20)
    var bestModel: Option[MatrixFactorizationModel] = None
    var bestValidationRmse = Double.MaxValue
    var bestRank = 0
    var bestLambda = -1.0
    var bestNumIter = -1
    for (rank <- ranks; lambda <- lambdas; numIter <- numIters) {
        val model = ALS.train(training, rank, numIter, lambda)
        val validationRmse = computeRmse(model, validation, numValidation)
        appendToFile("rank-lamda-it-rmse.txt", rank + "," + lamda + "," + numIter + ", " +
validationRmse)
        if (validationRmse < bestValidationRmse) {
            bestModel = Some(model)
            bestValidationRmse = validationRmse
            bestRank = rank
            bestLambda = lambda
            bestNumIter = numIter
        }
    }

________________________________
This message may contain confidential information and is intended for specific recipients
unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient
of this message, please delete it and notify the sender. This message may not represent the
opinion of Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, and does
not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the
recipient of this message is expected to provide safeguards from viruses and pursue alternate
means of communication where privacy or a binding message is desired.

Mime
View raw message