spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From buntu <buntu...@gmail.com>
Subject Re: spark-shell -- running into ArrayIndexOutOfBoundsException
Date Wed, 23 Jul 2014 17:29:33 GMT
Just wanted to add more info.. I was using SparkSQL reading in the
tab-delimited raw data files converting the timestamp to Date format:

  sc.textFile("rawdata/*").map(_.split("\t")).map(p => Point(df.format(new
Date( p(0).trim.toLong*1000L )), p(1), p(2).trim.toInt ,p(3).trim.toInt,
p(4).trim.toInt ,p(5)))

Then I go about registering it as table and when I run simple query like
select count(*) from 
, I get the ArrayIndexOutOfBoundsException.

I bumped up the SPARK_DRIVER_MEMORY to 8g but still didn't help: 
  export SPARK_DRIVER_MEMORY=8g

Let me know if I'm missing any steps.. thanks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-running-into-ArrayIndexOutOfBoundsException-tp10480p10520.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message