spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "" <>
Subject Re: PermGen Space Error
Date Wed, 29 Jul 2015 09:19:54 GMT
Hi, Sarath

Did you try to use and increase spark.excecutor.extraJaveOptions -XX:PermSize= -XX:MaxPermSize=
From: Sarath Chandra
Date: 2015-07-29 17:39
Subject: PermGen Space Error
Dear All,

I'm using - 
 => Spark 1.2.0
 => Hive 0.13.1
 => Mesos 0.18.1
 => Spring
 => JDK 1.7

I've written a scala program which 
  => instantiates a spark and hive context
  => parses an XML file which provides the where clauses for queries
  => generates full fledged hive queries to be run on hive tables
  => registers obtained SchemaRDD as temp tables to get reduced data sets to be queried
  => prints the count of finally obtained data set

I'm running this scala programatically through java command (command invokes a controller
program to create some useful value objects using input parameters and properties files and
then calls the above scala program).

I'm getting PermGen Space error when it hits the last line to print the count.

I'm printing to console the generated hive queries from the scala program. When I run the
same from a spark shell it works fine.

As mentioned in some posts and blogs I tried using the option spark.driver.extraJavaOptions
to increase the size, tried with 256 and 512 but still no luck.

Please help me in resolving the space issue

Thanks & Regards,
View raw message