I'm using -
=> Spark 1.2.0
=> Hive 0.13.1
=> Mesos 0.18.1
=> JDK 1.7
I've written a scala program which
=> instantiates a spark and hive context
=> parses an XML file which provides the where clauses for queries
=> generates full fledged hive queries to be run on hive tables
=> registers obtained SchemaRDD as temp tables to get reduced data sets to be queried further
=> prints the count of finally obtained data set
I'm running this scala programatically through java command (command invokes a controller program to create some useful value objects using input parameters and properties files and then calls the above scala program).
I'm getting PermGen Space error when it hits the last line to print the count.
I'm printing to console the generated hive queries from the scala program. When I run the same from a spark shell it works fine.
As mentioned in some posts and blogs I tried using the option spark.driver.extraJavaOptions to increase the size, tried with 256 and 512 but still no luck.
Please help me in resolving the space issue
Thanks & Regards,