spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Zaitlen <quasi...@gmail.com>
Subject Re: 1.5 Build Errors
Date Tue, 08 Sep 2015 17:05:03 GMT
I'm running zinv while compiling.  It seems that MAVEN_OPTS doesn't really
change much?  Or perhaps I'm misunderstanding something -- grepping for
java i see

root     24355  102  8.8 4687376 1350724 pts/4 Sl   16:51  11:08
> /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server -Xmx2g
> -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m
> -Dzinc.home=/root/spark/build/zinc-0.3.5.3 -classpath
> /root/spark/build/zinc-0.3.5.3/lib/compiler-interface-sources.jar:/root/spark/build/zinc-0.3.5.3/lib/incremental-compiler.jar:/root/spark/build/zinc-0.3.5.3/lib/nailgun-server.jar:/root/spark/build/zinc-0.3.5.3/lib/sbt-interface.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-compiler.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-library.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-reflect.jar:/root/spark/build/zinc-0.3.5.3/lib/zinc.jar
> com.typesafe.zinc.Nailgun 3030 0
> root     25151 22.0  3.2 2269092 495276 pts/4  Sl+  16:53   1:56
> /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms256m -Xmx512m -classpath
> /opt/anaconda/envs/spark_build/share/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar
> -Dclassworlds.conf=/opt/anaconda/envs/spark_build/share/apache-maven-3.3.3/bin/m2.conf
> -Dmaven.home=/opt/anaconda/envs/spark_build/share/apache-maven-3.3.3
> -Dmaven.multiModuleProjectDirectory=/root/spark
> org.codehaus.plexus.classworlds.launcher.Launcher -DzincPort=3030 clean
> package -DskipTests -Pyarn -Phive -Phive-thriftserver -Phadoop-2.4
> -Dhadoop.version=2.4.0


So the heap size is still 2g even with MAVEN_OPTS set with 4g.  I noticed
that within build/mvn _COMPILE_JVM_OPTS is set to 2g and this is what
ZINC_OPTS is set to.

--Ben


On Tue, Sep 8, 2015 at 11:06 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Do you run Zinc while compiling ?
>
> Cheers
>
> On Tue, Sep 8, 2015 at 7:56 AM, Benjamin Zaitlen <quasiben@gmail.com>
> wrote:
>
>> I'm still getting errors with 3g.  I've increase to 4g and I'll report
>> back
>>
>> To be clear:
>>
>> export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M
>> -XX:ReservedCodeCacheSize=1024m"
>>
>> [ERROR] GC overhead limit exceeded -> [Help 1]
>>> [ERROR]
>>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>>> -e switch.
>>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>>> [ERROR]
>>> [ERROR] For more information about the errors and possible solutions,
>>> please read the following articles:
>>> [ERROR] [Help 1]
>>> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
>>> + return 1
>>> + exit 1
>>
>>
>> On Tue, Sep 8, 2015 at 10:03 AM, Sean Owen <sowen@cloudera.com> wrote:
>>
>>> It might need more memory in certain situations / running certain
>>> tests. If 3gb works for your relatively full build, yes you can open a
>>> PR to change any occurrences of lower recommendations to 3gb.
>>>
>>> On Tue, Sep 8, 2015 at 3:02 PM, Benjamin Zaitlen <quasiben@gmail.com>
>>> wrote:
>>> > Ah, right.  Should've caught that.
>>> >
>>> > The docs seem to recommend 2gb.  Should that be increased as well?
>>> >
>>> > --Ben
>>> >
>>> > On Tue, Sep 8, 2015 at 9:33 AM, Sean Owen <sowen@cloudera.com> wrote:
>>> >>
>>> >> It shows you there that Maven is out of memory. Give it more heap. I
>>> use
>>> >> 3gb.
>>> >>
>>> >> On Tue, Sep 8, 2015 at 1:53 PM, Benjamin Zaitlen <quasiben@gmail.com>
>>> >> wrote:
>>> >> > Hi All,
>>> >> >
>>> >> > I'm trying to build a distribution off of the latest in master
and I
>>> >> > keep
>>> >> > getting errors on MQTT and the build fails.   I'm running the build
>>> on a
>>> >> > m1.large which has 7.5 GB of RAM and no other major processes are
>>> >> > running.
>>> >> >
>>> >> >> MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M
>>> -XX:ReservedCodeCacheSize=512m"
>>> >> >> ./make-distribution.sh  --name continuum-custom-spark-1.5 --tgz
>>> -Pyarn
>>> >> >> -Phive -Phive-thriftserver -Phadoop-2.4 -Dhadoop.version=2.4.0
>>> >> >
>>> >> >
>>> >> >
>>> >> >> INFO] Spark Project GraphX ...............................
SUCCESS
>>> [
>>> >> >> 33.345 s]
>>> >> >> [INFO] Spark Project Streaming ............................
SUCCESS
>>> >> >> [01:08
>>> >> >> min]
>>> >> >> [INFO] Spark Project Catalyst .............................
SUCCESS
>>> >> >> [01:39
>>> >> >> min]
>>> >> >> [INFO] Spark Project SQL ..................................
SUCCESS
>>> >> >> [02:06
>>> >> >> min]
>>> >> >> [INFO] Spark Project ML Library ...........................
SUCCESS
>>> >> >> [02:16
>>> >> >> min]
>>> >> >> [INFO] Spark Project Tools ................................
>>> SUCCESS [
>>> >> >> 4.087 s]
>>> >> >> [INFO] Spark Project Hive .................................
SUCCESS
>>> >> >> [01:28
>>> >> >> min]
>>> >> >> [INFO] Spark Project REPL .................................
>>> SUCCESS [
>>> >> >> 16.291 s]
>>> >> >> [INFO] Spark Project YARN Shuffle Service .................
>>> SUCCESS [
>>> >> >> 13.671 s]
>>> >> >> [INFO] Spark Project YARN .................................
>>> SUCCESS [
>>> >> >> 20.554 s]
>>> >> >> [INFO] Spark Project Hive Thrift Server ...................
>>> SUCCESS [
>>> >> >> 14.332 s]
>>> >> >> [INFO] Spark Project Assembly .............................
SUCCESS
>>> >> >> [03:33
>>> >> >> min]
>>> >> >> [INFO] Spark Project External Twitter .....................
>>> SUCCESS [
>>> >> >> 14.208 s]
>>> >> >> [INFO] Spark Project External Flume Sink ..................
>>> SUCCESS [
>>> >> >> 11.535 s]
>>> >> >> [INFO] Spark Project External Flume .......................
>>> SUCCESS [
>>> >> >> 19.010 s]
>>> >> >> [INFO] Spark Project External Flume Assembly ..............
>>> SUCCESS [
>>> >> >> 5.210 s]
>>> >> >> [INFO] Spark Project External MQTT ........................
FAILURE
>>> >> >> [01:10
>>> >> >> min]
>>> >> >> [INFO] Spark Project External MQTT Assembly ...............
SKIPPED
>>> >> >> [INFO] Spark Project External ZeroMQ ......................
SKIPPED
>>> >> >> [INFO] Spark Project External Kafka .......................
SKIPPED
>>> >> >> [INFO] Spark Project Examples .............................
SKIPPED
>>> >> >> [INFO] Spark Project External Kafka Assembly ..............
SKIPPED
>>> >> >> [INFO]
>>> >> >>
>>> >> >>
>>> ------------------------------------------------------------------------
>>> >> >> [INFO] BUILD FAILURE
>>> >> >> [INFO]
>>> >> >>
>>> >> >>
>>> ------------------------------------------------------------------------
>>> >> >> [INFO] Total time: 22:55 min
>>> >> >> [INFO] Finished at: 2015-09-07T22:42:57+00:00
>>> >> >> [INFO] Final Memory: 240M/455M
>>> >> >> [INFO]
>>> >> >>
>>> >> >>
>>> ------------------------------------------------------------------------
>>> >> >> [ERROR] GC overhead limit exceeded -> [Help 1]
>>> >> >> [ERROR]
>>> >> >> [ERROR] To see the full stack trace of the errors, re-run Maven
>>> with
>>> >> >> the
>>> >> >> -e switch.
>>> >> >> [ERROR] Re-run Maven using the -X switch to enable full debug
>>> logging.
>>> >> >> [ERROR]
>>> >> >> [ERROR] For more information about the errors and possible
>>> solutions,
>>> >> >> please read the following articles:
>>> >> >> [ERROR] [Help 1]
>>> >> >> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
>>> >> >> + return 1
>>> >> >> + exit 1
>>> >> >
>>> >> >
>>> >> > Any thoughts would be extremely helpful.
>>> >> >
>>> >> > --Ben
>>> >
>>> >
>>>
>>
>>
>

Mime
View raw message