spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fred Reiss <freiss....@gmail.com>
Subject Re: building Spark 2.1 vs Java 1.8 on Ubuntu 16/06
Date Thu, 06 Oct 2016 19:58:48 GMT
There's no option to prevent build/mvn from starting the zinc server, but
you should be able to prevent the maven build from using the zinc server by
changing the <useZincServer> option at line 1935 of the master pom.xml.

Note that the zinc-based compile works on my Ubuntu 16.04 box. You might be
able to get zinc-based compiles working by tweaking your settings. A few
things to try:
-- Make sure another build hasn't left a second, incompatible copy of zinc
squatting on the port that Spark expects to use
-- Try setting the environment variable JAVA_7_HOME to point to and OpenJDK
7 installation. build/mvn runs zinc with Java 7 if that is available.

Note that setting JAVA_7_HOME will break incremental compilation for
sbt-based builds. Use that environment variable with restraint.

Fred



On Thu, Oct 6, 2016 at 2:22 AM, Marco Mistroni <mmistroni@gmail.com> wrote:

> Thanks Fred
> The build/mvn will trigger compilation using zinc and I want to avoid that
> as every time I have tried it runs into errors while compiling spark core.
> How can I disable zinc by default?
> Kr
>
> On 5 Oct 2016 10:53 pm, "Fred Reiss" <freiss.oss@gmail.com> wrote:
>
>> Actually the memory options *are* required for Java 1.8. Without them the
>> build will fail intermittently. We just updated the documentation with
>> regard to this fact in Spark 2.0.1. Relevant PR is here:
>> https://github.com/apache/spark/pull/15005
>>
>> Your best bet as the project transitions from Java 7 to Java 8 is to use
>> the scripts build/mvn and build/sbt, which should be updated on a regular
>> basis with safe JVM options.
>>
>> Fred
>>
>> On Wed, Oct 5, 2016 at 1:40 AM, Marco Mistroni <mmistroni@gmail.com>
>> wrote:
>>
>>> Thanks Richard.  It also says that for Java 1.8 the mavenopts are not
>>> required..unless I misinterpreted the instructions...
>>> Kr
>>>
>>> On 5 Oct 2016 9:20 am, "Richard Siebeling" <rsiebeling@gmail.com> wrote:
>>>
>>>> sorry, now with the link included, see http://spark.apache.org/do
>>>> cs/latest/building-spark.html
>>>>
>>>> On Wed, Oct 5, 2016 at 10:19 AM, Richard Siebeling <
>>>> rsiebeling@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> did you set the following option: export MAVEN_OPTS="-Xmx2g
>>>>> -XX:ReservedCodeCacheSize=512m"
>>>>>
>>>>> kind regards,
>>>>> Richard
>>>>>
>>>>> On Tue, Oct 4, 2016 at 10:21 PM, Marco Mistroni <mmistroni@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all
>>>>>>  my mvn build of Spark 2.1 using Java 1.8 is spinning out of memory
>>>>>> with an error saying it cannot allocate enough memory during maven
>>>>>> compilation
>>>>>>
>>>>>> Instructions (in the Spark 2.0 page) says that MAVENOPTS are not
>>>>>> needed for Java 1.8 and , accoding to my understanding, spark build
process
>>>>>> will add it
>>>>>> during the build via mvn
>>>>>> Note; i am not using Zinc. Rather, i am using my own Maven version
>>>>>> (3.3.9), launching this command from the main spark directory. The
same
>>>>>> build works when i use Java 1.7(and MAVENOPTS)
>>>>>>
>>>>>> mvn -Pyarn -Dscala-2.11 -DskipTests clean package
>>>>>>
>>>>>> Could anyone assist?
>>>>>> kr
>>>>>>   marco
>>>>>>
>>>>>
>>>>>
>>>>
>>

Mime
View raw message