spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Mistroni <mmistr...@gmail.com>
Subject Re: building Spark 2.1 vs Java 1.8 on Ubuntu 16/06
Date Thu, 06 Oct 2016 20:01:59 GMT
Thanks Fred for pointers...so far I was only able to build 2.1 with Java 7
and no zinc.
Will try options u suggest. FYI building with sbt ends up in oom even with
Java 7
I will try and update this thread
Kr

On 6 Oct 2016 8:58 pm, "Fred Reiss" <freiss.oss@gmail.com> wrote:

> There's no option to prevent build/mvn from starting the zinc server, but
> you should be able to prevent the maven build from using the zinc server by
> changing the <useZincServer> option at line 1935 of the master pom.xml.
>
> Note that the zinc-based compile works on my Ubuntu 16.04 box. You might
> be able to get zinc-based compiles working by tweaking your settings. A few
> things to try:
> -- Make sure another build hasn't left a second, incompatible copy of zinc
> squatting on the port that Spark expects to use
> -- Try setting the environment variable JAVA_7_HOME to point to and
> OpenJDK 7 installation. build/mvn runs zinc with Java 7 if that is
> available.
>
> Note that setting JAVA_7_HOME will break incremental compilation for
> sbt-based builds. Use that environment variable with restraint.
>
> Fred
>
>
>
> On Thu, Oct 6, 2016 at 2:22 AM, Marco Mistroni <mmistroni@gmail.com>
> wrote:
>
>> Thanks Fred
>> The build/mvn will trigger compilation using zinc and I want to avoid
>> that as every time I have tried it runs into errors while compiling spark
>> core. How can I disable zinc by default?
>> Kr
>>
>> On 5 Oct 2016 10:53 pm, "Fred Reiss" <freiss.oss@gmail.com> wrote:
>>
>>> Actually the memory options *are* required for Java 1.8. Without them
>>> the build will fail intermittently. We just updated the documentation with
>>> regard to this fact in Spark 2.0.1. Relevant PR is here:
>>> https://github.com/apache/spark/pull/15005
>>>
>>> Your best bet as the project transitions from Java 7 to Java 8 is to use
>>> the scripts build/mvn and build/sbt, which should be updated on a regular
>>> basis with safe JVM options.
>>>
>>> Fred
>>>
>>> On Wed, Oct 5, 2016 at 1:40 AM, Marco Mistroni <mmistroni@gmail.com>
>>> wrote:
>>>
>>>> Thanks Richard.  It also says that for Java 1.8 the mavenopts are not
>>>> required..unless I misinterpreted the instructions...
>>>> Kr
>>>>
>>>> On 5 Oct 2016 9:20 am, "Richard Siebeling" <rsiebeling@gmail.com>
>>>> wrote:
>>>>
>>>>> sorry, now with the link included, see http://spark.apache.org/do
>>>>> cs/latest/building-spark.html
>>>>>
>>>>> On Wed, Oct 5, 2016 at 10:19 AM, Richard Siebeling <
>>>>> rsiebeling@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> did you set the following option: export MAVEN_OPTS="-Xmx2g
>>>>>> -XX:ReservedCodeCacheSize=512m"
>>>>>>
>>>>>> kind regards,
>>>>>> Richard
>>>>>>
>>>>>> On Tue, Oct 4, 2016 at 10:21 PM, Marco Mistroni <mmistroni@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all
>>>>>>>  my mvn build of Spark 2.1 using Java 1.8 is spinning out of
memory
>>>>>>> with an error saying it cannot allocate enough memory during
maven
>>>>>>> compilation
>>>>>>>
>>>>>>> Instructions (in the Spark 2.0 page) says that MAVENOPTS are
not
>>>>>>> needed for Java 1.8 and , accoding to my understanding, spark
build process
>>>>>>> will add it
>>>>>>> during the build via mvn
>>>>>>> Note; i am not using Zinc. Rather, i am using my own Maven version
>>>>>>> (3.3.9), launching this command from the main spark directory.
The same
>>>>>>> build works when i use Java 1.7(and MAVENOPTS)
>>>>>>>
>>>>>>> mvn -Pyarn -Dscala-2.11 -DskipTests clean package
>>>>>>>
>>>>>>> Could anyone assist?
>>>>>>> kr
>>>>>>>   marco
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>

Mime
View raw message