spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Han JU <ju.han.fe...@gmail.com>
Subject Re: sbt assembly question
Date Thu, 12 Sep 2013 15:48:42 GMT
Hi,

I've had the same problem using sbt-assembly and I found out that the
problem is that sbt doesn't have sufficient memory.
Try something like: sbt -mem 3000 assembly



2013/9/12 Matei Zaharia <matei.zaharia@gmail.com>

> We could also have misconfigured something, or maybe it's different on
> Linux. Definitely worth looking into.
>
> By the way, sbt assembly is only required in the development process for a
> few unit tests that spawn a cluster (e.g. DistributedSuite and some REPL
> tests). Otherwise you can just sbt compile.
>
>
> Matei
>
> On Sep 11, 2013, at 7:21 PM, "Shao, Saisai" <saisai.shao@intel.com> wrote:
>
> Hi Matei,****
>
> Thanks a lot. My colleague meets the same problem, so I’m just wondering
> whether this command is so slow. I will try it on SSD or in-memory FS.****
>
> Thanks****
> Jerry****
>
> *From:* Matei Zaharia [mailto:matei.zaharia@gmail.com]
> *Sent:* Thursday, September 12, 2013 10:08 AM
> *To:* user@spark.incubator.apache.org
> *Subject:* Re: sbt assembly question****
>
> That's weird, it takes 30-60 seconds for me. If you can put this on an SSD
> or in-memory filesystem in any way that would help a lot. I have an SSD on
> my laptop.****
>
> Matei****
>
> On Sep 11, 2013, at 6:40 PM, "Shao, Saisai" <saisai.shao@intel.com> wrote:
> ****
>
>
> ****
> Hi all,****
>  ****
> Now Spark changes sbt package to sbt assembly, and class path depends on
> assembly jar, but in my desktop assembly is quite slow, it costs almost 15
> minutes to assemble the jars, is there any easy and fast way to assemble
> the jars.****
>  ****
> Thanks****
> Jerry Shao****
>
>
>


-- 
*JU Han*

Data Engineer @ Botify.com

+33 0619608888

Mime
View raw message