spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: sbt assembly question
Date Thu, 12 Sep 2013 02:40:24 GMT
We could also have misconfigured something, or maybe it's different on Linux. Definitely worth
looking into.

By the way, sbt assembly is only required in the development process for a few unit tests
that spawn a cluster (e.g. DistributedSuite and some REPL tests). Otherwise you can just sbt
compile.

Matei

On Sep 11, 2013, at 7:21 PM, "Shao, Saisai" <saisai.shao@intel.com> wrote:

> Hi Matei,
>  
> Thanks a lot. My colleague meets the same problem, so I’m just wondering whether this
command is so slow. I will try it on SSD or in-memory FS.
>  
> Thanks
> Jerry
>  
> From: Matei Zaharia [mailto:matei.zaharia@gmail.com] 
> Sent: Thursday, September 12, 2013 10:08 AM
> To: user@spark.incubator.apache.org
> Subject: Re: sbt assembly question
>  
> That's weird, it takes 30-60 seconds for me. If you can put this on an SSD or in-memory
filesystem in any way that would help a lot. I have an SSD on my laptop.
>  
> Matei
>  
> On Sep 11, 2013, at 6:40 PM, "Shao, Saisai" <saisai.shao@intel.com> wrote:
> 
> 
> Hi all,
>  
> Now Spark changes sbt package to sbt assembly, and class path depends on assembly jar,
but in my desktop assembly is quite slow, it costs almost 15 minutes to assemble the jars,
is there any easy and fast way to assemble the jars.
>  
> Thanks
> Jerry Shao


Mime
View raw message