spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andre Schumacher <schum...@icsi.berkeley.edu>
Subject Re: sbt assembly question
Date Thu, 12 Sep 2013 16:02:16 GMT

Hi,

on my Ubuntu 12.04 disk based laptop I set the target directory variable
inside SparkBuild.scala to something like

target := file("/run/shm/spark/target")

(note: you need to do that for all subprojects, including examples and such)

Thats a default tmpfs mount point but any other would do of course, too.
That helped a lot for me.

Andre

On 09/12/2013 08:48 AM, Han JU wrote:
> Hi,
> 
> I've had the same problem using sbt-assembly and I found out that the
> problem is that sbt doesn't have sufficient memory.
> Try something like: sbt -mem 3000 assembly
> 
> 
> 
> 2013/9/12 Matei Zaharia <matei.zaharia@gmail.com
> <mailto:matei.zaharia@gmail.com>>
> 
>     We could also have misconfigured something, or maybe it's different
>     on Linux. Definitely worth looking into.
> 
>     By the way, sbt assembly is only required in the development process
>     for a few unit tests that spawn a cluster (e.g. DistributedSuite and
>     some REPL tests). Otherwise you can just sbt compile.
> 
> 
>     Matei
> 
>     On Sep 11, 2013, at 7:21 PM, "Shao, Saisai" <saisai.shao@intel.com
>     <mailto:saisai.shao@intel.com>> wrote:
> 
>>     Hi Matei,____
>>      
>>     Thanks a lot. My colleague meets the same problem, so I’m just
>>     wondering whether this command is so slow. I will try it on SSD or
>>     in-memory FS.____
>>      
>>     Thanks____
>>     Jerry____
>>      
>>     *From:* Matei Zaharia [mailto:matei.zaharia@
>>     <mailto:matei.zaharia@>gmail.com <http://gmail.com>] 
>>     *Sent:* Thursday, September 12, 2013 10:08 AM
>>     *To:* user@spark.incubator.apache.org
>>     <mailto:user@spark.incubator.apache.org>
>>     *Subject:* Re: sbt assembly question____
>>      
>>     That's weird, it takes 30-60 seconds for me. If you can put this
>>     on an SSD or in-memory filesystem in any way that would help a
>>     lot. I have an SSD on my laptop.____
>>      
>>     Matei____
>>      
>>     On Sep 11, 2013, at 6:40 PM, "Shao, Saisai" <saisai.shao@intel.com
>>     <mailto:saisai.shao@intel.com>> wrote:____
>>
>>
>>     ____
>>     Hi all,____
>>      ____
>>     Now Spark changes sbt package to sbt assembly, and class path
>>     depends on assembly jar, but in my desktop assembly is quite slow,
>>     it costs almost 15 minutes to assemble the jars, is there any easy
>>     and fast way to assemble the jars.____
>>      ____
>>     Thanks____
>>     Jerry Shao____
>>
> 
> 
> 
> 
> -- 
> *JU Han*
> 
> Data Engineer @ Botify.com
> 
> +33 0619608888


Mime
View raw message