spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangrui Meng <men...@gmail.com>
Subject Re: gc time too long when using mllib als
Date Tue, 03 Mar 2015 20:36:51 GMT
Also try 1.3.0-RC1 or the current master. ALS should performance much
better in 1.3. -Xiangrui

On Tue, Mar 3, 2015 at 1:00 AM, Akhil Das <akhil@sigmoidanalytics.com> wrote:
> You need to increase the parallelism/repartition the data to a higher number
> to get ride of those.
>
> Thanks
> Best Regards
>
> On Tue, Mar 3, 2015 at 2:26 PM, lisendong <lisendong@163.com> wrote:
>>
>> why does the gc time so long?
>>
>> i 'm using als in mllib,  while the garbage collection time is too long
>> (about 1/3 of total time)
>>
>> I have tried some measures in the "tunning spark guide", and try to set
>> the
>> new generation memory, but it still does not work...
>>
>>
>>
>>
>>
>> Tasks
>>
>> Task Index      Task ID Status  Locality Level  Executor        Launch
>> Time     Duration        GC
>> Time    Result Ser Time Shuffle Read    Write Time      Shuffle Write
>> Errors
>> 1       2801    SUCCESS PROCESS_LOCAL   h1.zw   2015/03/03 16:35:15
>> 8.6 min 3.3 min
>> 1238.3 MB       57 ms   69.2 MB
>> 0       2800    SUCCESS PROCESS_LOCAL   h11.zw  2015/03/03 16:35:15
>> 6.0 min 1.1 min
>> 1261.0 MB       55 ms   68.6 MB
>> 2       2802    SUCCESS PROCESS_LOCAL   h9.zw   2015/03/03 16:35:15
>> 5.0 min 1.5 min
>> 834.4 MB        60 ms   69.6 MB
>> 4       2804    SUCCESS PROCESS_LOCAL   h4.zw   2015/03/03 16:35:15
>> 4.4 min 59 s            689.8
>> MB      62 ms   71.4 MB
>> 3       2803    SUCCESS PROCESS_LOCAL   h8.zw   2015/03/03 16:35:15
>> 4.2 min 1.6 min
>> 803.6 MB        66 ms   71.5 MB
>> 7       2807    SUCCESS PROCESS_LOCAL   h6.zw   2015/03/03 16:35:15
>> 4.3 min 1.4 min
>> 733.1 MB        9 s     66.5 MB
>> 6       2806    SUCCESS PROCESS_LOCAL   h10.zw  2015/03/03 16:35:15
>> 6.4 min 3.1 min
>> 950.5 MB        68 ms   69.3 MB
>> 5       2805    SUCCESS PROCESS_LOCAL   h3.zw   2015/03/03 16:35:15
>> 8.0 min 2.7 min
>> 1132.0 MB       64 ms   70.3 MB
>> 8       2808    SUCCESS PROCESS_LOCAL   h12.zw  2015/03/03 16:35:15
>> 4.5 min 2.2 min
>> 1304.2 MB       60 ms   69.4 MB
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/gc-time-too-long-when-using-mllib-als-tp21891.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message