spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lisendong <lisend...@163.com>
Subject gc time too long when using mllib als
Date Tue, 03 Mar 2015 08:56:11 GMT
why does the gc time so long?

i 'm using als in mllib,  while the garbage collection time is too long
(about 1/3 of total time)

I have tried some measures in the "tunning spark guide", and try to set the
new generation memory, but it still does not work...





Tasks

Task Index	Task ID	Status	Locality Level	Executor	Launch Time	Duration	GC
Time	Result Ser Time	Shuffle Read	Write Time	Shuffle Write	Errors
1	2801	SUCCESS	PROCESS_LOCAL	h1.zw	2015/03/03 16:35:15	8.6 min	3.3 min	
1238.3 MB	57 ms	69.2 MB	
0	2800	SUCCESS	PROCESS_LOCAL	h11.zw	2015/03/03 16:35:15	6.0 min	1.1 min	
1261.0 MB	55 ms	68.6 MB	
2	2802	SUCCESS	PROCESS_LOCAL	h9.zw	2015/03/03 16:35:15	5.0 min	1.5 min	
834.4 MB	60 ms	69.6 MB	
4	2804	SUCCESS	PROCESS_LOCAL	h4.zw	2015/03/03 16:35:15	4.4 min	59 s		689.8
MB	62 ms	71.4 MB	
3	2803	SUCCESS	PROCESS_LOCAL	h8.zw	2015/03/03 16:35:15	4.2 min	1.6 min	
803.6 MB	66 ms	71.5 MB	
7	2807	SUCCESS	PROCESS_LOCAL	h6.zw	2015/03/03 16:35:15	4.3 min	1.4 min	
733.1 MB	9 s	66.5 MB	
6	2806	SUCCESS	PROCESS_LOCAL	h10.zw	2015/03/03 16:35:15	6.4 min	3.1 min	
950.5 MB	68 ms	69.3 MB	
5	2805	SUCCESS	PROCESS_LOCAL	h3.zw	2015/03/03 16:35:15	8.0 min	2.7 min	
1132.0 MB	64 ms	70.3 MB	
8	2808	SUCCESS	PROCESS_LOCAL	h12.zw	2015/03/03 16:35:15	4.5 min	2.2 min	
1304.2 MB	60 ms	69.4 MB	



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/gc-time-too-long-when-using-mllib-als-tp21891.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message