spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From krishmah <krish...@gmail.com>
Subject Shuffle Service - Connection Inactive - Creating new one
Date Wed, 01 Jun 2016 11:13:36 GMT
 I am seeing this in my logs. It appears to reopen connection to Shuffle
Service. Whenever this happens, I am seeing the partition to take longer
time to complete. I was running a job with 1000 partitions. About 600
partitions, it was completing in less than 20 mins a partition and after
that I started seeing each partition taking about 1.5 hours. When I was
checking the logs to see what was going on, I was seeing lot of these
connections getting created. Any thing that I am doing wrongly which is
creating this issue?

16/05/31 04:49:47 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 04:50:27 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 04:57:59 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 04:58:15 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 04:58:26 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:02:33 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:02:39 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:06:19 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:06:26 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:09:45 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:12:27 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:17:10 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:19:20 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:19:20 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:21:34 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:30:43 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:32:57 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:33:01 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:35:08 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:38:06 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:38:53 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:40:23 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:41:47 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:43:40 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-Service-Connection-Inactive-Creating-new-one-tp27061.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message