spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankur Srivastava <ankur.srivast...@gmail.com>
Subject Spark Shuffle Issue
Date Wed, 12 Oct 2016 06:16:10 GMT
Hi,

I am upgrading my jobs to Spark 1.6 and am running into shuffle issues. I
have tried all options and now am falling back to legacy memory model but
still running into same issue.

I have set spark.shuffle.blockTransferService to nio.

16/10/12 06:00:10 INFO MapOutputTrackerMaster: Size of output statuses for
shuffle 6 is 57910504 bytes

16/10/12 06:00:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map
output locations for shuffle 6 to dedwfprshd023.de.neustar.com:52510

16/10/12 06:00:10 ERROR TransportRequestHandler: Error sending result
RpcResponse{requestId=7357168459686146036,
body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=57910531
cap=57910531]}} to <hostname>; closing connection


Any other settings that I should tune?


Thanks

Ankur

Mime
View raw message