spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pulasthi Supun Wickramasinghe <pulasthi...@gmail.com>
Subject Large variation in spark in Task Deserialization Time
Date Mon, 10 Oct 2016 17:53:05 GMT
Hi All,

I am seeing a huge variation on spark Task Deserialization Time for my
collect and reduce operations. while most tasks complete within 100ms a few
take mote than a couple of seconds which slows the entire program down. I
have attached a screen shot of the web ui where you can see the variation


As you can see the Task Deserialization Time time has a Max of 7s and 75th
percentile at 0.3 seconds.

Does anyone know the reasons that may cause these kind of numbers. Any help
would be greatly appreciated.

Best Regards,
‚Äč
-- 
Pulasthi S. Wickramasinghe
Graduate Student  | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035

Mime
View raw message