spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yeikel valdes <em...@yeikel.com>
Subject Re:Load Time from HDFS
Date Wed, 10 Apr 2019 17:55:01 GMT
What about a simple call to nanotime?

long startTime = System.nanoTime();

//Spark work here

long endTime = System.nanoTime();

long duration = (endTime - startTime)

println(duration)

Count recomputes the df so it makes sense it takes longer for you.

---- On Tue, 02 Apr 2019 07:06:30 -0700 kolokasis@ics.forth.gr wrote ----

Hello, 

I want to ask if there any way to measure HDFS data loading time at  
the start of my program. I tried to add an action e.g count() after val 
data = sc.textFile() call. But I notice that my program takes more time 
to finish than before adding count call. Is there any other way to do it ? 

Thanks, 
--Iacovos 

--------------------------------------------------------------------- 
To unsubscribe e-mail: user-unsubscribe@spark.apache.org 


Mime
View raw message