spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yeikel valdes <>
Subject Re:Load Time from HDFS
Date Wed, 10 Apr 2019 17:55:01 GMT
What about a simple call to nanotime?

long startTime = System.nanoTime();

//Spark work here

long endTime = System.nanoTime();

long duration = (endTime - startTime)


Count recomputes the df so it makes sense it takes longer for you.

---- On Tue, 02 Apr 2019 07:06:30 -0700 wrote ----


I want to ask if there any way to measure HDFS data loading time at  
the start of my program. I tried to add an action e.g count() after val 
data = sc.textFile() call. But I notice that my program takes more time 
to finish than before adding count call. Is there any other way to do it ? 


To unsubscribe e-mail: 

View raw message