spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manjunath Shetty H <>
Subject How to collect Spark dataframe write metrics
Date Sun, 01 Mar 2020 12:32:28 GMT
Hi all,

Basically my use case is to validate the DataFrame rows count before and after writing to
HDFS. Is this even to good practice ? Or Should relay on spark for guaranteed writes ?.

If it is a good practice to follow then how to get the DataFrame level write metrics ?

Any pointers would be helpful.

Thanks and Regards

View raw message