spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sayeh Roshan <>
Subject Reading the last line of each file in a set of text files
Date Mon, 02 Aug 2021 22:50:44 GMT
Hi users,
Does anyone here has experience with written spark code that just read the
last line of each text file in a directory, s3 bucket, etc?
I am looking for a solution that doesn’t require reading the whole file. I
basically wonder whether you can create a data frame/Rdd using file seek.
Not sure whether there is such a thing already available in spark.
Thank you very much in advance.

View raw message