spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vamsi Krishna <vamsi.attl...@gmail.com>
Subject spark streaming - how to purge old data files in data directory
Date Sun, 19 Jun 2016 00:28:59 GMT
Hi,

I'm on HDP 2.3.2 cluster (Spark 1.4.1).
I have a spark streaming app which uses 'textFileStream' to stream simple
CSV files and process.
I see the old data files that are processed are left in the data directory.
What is the right way to purge the old data files in data directory on HDFS?

Thanks,
Vamsi Attluri
-- 
Vamsi Attluri

Mime
View raw message