spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Singh <mohit1...@gmail.com>
Subject Lazyoutput format in spark
Date Sat, 01 Mar 2014 01:18:36 GMT
Hi,
  Is there something equivalent of LazyOutputFormat equivalent in spark
(pyspark)
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/output/LazyOutputFormat.html
Basically, something like where I only save files which has some data in it
rather than saving all the files as in some cases, your majority of files
can be empty?
Thanks

-- 
Mohit

"When you want success as badly as you want the air, then you will get it.
There is no other secret of success."
-Socrates

Mime
View raw message