spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Saving compressed textFiles from a DStream in Scala
Date Tue, 09 Jun 2015 07:10:01 GMT
like this?

myDStream.foreachRDD(rdd => rdd.saveAsTextFile("/sigmoid/", codec ))


Thanks
Best Regards

On Mon, Jun 8, 2015 at 8:06 PM, Bob Corsaro <rcorsaro@gmail.com> wrote:

> It looks like saveAsTextFiles doesn't support the compression parameter of
> RDD.saveAsTextFile. Is there a way to add the functionality in my client
> code without patching Spark? I tried making my own saveFunc function and
> calling DStream.foreachRDD but ran into trouble with invoking rddToFileName
> and making the RDD type parameter work properly. It's probably just do to
> my lack of Scala knowledge. Can anyone give me a hand?
>
>   def saveAsTextFiles(prefix: String, suffix: String = ""): Unit =
> ssc.withScope {
>     val saveFunc = (rdd: RDD[T], time: Time) => {
>       val file = rddToFileName(prefix, suffix, time)
>       rdd.saveAsTextFile(file)
>     }
>     this.foreachRDD(saveFunc)
>   }
>
>

Mime
View raw message