spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob Corsaro <rcors...@gmail.com>
Subject Re: Saving compressed textFiles from a DStream in Scala
Date Wed, 10 Jun 2015 15:05:39 GMT
Thanks Akhil. For posterity, I ended up with:

      https://gist.github.com/dokipen/aa07f351a970fe54fcff

I couldn't get rddToFilename() to work, but it's impl was pretty simple.
I'm a poet but I don't know it.

On Tue, Jun 9, 2015 at 3:10 AM Akhil Das <akhil@sigmoidanalytics.com> wrote:

> like this?
>
> myDStream.foreachRDD(rdd => rdd.saveAsTextFile("/sigmoid/", codec ))
>
>
> Thanks
> Best Regards
>
> On Mon, Jun 8, 2015 at 8:06 PM, Bob Corsaro <rcorsaro@gmail.com> wrote:
>
>> It looks like saveAsTextFiles doesn't support the compression parameter
>> of RDD.saveAsTextFile. Is there a way to add the functionality in my client
>> code without patching Spark? I tried making my own saveFunc function and
>> calling DStream.foreachRDD but ran into trouble with invoking rddToFileName
>> and making the RDD type parameter work properly. It's probably just do to
>> my lack of Scala knowledge. Can anyone give me a hand?
>>
>>   def saveAsTextFiles(prefix: String, suffix: String = ""): Unit =
>> ssc.withScope {
>>     val saveFunc = (rdd: RDD[T], time: Time) => {
>>       val file = rddToFileName(prefix, suffix, time)
>>       rdd.saveAsTextFile(file)
>>     }
>>     this.foreachRDD(saveFunc)
>>   }
>>
>>
>

Mime
View raw message