spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <>
Subject Re: store spark streaming dstream in hdfs or cassandra
Date Thu, 31 Jul 2014 19:12:57 GMT
Off the top of my head, you can use the ForEachDStream to which you pass 
in the code that writes to Hadoop, and then register that as an output 
stream, so the function you pass in is periodically executed and causes 
the data to be written to HDFS. If you are ok with the data being in 
text format - simply use saveAsTextFiles method in the RDD class.

salemi wrote:
> Hi,
> I was wondering what is the best way to store off dstreams in hdfs or
> casandra.
> Could somebody provide an example?
> Thanks,
> Ali
> --
> View this message in context: 
> Sent from the Apache Spark User List mailing list archive at

View raw message