spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Barnaby <>
Subject streaming sequence files?
Date Thu, 24 Jul 2014 01:43:31 GMT
If I save an RDD as a sequence file such as:

    val wordCounts = => (x, 1)).reduceByKey(_ + _)
    wordCounts.foreachRDD( d => {
        d.saveAsSequenceFile("tachyon://localhost:19998/files/WordCounts-" +
(new SimpleDateFormat("yyyyMMdd-HHmmss") format

How can I use these results in another Spark app since there is no


What is the best way to save RDDs of objects to files in one streaming app
so that another app can stream those files in? Basically, reuse partially
reduced RDDs for further processing so that it doesn't have to be done more
than once.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message