spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Holden Karau <hol...@pigscanfly.ca>
Subject Re: what does DStream.union() do?
Date Wed, 29 Oct 2014 22:42:55 GMT
On Wed, Oct 29, 2014 at 3:39 PM, spr <spr@yarcdata.com> wrote:

> I need more precision to understand.  If the elements of one DStream/RDD
> are
> (String) and the elements of the other are (Time, Int), what does "union"
> mean?  I'm hoping for (String, Time, Int) but that appears optimistic.  :)
>
It won't compile.

> Do the elements have to be of homogeneous type?
>
Yes. From the scaladoc (
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.DStream
)  you can see DStreams are generic/templated on a type (T) and the union
function works on a DStream of the same templated type. If you have
hetrogeneous data you can first map each DStream it to a case class with
options or try something like
http://stackoverflow.com/questions/3508077/does-scala-have-type-disjunction-union-types


>
>
> Holden Karau wrote
> > The union function simply returns a DStream with the elements from both.
> > This is the same behavior as when we call union on RDDs :) (You can think
> > of union as similar to the union operator on sets except without the
> > unique
> > element restrictions).
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/what-does-DStream-union-do-tp17673p17682.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
Cell : 425-233-8271

Mime
View raw message