spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alireza Salemi" <alireza.sal...@udo.edu>
Subject Re: Spark Streaming - how to implement multiple calculation using the same data set
Date Wed, 03 Sep 2014 03:36:43 GMT
Tobias,

That was what I was planing to do and technical lead is the opinion that
we should some how process a message only once and calculate all the
measures for the worker.

I was wondering if there is a solution out there for that?

Thanks,
Ali

> Hi,
>
> On Wed, Sep 3, 2014 at 6:54 AM, salemi <alireza.salemi@udo.edu> wrote:
>
>> I was able to calculate the individual measures separately and know I
>> have
>> to merge them and spark streaming doesn't support outer join yet.
>>
>
> Can't you assign some dummy key (e.g., index) before your processing and
> then join on that key using a function from
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.PairDStreamFunctions
> ?
>
> Tobias
>



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message