kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eduardo Costa Alfaia <e.costaalf...@unibs.it>
Subject Re: Spark and Kafka
Date Thu, 06 Nov 2014 17:50:38 GMT
This is my window:

reduceByKeyAndWindow(
       new Function2<Integer, Integer, Integer>() {
        @Override
         public Integer call(Integer i1, Integer i2) { return i1 + i2; }
       },
       new Function2<Integer, Integer, Integer>() {
         public Integer call(Integer i1, Integer i2) { return i1 - i2; }
       },
       new Duration(60 * 5 * 1000),
       new Duration(1 * 1000)
     );

> On Nov 6, 2014, at 18:37, Gwen Shapira <gshapira@cloudera.com> wrote:
> 
> What's the window size? If the window is around 10 seconds and you are
> sending data at very stable rate, this is expected.
> 
> 
> 
> On Thu, Nov 6, 2014 at 9:32 AM, Eduardo Costa Alfaia <e.costaalfaia@unibs.it
>> wrote:
> 
>> Hi Guys,
>> 
>> I am doing some tests with Spark Streaming and Kafka, but I have seen
>> something strange, I have modified the JavaKafkaWordCount to use
>> ReducebyKeyandWindow and to print in the screen the accumulated numbers of
>> the words, in the beginning spark works very well in each interaction the
>> numbers of the words increase but after 12 a 13 sec the results repeats
>> continually.
>> 
>> My program producer remain sending the words toward the kafka.
>> 
>> Does anyone have any idea about this?
>> 
>> 
>> -------------------------------------------
>> Time: 1415272266000 ms
>> -------------------------------------------
>> (accompanied
>> them,6)
>> (merrier,5)
>> (it
>> possessed,5)
>> (the
>> treacherous,5)
>> (Quite,12)
>> (offer,273)
>> (rabble,58)
>> (exchanging,16)
>> (Genoa,18)
>> (merchant,41)
>> ...
>> -------------------------------------------
>> Time: 1415272267000 ms
>> -------------------------------------------
>> (accompanied
>> them,12)
>> (merrier,12)
>> (it
>> possessed,12)
>> (the
>> treacherous,11)
>> (Quite,24)
>> (offer,602)
>> (rabble,132)
>> (exchanging,35)
>> (Genoa,36)
>> (merchant,84)
>> ...
>> -------------------------------------------
>> Time: 1415272268000 ms
>> -------------------------------------------
>> (accompanied
>> them,17)
>> (merrier,18)
>> (it
>> possessed,17)
>> (the
>> treacherous,17)
>> (Quite,35)
>> (offer,889)
>> (rabble,192)
>> (the
>> bed,1)
>> (exchanging,51)
>> (Genoa,54)
>> ...
>> -------------------------------------------
>> Time: 1415272269000 ms
>> -------------------------------------------
>> (accompanied
>> them,17)
>> (merrier,18)
>> (it
>> possessed,17)
>> (the
>> treacherous,17)
>> (Quite,35)
>> (offer,889)
>> (rabble,192)
>> (the
>> bed,1)
>> (exchanging,51)
>> (Genoa,54)
>> ...
>> 
>> -------------------------------------------
>> Time: 1415272270000 ms
>> -------------------------------------------
>> (accompanied
>> them,17)
>> (merrier,18)
>> (it
>> possessed,17)
>> (the
>> treacherous,17)
>> (Quite,35)
>> (offer,889)
>> (rabble,192)
>> (the
>> bed,1)
>> (exchanging,51)
>> (Genoa,54)
>> ...
>> 
>> 
>> --
>> Informativa sulla Privacy: http://www.unibs.it/node/8155
>> 


-- 
Informativa sulla Privacy: http://www.unibs.it/node/8155

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message