flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Efficiency with different approaches of aggregation in Flink
Date Thu, 19 Apr 2018 07:44:00 GMT
Hi Teena,

I'd go with approach 2. The performance difference shouldn't be significant
compared to 1. but it is much easier to implement, IMO.

Avoid approach 3. It will be much slower because you need at least one call
to an external data store and more difficult to implement.
Flink's checkpointing mechanism (as used by 1. and 2. ) gives you better
consistency and protection against failures than what you can achieve with
3.

Cheers, Fabian

2018-04-19 8:42 GMT+02:00 Puneet Kinra <puneet.kinra@customercentria.com>:

> Hi Teena
>
> If you are proceeding with point 3, no doubt it will add some overhead but
> major significance is that you are persisting the state as per
> some key. so there will not be data loss in case of the job failure.
>
>
>
> On Thu, Apr 19, 2018 at 11:45 AM, Teena Kappen // BPRISE <
> teena.kappen@bprise.com> wrote:
>
>> Hi,
>>
>>
>>
>> If I have to aggregate a value in a stream of records, which one of the
>> below approaches will be the most/least efficient?
>>
>>
>>
>>    1. Using a Global Window to aggregate the value and emit the record
>>    when it reaches a particular threshold value.
>>    2. Using a FlatMap with a State Variable which gets updated with each
>>    incoming record and emit the record when it reaches the threshold value.
>>    3. Using a FlatMap to store the aggregated value in an in-memory DB
>>    like Redis and query the value and update it with each incoming record, and
>>    emit the record when it reaches the threshold value.
>>
>>
>>
>> Please rate the three approaches according to their efficiency.
>>
>>
>>
>> Regards,
>>
>> Teena
>>
>
>
>
> --
> *Cheers *
>
> *Puneet Kinra*
>
> *Mobile:+918800167808 | Skype : puneet.kinra@customercentria.com
> <puneet.kinra@customercentria.com>*
>
> *e-mail :puneet.kinra@customercentria.com
> <puneet.kinra@customercentria.com>*
>
>
>

Mime
View raw message