flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shashank agarwal <shashank...@gmail.com>
Subject Can i use lot of keyd states or should i use 1 big key state.
Date Mon, 31 Jul 2017 16:01:29 GMT

I have to compute results on basis of lot of history data, parameters like
total transactions in last 1 month, last 1 day, last 1 hour etc. by email
id, ip, mobile, name, address, zipcode etc.

So my question is this right approach to create keyed state by email,
mobile, zipcode etc. or should i create 1 big mapped state (BS) and than
process that BS, may be in process function or by applying some loop and
filter logic in window or process function.

My main worry is i will end up with millions of states, because there can
be millions unique emails, phone numbers or zipcode if i create keyed state
by email, phone etc.

am i right ? is this impact on the performance or is this wrong approach ?
Which approach would you suggest in this use case.

Thanks Regards

 ---  Trying to mobilize the things....

View raw message