storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navin Ipe <navin....@searchlighthealth.com>
Subject Re: How long until fields grouping gets overwhelmed with data?
Date Wed, 10 Aug 2016 15:28:39 GMT
Oh that's good to know. I assume it works like this:
https://en.wikipedia.org/wiki/Hash_function#Hashing_uniformly_distributed_data

On Wed, Aug 10, 2016 at 6:23 PM, Nathan Leung <ncleung@gmail.com> wrote:

> It's based on a modulo of a hash of the field. The fields grouping is
> stateless.
>
> On Aug 10, 2016 8:18 AM, "Navin Ipe" <navin.ipe@searchlighthealth.com>
> wrote:
>
>> Hi,
>>
>> For spouts to be able to continuously send a fields grouped tuple to the
>> same bolt, it would have to store a key value map something like this,
>> right?
>>
>> field1023 ---> Bolt1
>> field1343 ---> Bolt3
>> field1629 ---> Bolt5
>> field1726 ---> Bolt1
>> field1481 ---> Bolt3
>>
>> So if my topology runs for a very long time and the spout generates many
>> unique field values, won't this key value map run out of memory eventually?
>>
>> OR is there a failsafe or a map limit that Storm has to handle this
>> without crashing?
>>
>> If memory problems could happen, what would be an alternative way to
>> solve this problem where many unique fields could get generated over time?
>>
>> --
>> Regards,
>> Navin
>>
>


-- 
Regards,
Navin

Mime
View raw message