storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Leung <ncle...@gmail.com>
Subject Re: How long until fields grouping gets overwhelmed with data?
Date Wed, 10 Aug 2016 12:53:39 GMT
It's based on a modulo of a hash of the field. The fields grouping is
stateless.

On Aug 10, 2016 8:18 AM, "Navin Ipe" <navin.ipe@searchlighthealth.com>
wrote:

> Hi,
>
> For spouts to be able to continuously send a fields grouped tuple to the
> same bolt, it would have to store a key value map something like this,
> right?
>
> field1023 ---> Bolt1
> field1343 ---> Bolt3
> field1629 ---> Bolt5
> field1726 ---> Bolt1
> field1481 ---> Bolt3
>
> So if my topology runs for a very long time and the spout generates many
> unique field values, won't this key value map run out of memory eventually?
>
> OR is there a failsafe or a map limit that Storm has to handle this
> without crashing?
>
> If memory problems could happen, what would be an alternative way to solve
> this problem where many unique fields could get generated over time?
>
> --
> Regards,
> Navin
>

Mime
View raw message