storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kobi Salant <ko...@liveperson.com>
Subject Re: Serialization of member variables in spouts
Date Thu, 04 Dec 2014 13:42:08 GMT
Hi Vincent,

You are right that there is no special function for rebalancing but the
deactivate method should be called. I am not sure if you can know the
reason why the topology was deactivated but you can add you cleanups in the
deactivate and the warm ups in the activate.

Kobi

On Thu, Dec 4, 2014 at 2:14 PM, Vincent Russell <vincent.russell@gmail.com>
wrote:

> Thank you for the response Kobi,
>
> What do mean time is given to clean up?  From what I have read it looks
> like the close method on the spout is not guaranteed to be called.   How
> can I clean up if the spout has no notice that it is being shut down?
> On Dec 4, 2014 2:00 AM, "Kobi Salant" <kobis@liveperson.com> wrote:
>
>> Hi Vincent,
>>
>> From our experience, rebalancing does not keep the inner state of the
>> bolts/spouts but prepare them from fresh.
>> It does give your components time to clean up or commit offsets so you
>> can recover cleanly afterwards.
>>
>> Kobi
>>
>> On Thu, Dec 4, 2014 at 12:21 AM, Vincent Russell <
>> vincent.russell@gmail.com> wrote:
>>
>>>
>>> I have a question regarding serialization of member variables in
>>> spouts.  We are using storm to parse files in HDFS with storm spouts.
>>> Because files usually emit many tuples we store which file we are parsing
>>> as a member variable in the spout.
>>>
>>> We have noticed that when rebalancing happens we tend to lose track of
>>> the file that is being parsed if the file isn't completed.
>>>
>>> Can member variables be deserialized during rebalancing?
>>>
>>> Is there some other good tricks to use where state must be saved between
>>> nextTuple calls to protect yourself from rebalancing.
>>>
>>>
>>> I'm using storm version 0.9.0.1.
>>>
>>> Thanks,
>>> Vincent
>>>
>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>

-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Mime
View raw message