storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cody A. Ray" <cody.a....@gmail.com>
Subject Re: How to recover from a Spout excpetion?
Date Thu, 22 May 2014 21:32:00 GMT
I think the problem is that the code bundled with your topology is
serialized by the nimbus and transferred to the supervisors, so all the
code you use needs to be serializable. Something in twitter4j must be a
using a non-serializable class (the ScopesImpl class).

I don't have a suggestion other than patch/contribute twitter4j or look at
a different twitter library... But I think that's the root cause.

Someone else may have a better suggestion.

-Cody
On May 22, 2014 1:00 PM, "Alexis Sukrieh" <sukria@gmail.com> wrote:

> Hi,
>
> I have built a topology with a spout that connects to Twitter's streaming
> API (with twitter4j), I have also 4 bolts that count words inside.
>
> This is mainly a proof of concept for me, in order to get up to speed with
> Storm.
>
> So far, it's working fine... until my Spout gets an exception from
> twitter4j:
>
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException:
twitter4j.ScopesImpl
>
> When this happens, my topology is somehow broken: the Redis database that gets updated
by the last bolt is not updated anymore.
>
> All I can do is killing the topology and submit it again. That's a bit annoying as I'd
like my topology to never stops running.
>
> I'd like to understand how to catch this exception, and make sure my topology is protected
against invalid tuples.
>
> In the Spout class, I really don't see how I could catch the exception.
>
> Any help is more than welcome.
>
>
> Thanks.
>
>
> --
> Alexis Sukrieh
>
>

Mime
View raw message