storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gergely Nagy <foge...@gmail.com>
Subject Re: java.io.NotSerializableException: redis.clients.jedis.BinaryJedis$SetFromList
Date Tue, 15 Dec 2015 04:21:55 GMT
Thank you Jungtaek!

Yup. You are exactly right. I was too tired or something and I didn't
realize that the error is complaining about the inner class. Thank you for
taking your time to look into this.

Regards,
Gergely

2015-12-14 23:41 GMT+09:00 임정택 <kabhwan@gmail.com>:

> Hi Gergely,
>
> Thanks for using storm-redis, and report issue.
>
> The thing Storm was trying to serialize is BinaryJedis$SetFromList, not
> BinaryJedis.
>
>
> https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java#L74
>
> Jedis.smembers() returns SetFromList, which wraps List to provide Set
> interface.
> (It is just one of the optimizations in Jedis.)
>
> Unfortunately SetFromList doesn't implement Serializable for now, so you
> can encounter  NotSerializableException when trying to emit the result of
> smembers directly.
>
> Two workarounds are here,
>
> 1. Implement custom Kyro serializer for SetFromList and register to
> topology.
> http://storm.apache.org/documentation/Serialization.html
>
> 2. Copy the result into HashSet and emit instead.
>
> I'll try to craft the patch to Jedis which makes SetFromList to implement
> Serializable.
> Before applying and releasing, I'll also try to make storm-redis to
> provide easy workaround. (Considering 1. for now.)
>
> Thanks!
> Jungtaek Lim (HeartSaVioR)
>
>
>
>
>
> 2015-12-14 21:54 GMT+09:00 Gergely Nagy <fogetti@gmail.com>:
>
>> Hello fellow storm troopers.
>>
>> I am trying to build a storm topology in Storm 0.10.0. I am also trying
>> to use storm-redis to save some interim results during operation.
>>
>> Now my problem is that when I run my topology locally everything works
>> fine, but when I run it remotely, storm sends me this nice stacktrace:
>> https://gist.github.com/Fogetti/23304178a534fa1f8f63
>>
>> I don't get it. What's going on? Why is storm serializing BinaryJedis?
>> Shouldn't BinaryJedis be transient? And anyway, what is this cryptic thing
>> called 'consumeBatchToCursor()' doing?
>>
>> The "offending" class can be seen below. It's just a simple bolt
>> extending AbstractRedisBolt.
>>
>> https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java
>>
>> Does anyone know what might be the root cause for this?
>>
>> Regards,
>> Gergely
>>
>
>
>
> --
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>

Mime
View raw message