storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Yost <hokiege...@gmail.com>
Subject Re: Is Storm 1.0.0 compatible with Kafka 0.8.2.x?
Date Tue, 03 May 2016 12:41:02 GMT
Hey Abhishek,

Cool, yeah, switched my kafka dependency (my topo reads and writes to
Kafka) to 0.8.2.1 and everything deploys correctly. The next issue I am
encountering is no data is flowing through the KafkaSpout.  Do you have
data flowing in your topo?

--John

On Mon, May 2, 2016 at 7:17 AM, Abhishek Agarwal <abhishcool@gmail.com>
wrote:

> good thing is that storm-kafka artifact itself is compatible with kafka
> 0.8.2.1. So in your maven project, you can simply exclude the
> org.apache.kafka dependencies coming out of storm-kafka project and use the
> version you need.
>
> On Mon, May 2, 2016 at 3:03 PM, Abhishek Agarwal <abhishcool@gmail.com>
> wrote:
>
>> John, I think you have hit it right. I started using storm-kafka 1.0 with
>> kafka server 0.8.2.1 and started running into all sorts of issues including
>> the one you pointed out. Also 0.9 documentation clearly states that
>> upgraded clients will not be compatible with older kafka version.
>> http://kafka.apache.org/090/documentation.html#upgrade
>>
>> On Wed, Apr 20, 2016 at 5:42 PM, John Yost <hokiegeek2@gmail.com> wrote:
>>
>>> Argh, fat fingers...I am attempting to write to Kafka 0.8.2.1 from Storm
>>> 1.0.0 which is has a dependency upon Kafka 0.9.0.1.
>>>
>>> @Abhishek -> interesting you are seeing the same exception for Storm
>>> 0.10.0 because that has a dependency upon Kafka 0.8.1.1.
>>>
>>> On Wed, Apr 20, 2016 at 8:06 AM, John Yost <hokiegeek2@gmail.com> wrote:
>>>
>>>> Oh, gotcha, okay, will do. BTW, here's the link I failed to provide the
>>>> first time: https://github.com/confluentinc/examples/issues/15
>>>>
>>>> --John
>>>>
>>>> On Wed, Apr 20, 2016 at 7:44 AM, Abhishek Agarwal <abhishcool@gmail.com
>>>> > wrote:
>>>>
>>>>> @John -
>>>>> can you file a JIRA for this? I doubt it is related to 1.0.0 version
>>>>> in particular. I have run into "illegalArugmentExceptions" in KafkaSpout
>>>>> (0.10.0).
>>>>>
>>>>> On Wed, Apr 20, 2016 at 4:44 PM, John Yost <hokiegeek2@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Also, I found this link that indicates the exception I reported
>>>>>> yesterday can be symptomatic of a mismatch between the client and
broker
>>>>>> where the client is one version newer.  I am not saying that's the
case
>>>>>> here with Storm 1.0.0, but wanted to provide this info troubleshooting-wise.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> --John
>>>>>>
>>>>>> On Tue, Apr 19, 2016 at 3:26 PM, John Yost <hokiegeek2@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Harsha,
>>>>>>>
>>>>>>> When the Storm 1.0.0 KafkaSpout (from the storm-kafka jar) attempts
>>>>>>> to read from the Kafka 0.8.2.1 partition an IlegalArgumentException
is
>>>>>>> thrown, the root exception of which is as follows:
>>>>>>>
>>>>>>> at java.nio.Buffer.limit(Buffer.java:267)
>>>>>>> at
>>>>>>> kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:37)
>>>>>>> at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:99)
>>>>>>> at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:97)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>>>>>>> at scala.collection.immutable.Range.foreach(Range.scala:141)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>>>>>>> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>>>>>>> at kafka.api.TopicData$.readFrom(FetchResponse.scala:97)
>>>>>>> at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
>>>>>>> at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:168)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
>>>>>>> at scala.collection.immutable.Range.foreach(Range.scala:141)
>>>>>>> at
>>>>>>> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
>>>>>>> at
>>>>>>> scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
>>>>>>>
>>>>>>> The corresponding source code in Kafka where the root exception
is
>>>>>>> thrown is bolded:
>>>>>>>
>>>>>>> object FetchResponsePartitionData {
>>>>>>>   def readFrom(buffer: ByteBuffer): FetchResponsePartitionData
= {
>>>>>>>     val error = buffer.getShort
>>>>>>>     val hw = buffer.getLong
>>>>>>>     val messageSetSize = buffer.getInt
>>>>>>>     val messageSetBuffer = buffer.slice()
>>>>>>>  *   messageSetBuffer.limit(messageSetSize)*
>>>>>>>     buffer.position(buffer.position + messageSetSize)
>>>>>>>     new FetchResponsePartitionData(error, hw, new
>>>>>>> ByteBufferMessageSet(messageSetBuffer))
>>>>>>>   }
>>>>>>>
>>>>>>> I am using all the default KafkaConfig settings for the KafkaSpout
>>>>>>> with the exception of startOffsetTime, so I don't *think* I have
a
>>>>>>> misconfiguration, but I may be wrong.
>>>>>>>
>>>>>>> Please confirm if there is anything I need to do config-wise
to make
>>>>>>> this work.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> --John
>>>>>>>
>>>>>>> On Sat, Apr 16, 2016 at 10:49 PM, <hokiegeek2@gmail.com>
wrote:
>>>>>>>
>>>>>>>> Awesome, thanks Harsha!
>>>>>>>>
>>>>>>>> --John
>>>>>>>>
>>>>>>>> Sent from my iPhone
>>>>>>>>
>>>>>>>> > On Apr 16, 2016, at 1:28 PM, Harsha <mail@harsha.io>
wrote:
>>>>>>>> >
>>>>>>>> > John,
>>>>>>>> >             I think you are asking if you will be able
to run
>>>>>>>> 0.8.2 kafka consumer in storm 1.0.0 . Yes we are shipping
>>>>>>>> storm-kafka-client which uses the new consumer api in kafka
0.9.0.1 but
>>>>>>>> storm 1.0.0 still ships with storm-kafka which uses older
consumer api
>>>>>>>> which can work with all versions of kafka including 0.9.0.1.
>>>>>>>> >
>>>>>>>> > "I checked out the v1.0.0 tag, changed the kafka version
to
>>>>>>>> 0.8.2.1, and I am getting compile errors in storm-kafka-client.
I don't
>>>>>>>> have a problem fixing the errors, but I want to ensure I
am not wasting my
>>>>>>>> time. :)"
>>>>>>>> >
>>>>>>>> > You shouldn't be changing kafka version. As I said above
Kafka
>>>>>>>> 0.9.0.1 contains two kafka apis new ones which will only
works with 0.9.0.1
>>>>>>>> kafka cluster and old consumer apis which can work 0.8.2.
Even though you
>>>>>>>> compile with 0.9.0.1 version it will work with 0.8.2.1 kafka
cluster.
>>>>>>>> >
>>>>>>>> > Let me know if you have any questions.
>>>>>>>> >
>>>>>>>> > Thanks,
>>>>>>>> > Harsha
>>>>>>>> >
>>>>>>>> >> On Fri, Apr 15, 2016, at 11:39 AM, John Yost wrote:
>>>>>>>> >> Hi Everyone,
>>>>>>>> >> I know that Storm 1.0.0 is designed to work with
Kafka 0.9.0.1,
>>>>>>>> but is it possible for Storm 1.0.0 to work with 0.8.2.1?
>>>>>>>> >>
>>>>>>>> >> I did some integration testing with 0.9.0.1 client
with 0.8.2.1
>>>>>>>> broker, which is yucky, and it appears this won't work.
>>>>>>>> >>
>>>>>>>> >> I checked out the v1.0.0 tag, changed the kafka
version to
>>>>>>>> 0.8.2.1, and I am getting compile errors in storm-kafka-client.
I don't
>>>>>>>> have a problem fixing the errors, but I want to ensure I
am not wasting my
>>>>>>>> time. :)
>>>>>>>> >>
>>>>>>>> >> Please confirm if it is anticipated that Storm 1.0.0
can be made
>>>>>>>> compatible with Kafka 0.8.2.x--thanks!
>>>>>>>> >> --John
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Abhishek Agarwal
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Abhishek Agarwal
>>
>>
>
>
> --
> Regards,
> Abhishek Agarwal
>
>

Mime
View raw message