@John - 
can you file a JIRA for this? I doubt it is related to 1.0.0 version in particular. I have run into "illegalArugmentExceptions" in KafkaSpout (0.10.0). 

On Wed, Apr 20, 2016 at 4:44 PM, John Yost <hokiegeek2@gmail.com> wrote:
Also, I found this link that indicates the exception I reported yesterday can be symptomatic of a mismatch between the client and broker where the client is one version newer.  I am not saying that's the case here with Storm 1.0.0, but wanted to provide this info troubleshooting-wise.

Thanks

--John

On Tue, Apr 19, 2016 at 3:26 PM, John Yost <hokiegeek2@gmail.com> wrote:
Hi Harsha,

When the Storm 1.0.0 KafkaSpout (from the storm-kafka jar) attempts to read from the Kafka 0.8.2.1 partition an IlegalArgumentException is thrown, the root exception of which is as follows:

at java.nio.Buffer.limit(Buffer.java:267)
at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:37)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:99)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:97)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.api.TopicData$.readFrom(FetchResponse.scala:97)
at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:168)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)

The corresponding source code in Kafka where the root exception is thrown is bolded:

object FetchResponsePartitionData {
  def readFrom(buffer: ByteBuffer): FetchResponsePartitionData = {
    val error = buffer.getShort
    val hw = buffer.getLong
    val messageSetSize = buffer.getInt
    val messageSetBuffer = buffer.slice()
    messageSetBuffer.limit(messageSetSize)
    buffer.position(buffer.position + messageSetSize)
    new FetchResponsePartitionData(error, hw, new ByteBufferMessageSet(messageSetBuffer))
  }

I am using all the default KafkaConfig settings for the KafkaSpout with the exception of startOffsetTime, so I don't *think* I have a misconfiguration, but I may be wrong.

Please confirm if there is anything I need to do config-wise to make this work.

Thanks

--John

On Sat, Apr 16, 2016 at 10:49 PM, <hokiegeek2@gmail.com> wrote:
Awesome, thanks Harsha!

--John

Sent from my iPhone

> On Apr 16, 2016, at 1:28 PM, Harsha <mail@harsha.io> wrote:
>
> John,
>             I think you are asking if you will be able to run 0.8.2 kafka consumer in storm 1.0.0 . Yes we are shipping storm-kafka-client which uses the new consumer api in kafka 0.9.0.1 but storm 1.0.0 still ships with storm-kafka which uses older consumer api which can work with all versions of kafka including 0.9.0.1.
>
> "I checked out the v1.0.0 tag, changed the kafka version to 0.8.2.1, and I am getting compile errors in storm-kafka-client. I don't have a problem fixing the errors, but I want to ensure I am not wasting my time. :)"
>
> You shouldn't be changing kafka version. As I said above Kafka 0.9.0.1 contains two kafka apis new ones which will only works with 0.9.0.1 kafka cluster and old consumer apis which can work 0.8.2. Even though you compile with 0.9.0.1 version it will work with 0.8.2.1 kafka cluster.
>
> Let me know if you have any questions.
>
> Thanks,
> Harsha
>
>> On Fri, Apr 15, 2016, at 11:39 AM, John Yost wrote:
>> Hi Everyone,
>> I know that Storm 1.0.0 is designed to work with Kafka 0.9.0.1, but is it possible for Storm 1.0.0 to work with 0.8.2.1?
>>
>> I did some integration testing with 0.9.0.1 client with 0.8.2.1 broker, which is yucky, and it appears this won't work.
>>
>> I checked out the v1.0.0 tag, changed the kafka version to 0.8.2.1, and I am getting compile errors in storm-kafka-client. I don't have a problem fixing the errors, but I want to ensure I am not wasting my time. :)
>>
>> Please confirm if it is anticipated that Storm 1.0.0 can be made compatible with Kafka 0.8.2.x--thanks!
>> --John
>





--
Regards,
Abhishek Agarwal