kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Heath Ivie <hi...@AutoAnything.com>
Subject RE: Protocol Question
Date Thu, 04 Feb 2016 16:19:23 GMT
Dana, Again thanks for the feedback.

I will try that today and see what happens.

Is there anything special to start the loop?

What I mean is there any buffer byte to signify the start of the arrays?

-----Original Message-----
From: Dana Powers [mailto:dana.powers@gmail.com] 
Sent: Wednesday, February 03, 2016 8:33 PM
To: users@kafka.apache.org
Subject: Re: Protocol Question

Hi Heath, a few comments:

(1) you should be looping on brokerCount
(2) you are missing the topic array count (4 bytes), and loop
(3) topic error code is int16, so only 2 bytes not 4
(4) you are missing the partition metadata array count (4 bytes), and loop
(5) you are missing the replicas and isr parsing.

-Dana

On Wed, Feb 3, 2016 at 6:01 PM, Heath Ivie <hivie@autoanything.com> wrote:

> Hi,
>
> I am trying to navigate through the protocol and I am seeing some 
> inconsistencies with the data.
>
> I am trying to parse out the MetadataResponse and I am seeing bytes in 
> between where they shouldn't be.
>
> I know they are extra, because if I up the offset the data after is 
> correct.
>
> Here is the part of the response that I am trying to parse:
> MetadataResponse => [Broker][TopicMetadata]
>   Broker => NodeId Host Port  (any number of brokers may be returned)
>     NodeId => int32
>     Host => string
>     Port => int32
>   TopicMetadata => TopicErrorCode TopicName [PartitionMetadata]
>     TopicErrorCode => int16
>   PartitionMetadata => PartitionErrorCode PartitionId Leader Replicas Isr
>     PartitionErrorCode => int16
>     PartitionId => int32
>     Leader => int32
>     Replicas => [int32]
>     Isr => [int32]
>
>
> I am seeing extra bytes after the topic error.
>
> I am not sure if this padding is expected or not?
>
> It may not be very readable, but here is my code (be nice :)).
>
> int offset =0;
>                     int correlationId = 
> BitConverter.ToInt32(ReverseBytes(bytes.Skip(offset).Take(4).ToArray()), 0);
>                     offset += 4;
>                     int brokerCount =
> BitConverter.ToInt32(ReverseBytes(bytes.Skip(offset).Take(4).ToArray()), 0);
>                     offset += 4;
>                     int nodeId =
> BitConverter.ToInt32(ReverseBytes(bytes.Skip(offset).Take(4).ToArray()), 0);
>                     offset += 4;
>                     int hostNameLength = 
> BitConverter.ToInt16(ReverseBytes(bytes.Skip(offset).Take(2).ToArray()), 0);
>                     offset += 2;
>
>                     string hostName = Encoding.ASCII.GetString(bytes, 
> offset, hostNameLength);
>                     offset += hostNameLength;
>
>                     int port =
> BitConverter.ToInt32(ReverseBytes(bytes.Skip(offset).Take(4).ToArray()), 0);
>                     offset += 4;
>
>                     int topicErrorCode = 
> BitConverter.ToInt32(ReverseBytes(bytes.Skip(offset).Take(4).ToArray()), 0);
>                     offset += 4;
>
>                     int topicNameLength = 
> BitConverter.ToInt16(ReverseBytes(bytes.Skip(offset).Take(2).ToArray()), 0);
>                     offset += 2;
>
>                     string topicName = Encoding.ASCII.GetString(bytes, 
> offset, topicNameLength);
>                     offset += topicNameLength;
>
>                     int partitionErrorCode = 
> BitConverter.ToInt16(ReverseBytes(bytes.Skip(offset).Take(2).ToArray()), 0);
>                     offset += 2;
>
>                     int partitionId =
> BitConverter.ToInt32(ReverseBytes(bytes.Skip(offset).Take(4).ToArray()), 0);
>                     offset += 4;
>
>                     int leaderId =
> BitConverter.ToInt32(ReverseBytes(bytes.Skip(offset).Take(4).ToArray()), 0);
>                     offset += 4;
>
>
> Any help would be cool, thanks
>
> Heath Ivie
> Solutions Architect
>
>
> Warning: This e-mail may contain information proprietary to 
> AutoAnything Inc. and is intended only for the use of the intended 
> recipient(s). If the reader of this message is not the intended 
> recipient(s), you have received this message in error and any review, 
> dissemination, distribution or copying of this message is strictly 
> prohibited. If you have received this message in error, please notify 
> the sender immediately and delete all copies.
>
Mime
View raw message