kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Peterson <d...@academia.edu>
Subject Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message
Date Mon, 25 Jan 2016 04:41:46 GMT
On Wed, Jan 20, 2016 at 11:43 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> Yes - compaction is a topic-level property. You can use --describe to
> verify that the topic is compacted or not and if you didn't intend it to be
> compacted you can alter the configuration.
>

I tried using the --describe option with kafka-topics.sh and didn't
see any information on which topics are compacted.  Do all brokers
need to be upgraded to 0.9.0.0 for this information to appear?
I was trying it with only one broker upgraded, and the others
were still running 0.8.2.1.  Also the 0.9.0.0 broker had
inter.broker.protocol.version=0.8.2.X set in its server.properties
file.  Also what is the command syntax for altering the config?

In 0.9 you cannot send unkeyed messages to compacted topics. In 0.8.x this
> would actually cause the log compaction thread to subsequently complain and
> quit (and stop compacting all compacted topics). We did consider the
> possibility of allowing producers to send both keyed and unkeyed but after
> discussion we felt it would be better to fail fast and prevent unkeyed
> messages from getting in. This was on the premise that supporting mixed
> messages and only compacting a subset that have keys may not work very well
> since the non-keyed messages would stick around indefinitely; however let
> me know if you think differently on this and we can revisit.


Ok, thanks for the information.  Letting compaction be a property
of the topic totally makes sense to me.  I can imagine it would
simplify the design.

The protocol documentation describes error 2 as indicating that
the message content does not match its CRC, which is a bit
confusing.  In the case of invalid CRC, a natural response would
be to resend.  For instance, Bruce increments a failed delivery
count on the message, discards it if the count exceeds a
configurable threshold, or otherwise resends it.  If error 2 is
received because an unkeyed message was sent to a
compacted topic, then the preferred action would be to
immediately discard.  It would be helpful to avoid the ambiguity
by using a distinct error code for the latter case.

To further assist producers, it might also make sense to include
compaction info in metadata responses.  This would allow the
producer to drop a message before even attempting to send it
in the case where its compaction status is wrong for the topic.
However I understand that this would involve substantially more
work, including incrementing the API version for metadata
requests and responses, and therefore may not be worth the
effort.  If one were to make such a protocol change, it could be
done by adding a "flags" bitfield to the following production,

    TopicMetadata => TopicErrorCode TopicName [PartitionMetadata]

choosing a bit to indicate compaction status, and reserving the
remaining bits for future use.

Out of curiosity, how does a broker react on receipt of a keyed
message for a noncompacted topic?  I would expect the
behavior to be the same as for the opposite case.

Thanks,
Dave

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message