kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garrett Barton <garrett.bar...@gmail.com>
Subject high level consumer usage question with auto topic creation
Date Tue, 30 Jul 2013 14:44:45 GMT
I am trying to write a junit test for working with an embedded Kafka 0.8
server where I send and receive messages just to verify my embedded server,
producer and consumer wrappers work.  Order of operation in the junit looks
something like:

-Start zk. [own thread] (wait for init)
-Start Kafka [own thread] (wait for init)
-Start consumer [own thread] (I block on my threads run method releasing a
lock once it starts). consumer is copied from:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
-Start producer (dont see anything to wait/block on, so I move along)

Now that things are online, I send 5 messages out from the producer.  There
exists no topic, so one is created auto-magically by kafka (btw, why is
there no api to create topics in java?)  I then read from the consumer and
compare how many I received against the sent ones.

What I notice is that when I run my test over and over, I will get anywhere
from 0-5 messages received.  The logs look like for some reason the client
seems to re-establish its connection when messages start being sent and
depending on how long that takes depends on how many of the messages I'll
get to read.

Is the high level consumer meant to be used this way? If a high level
consumer is started first am I not guaranteed all messages produced from
that point on? I have a single stream asked for from the topic so according
to the doc that single stream should get all messages on the topic (which
inits with 2 partitions).  Does anyone have any more insight or should I
just migrate into the simple consumer example?

~Garrett

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message