kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Bonaći <marko.bon...@sematext.com>
Subject Re: Leader not available (initially)
Date Fri, 08 Jan 2016 17:19:24 GMT
I think that the attempt to write a message to a non-existent topic creates
that topic (when auto.create is set to true).
If it's set to false you get back error.
Have you tried that?

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Fri, Jan 8, 2016 at 5:51 PM, Cosmin Marginean <cosmarginean@gmail.com>
wrote:

> Hi Dana
>
> I am actually creating the topic manually from our code. This is mainly
> because I couldn’t tell from the documentation when exactly is the topic
> auto-created - maybe you can shed some light on that.
>
> The docs (http://kafka.apache.org/documentation.html) only mention
> auto.create.topics.enable Enable auto creation of topic on the server
> which is a bit without context.
>
> Either way (automated or not), it is crucial for us to know if the message
> actually gets sent. The problem is we need to get the producer to send some
> messages as soon as the system bootstraps (and potentially creates the
> topic). So is it possible to know (listener, etc) on the producer side when
> the topic is “operational”, or at least to handle a specific exception?
>
> Or is it a case where the producer will buffer these on the client side?
>
> Thanks
> Cos
>
>
>
> On Friday, 8 January 2016 at 16:45, Dana Powers wrote:
>
> > That is expected when a topic is first created. Once the topic and all
> > partitions are initialized, the leader(s) will be available and you can
> > produce messages.
> >
> > Note that if you disable topic auto creation, you should get an
> > UnknownTopicOrPartitionError, and you would need to create the topic
> > manually.
> >
> > -Dana
> > On Jan 8, 2016 03:20, "Cosmin Marginean" <cosmarginean@gmail.com
> (mailto:cosmarginean@gmail.com)> wrote:
> >
> > > Hi guys
> > >
> > > I’ve been seeing this in our logs a few times, when initially
> > > bootstrapping the system.
> > >
> > > WARN 2016-01-08 10:59:24,232 [pool-3-thread-11] [none]
> > > o.a.kafka.clients.NetworkClient: Error while fetching metadata with
> > > correlation id 3 : {mytopic=LEADER_NOT_AVAILABLE}
> > >
> > > It doesn’t seem to break anything, and it seems to be related to the
> fact
> > > that we do processing on the topic right after creating it. Below is
> our
> > > workflow (roughly)
> > > * Create topic (using AdminUtils)
> > > * Register some tens of consumers
> > > * Send some messages with a producer
> > >
> > > I’d like to know if this is something to worry about (the warning that
> is)
> > > or if it’s something we should investigate further.
> > >
> > > Thank you
> > > Cosmin
> > >
> >
> >
> >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message