kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cosmin Marginean <cosmargin...@gmail.com>
Subject Re: Leader not available (initially)
Date Fri, 08 Jan 2016 17:28:34 GMT
Hi yes, was actually just writing a reply to that :)

Found the rest of the context: "Alternatively, instead of manually creating topics you can
also configure your brokers to auto-create topics when a non-existent topic is published to.”

I tried this: commented out my code that creates the topic and just throw something at the
publisher. So basically:
1. Publisher.send
2. Consumer.register

The warning is still there (as per Dana’s comment, is expected), but no messages are being
consumed.

However, this is less material, as it seems to be a *different* problem altogether.

My concern now is what happens if the topic is freshly created (manually or not) and hundreds
of messages are being published (and that warning is being displayed).

I also saw things like "A side note is that you may want to use waitUntil to check the topic
is
created after using AdminUtils.createTopic since it is async.” (from here http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCAHwHRrUUAuazRtjdzr6geeG6D0hhPeneAU412N-C7Xk3UwLbGA@mail.gmail.com%3E)

but I can’t find anything like waitUntil in that API

Thank you again for your support
Cos



On Friday, 8 January 2016 at 17:19, Marko Bonaći wrote:

> I think that the attempt to write a message to a non-existent topic creates
> that topic (when auto.create is set to true).
> If it's set to false you get back error.
> Have you tried that?
>  
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>  
> On Fri, Jan 8, 2016 at 5:51 PM, Cosmin Marginean <cosmarginean@gmail.com (mailto:cosmarginean@gmail.com)>
> wrote:
>  
> > Hi Dana
> >  
> > I am actually creating the topic manually from our code. This is mainly
> > because I couldn’t tell from the documentation when exactly is the topic
> > auto-created - maybe you can shed some light on that.
> >  
> > The docs (http://kafka.apache.org/documentation.html) only mention
> > auto.create.topics.enable Enable auto creation of topic on the server
> > which is a bit without context.
> >  
> > Either way (automated or not), it is crucial for us to know if the message
> > actually gets sent. The problem is we need to get the producer to send some
> > messages as soon as the system bootstraps (and potentially creates the
> > topic). So is it possible to know (listener, etc) on the producer side when
> > the topic is “operational”, or at least to handle a specific exception?
> >  
> > Or is it a case where the producer will buffer these on the client side?
> >  
> > Thanks
> > Cos
> >  
> >  
> >  
> > On Friday, 8 January 2016 at 16:45, Dana Powers wrote:
> >  
> > > That is expected when a topic is first created. Once the topic and all
> > > partitions are initialized, the leader(s) will be available and you can
> > > produce messages.
> > >  
> > > Note that if you disable topic auto creation, you should get an
> > > UnknownTopicOrPartitionError, and you would need to create the topic
> > > manually.
> > >  
> > > -Dana
> > > On Jan 8, 2016 03:20, "Cosmin Marginean" <cosmarginean@gmail.com (mailto:cosmarginean@gmail.com)
> > >  
> >  
> > (mailto:cosmarginean@gmail.com)> wrote:
> > >  
> > > > Hi guys
> > > >  
> > > > I’ve been seeing this in our logs a few times, when initially
> > > > bootstrapping the system.
> > > >  
> > > > WARN 2016-01-08 10:59:24,232 [pool-3-thread-11] [none]
> > > > o.a.kafka.clients.NetworkClient: Error while fetching metadata with
> > > > correlation id 3 : {mytopic=LEADER_NOT_AVAILABLE}
> > > >  
> > > > It doesn’t seem to break anything, and it seems to be related to the
> > fact
> > > > that we do processing on the topic right after creating it. Below is
> > >  
> >  
> > our
> > > > workflow (roughly)
> > > > * Create topic (using AdminUtils)
> > > > * Register some tens of consumers
> > > > * Send some messages with a producer
> > > >  
> > > > I’d like to know if this is something to worry about (the warning that
> > is)
> > > > or if it’s something we should investigate further.
> > > >  
> > > > Thank you
> > > > Cosmin
> > > >  
> > >  
> >  
> >  
>  
>  
>  



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message