kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Montgomery <davidmontgom...@gmail.com>
Subject Re: failed with LeaderNotAvailableError -
Date Fri, 18 Dec 2015 02:17:19 GMT
So what do I do?  Kill my production servers and rebuild?  Restarting all
services does nit work.  This seems kinda extreme.  At this point I feel I
have to kill all servers and rebuild.

Thanks

On Fri, Dec 18, 2015 at 2:28 AM, Dana Powers <dana.powers@gmail.com> wrote:

> Hi Ben and Marko -- great suggestions re: connection failures and docker.
>
> The specific error here is: LeaderNotAvailableError:
> TopicMetadata(topic='topic-test-production', error=5, partitions=[])
>
> That is an error code (5) returned from a MetadataRequest. In this context
> it means that the topic did not exist and so the request triggered an
> auto-create initialization (i.e., the connection was fine). Topic
> initialization tends to take a few seconds to complete, but only needs to
> happen once per topic. A retry here is generally fine. This retry should
> probably be handled under the covers by the client code. So in this case I
> would treat it as a simple kafka-python issue (#488).
>
> -Dana
>
> On Thu, Dec 17, 2015 at 4:58 AM, Ben Davison <ben.davison@7digital.com>
> wrote:
>
> > I probably should of mentioned that this was using Amazon ECS.
> >
> > On Thu, Dec 17, 2015 at 12:18 PM, Marko Bonaći <
> marko.bonaci@sematext.com>
> > wrote:
> >
> > > It doesn't have to be FQDN.
> > >
> > > Here's how I run Kafka in a container:
> > > docker run --name st-kafka -p 2181:2181 -p 9092:9092 -e
> > > ADVERTISED_HOST=`docker-machine ip dev-st` -e ADVERTISED_PORT=9092 -d
> > > spotify/kafka
> > >
> > > And then you have access to Kafka on the docker host VM from any other
> > > machine.
> > > BTW I use Spotify's image since it contains both ZK and Kafka, but I
> > think
> > > the latest version they built is 0.8.2.1, so you might have to build
> the
> > > new image yourself if you need 0.9, but that's trivial to do.
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Thu, Dec 17, 2015 at 11:33 AM, Ben Davison <
> ben.davison@7digital.com>
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > Are you running in docker? Are you trying to connect from to a remote
> > > box?
> > > > We found we could connect locally but couldn't connect from another
> > > remote
> > > > host.
> > > >
> > > > (I've just started using kafka also)
> > > >
> > > > We had the same issue and found out: host.name=<%=@ipaddress%>
> needed
> > to
> > > > be
> > > > the FQDN of the box.
> > > >
> > > > Thanks,
> > > >
> > > > Ben
> > > >
> > > > On Thu, Dec 17, 2015 at 5:40 AM, David Montgomery <
> > > > davidmontgomery@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am very concerned about using kafka in production given the below
> > > > > errors:
> > > > >
> > > > > Now issues with myt zookeeper.  Other services use ZK.  Only kafka
> > > fails.
> > > > > I have 2 kafka servers using 8.x.  How do I resolve?  I tried
> > > restarting
> > > > > services for kafka.  Below is my kafka server.properties file
> > > > >
> > > > > 'Traceback (most recent call last):
> > > > >   File
> > > > >
> > > > >
> > > >
> > >
> >
> "/usr/local/lib/python2.7/dist-packages/gevent-1.1b6-py2.7-linux-x86_64.egg/gevent/greenlet.py",
> > > > > line 523, in run
> > > > >     result = self._run(*self.args, **self.kwargs)
> > > > >   File "/var/feed-server/ad-server/pixel-server.py", line 145, in
> > > > > send_kafka_message
> > > > >     res = producer.send_messages(topic, message)
> > > > >   File "build/bdist.linux-x86_64/egg/kafka/producer/simple.py",
> line
> > > 52,
> > > > in
> > > > > send_messages
> > > > >     partition = self._next_partition(topic)
> > > > >   File "build/bdist.linux-x86_64/egg/kafka/producer/simple.py",
> line
> > > 36,
> > > > in
> > > > > _next_partition
> > > > >     self.client.load_metadata_for_topics(topic)
> > > > >   File "build/bdist.linux-x86_64/egg/kafka/client.py", line 383,
in
> > > > > load_metadata_for_topics
> > > > >     kafka.common.check_error(topic_metadata)
> > > > >   File "build/bdist.linux-x86_64/egg/kafka/common.py", line 233,
in
> > > > > check_error
> > > > >     raise error_class(response)
> > > > > LeaderNotAvailableError:
> TopicMetadata(topic='topic-test-production',
> > > > > error=5, partitions=[])
> > > > > <Greenlet at 0x7f7acd1654b0:
> > > send_kafka_message('topic-test-production',
> > > > > '{"adfadfadf)> failed with LeaderNotAvailableError
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > # limitations under the License.
> > > > > # see kafka.server.KafkaConfig for additional details and defaults
> > > > >
> > > > > ############################# Server Basics
> > > #############################
> > > > >
> > > > > # The id of the broker. This must be set to a unique integer for
> each
> > > > > broker.
> > > > > broker.id=<%=@broker_id%>
> > > > > advertised.host.name=<%=@ipaddress%>
> > > > > advertised.port=9092
> > > > > ############################# Socket Server Settings
> > > > > #############################
> > > > >
> > > > > # The port the socket server listens on
> > > > > port=9092
> > > > >
> > > > > # Hostname the broker will bind to and advertise to producers and
> > > > > consumers.
> > > > > # If not set, the server will bind to all interfaces and advertise
> > the
> > > > > value returned from
> > > > > # from java.net.InetAddress.getCanonicalHostName().
> > > > > host.name=<%=@ipaddress%>
> > > > >
> > > > > # The number of threads handling network requests
> > > > > num.network.threads=2
> > > > >
> > > > > # The number of threads doing disk I/O
> > > > > num.io.threads=2
> > > > >
> > > > > # The send buffer (SO_SNDBUF) used by the socket server
> > > > > socket.send.buffer.bytes=1048576
> > > > >
> > > > > # The receive buffer (SO_RCVBUF) used by the socket server
> > > > > socket.receive.buffer.bytes=1048576
> > > > >
> > > > > # The maximum size of a request that the socket server will accept
> > > > > (protection against OOM)
> > > > > socket.request.max.bytes=104857600
> > > > >
> > > > >
> > > > > ############################# Log Basics
> > #############################
> > > > >
> > > > > # A comma seperated list of directories under which to store log
> > files
> > > > > log.dirs=/tmp/kafka-logs
> > > > >
> > > > > # The number of logical partitions per topic per server. More
> > > partitions
> > > > > allow greater parallelism
> > > > > # for consumption, but also mean more files.
> > > > > num.partitions=2
> > > > >
> > > > > ############################# Log Flush Policy
> > > > > #############################
> > > > >
> > > > > # The following configurations control the flush of data to disk.
> > This
> > > is
> > > > > among the most
> > > > > # important performance knob in kafka.
> > > > > # There are a few important trade-offs here:
> > > > > #    1. Durability: Unflushed data may be lost if you are not using
> > > > > replication.
> > > > > #    2. Latency: Very large flush intervals may lead to latency
> > spikes
> > > > when
> > > > > the flush does occur as there will be a lot of data to flush.
> > > > > #    3. Throughput: The flush is generally the most expensive
> > > operation,
> > > > > and a small flush interval may lead to exceessive seeks.
> > > > > # The settings below allow one to configure the flush policy to
> flush
> > > > data
> > > > > after a period of time or
> > > > > # every N messages (or both). This can be done globally and
> > overridden
> > > > on a
> > > > > per-topic basis.
> > > > >
> > > > > # The number of messages to accept before forcing a flush of data
> to
> > > disk
> > > > > log.flush.interval.messages=10000
> > > > >
> > > > > # The maximum amount of time a message can sit in a log before we
> > > force a
> > > > > flush
> > > > > log.flush.interval.ms=1000
> > > > >
> > > > > # Per-topic overrides for log.flush.interval.ms
> > > > > #log.flush.intervals.ms.per.topic=topic1:1000, topic2:3000
> > > > >
> > > > > ############################# Log Retention Policy
> > > > > #############################
> > > > >
> > > > > # The following configurations control the disposal of log
> segments.
> > > The
> > > > > policy can
> > > > > # be set to delete segments after a period of time, or after a
> given
> > > size
> > > > > has accumulated.
> > > > > # A segment will be deleted whenever *either* of these criteria are
> > > met.
> > > > > Deletion always happens
> > > > > # from the end of the log.
> > > > >
> > > > > # The minimum age of a log file to be eligible for deletion
> > > > > log.retention.hours=168
> > > > >
> > > > > # A size-based retention policy for logs. Segments are pruned from
> > the
> > > > log
> > > > > as long as the remaining
> > > > > # segments don't drop below log.retention.bytes.
> > > > > #log.retention.bytes=1073741824
> > > > >
> > > > > # The maximum size of a log segment file. When this size is
> reached a
> > > new
> > > > > log segment will be created.
> > > > > log.segment.bytes=536870912
> > > > >
> > > > > # The interval at which log segments are checked to see if they can
> > be
> > > > > deleted according
> > > > > # to the retention policies
> > > > > log.cleanup.interval.mins=1
> > > > >
> > > > > ############################# Zookeeper
> #############################
> > > > >
> > > > > # Zookeeper connection string (see zookeeper docs for details).
> > > > > # This is a comma separated host:port pairs, each corresponding to
> a
> > zk
> > > > > # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
> > > > > # You can also append an optional chroot string to the urls to
> > specify
> > > > the
> > > > > # root directory for all kafka znodes.
> > > > > #zookeeper.connect=localhost:2181
> > > > > zookeeper.connect=<%=@zookeeper%>
> > > > >
> > > > >
> > > > > # Timeout in ms for connecting to zookeeper
> > > > > zookeeper.connection.timeout.ms=1000000
> > > > >
> > > > > num.replica.fetchers=4
> > > > > default.replication.factor=2
> > > > > delete.topic.enable=true
> > > > > unclean.leader.election.enable=true
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > > > This email, including attachments, is private and confidential. If
> you
> > > have
> > > > received this email in error please notify the sender and delete it
> > from
> > > > your system. Emails are not secure and may contain viruses. No
> > liability
> > > > can be accepted for viruses that might be transferred by this email
> or
> > > any
> > > > attachment. Any unauthorised copying of this message or unauthorised
> > > > distribution and publication of the information contained herein are
> > > > prohibited.
> > > >
> > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A
> 2BB.
> > > > Registered in England and Wales. Registered No. 04843573.
> > > >
> > >
> >
> > --
> >
> >
> > This email, including attachments, is private and confidential. If you
> have
> > received this email in error please notify the sender and delete it from
> > your system. Emails are not secure and may contain viruses. No liability
> > can be accepted for viruses that might be transferred by this email or
> any
> > attachment. Any unauthorised copying of this message or unauthorised
> > distribution and publication of the information contained herein are
> > prohibited.
> >
> > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > Registered in England and Wales. Registered No. 04843573.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message