kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinay sharma <vinsharma.t...@gmail.com>
Subject Re: Metadata Request Loop?
Date Thu, 28 Apr 2016 20:10:37 GMT
Can you try using 0.9.0.1 for both client and broker? I tried replicating
the issue with both on 0.9.0.1 and i could not. I also tried what Phil has
suggested but in my case i see only one meta data refresh. I have seen
issues like this only for a short time when leader went down. Continuous
occurrence of this issue i have seen only once where leader went down and
probably there was no ISR.

Vinay

On Thu, Apr 28, 2016 at 9:06 AM, Phil Luckhurst <phil.luckhurst@encycle.com>
wrote:

> I see from your debug messages it's the kafka-producer-network-thread that
> is doing this so you might be hitting the same issue as I had in
> https://www.mail-archive.com/users%40kafka.apache.org/msg18562.html.
>
> I was creating a producer when my application started but if it didn't
> publish a message within the first 5 minutes then I also saw a metadata
> request every 100ms.
>
> Regards,
> Phil Luckhurst
>
> -----Original Message-----
> From: Fumo, Vincent [mailto:Vincent_Fumo@cable.comcast.com]
> Sent: 27 April 2016 20:48
> To: users@kafka.apache.org
> Subject: Re: Metadata Request Loop?
>
> client/producer : 0.9.0.1
> server/broker : 0.9.0.0
>
> > On Apr 26, 2016, at 10:05 PM, Sharma Vinay <vinaysharmahk@gmail.com>
> wrote:
> >
> > What versions of kafka and client API are you using now?
> > On Apr 26, 2016 3:35 PM, "Fumo, Vincent"
> > <Vincent_Fumo@cable.comcast.com>
> > wrote:
> >
> >> I spoke too soon. It's back doing the same thing.. which is really odd.
> >>
> >>> On Apr 26, 2016, at 12:24 PM, Fumo, Vincent <
> >> Vincent_Fumo@cable.comcast.com> wrote:
> >>>
> >>> Hello. I found the issue. The Ops team deployed kafka 0.8.1 and all
> >>> my
> >> code was 0.9.0. Simple mistake and one that I should have thought of
> >> sooner. Once I had them bump up to the latest kafka all was well.
> >> Thank you for your help!
> >>>
> >>> v
> >>>
> >>>
> >>>> On Apr 25, 2016, at 2:54 PM, vinay sharma
> >>>> <vinsharma.tech@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> Your code looks good. i don't see any reason there for frequent
> >>>> meta
> >> data
> >>>> fetch but i will run it to verify.
> >>>>
> >>>> I was able to replicate this issue with my consumer today where I
> >> killed 2
> >>>> brokers out or 3 and ran consumer. I saw a lot of meta data
> >>>> requests in wait for new leader for partitions for which those 2
> >>>> brokers were
> >> leader. i
> >>>> see below 2 lines frequently as you mentioned in logs
> >>>>
> >>>> [Test-KafkaPoller-1] 04/25/16 13:12:28 DEBUG Metadata:172 - Updated
> >> cluster
> >>>> metadata version 275 to Cluster(nodes = [Node(1, node1.example.com,
> >> 9093)],
> >>>> partitions = [Partition(topic = kafkaPOCTopic, partition = 1,
> >>>> leader =
> >> 1,
> >>>> replicas = [1,], isr = [1,], Partition(topic = kafkaPOCTopic,
> >>>> partition
> >> =
> >>>> 2, leader = none, replicas = [], isr = [], Partition(topic =
> >> kafkaPOCTopic,
> >>>> partition = 0, leader = none, replicas = [], isr = []])
> >>>>
> >>>> [Test-KafkaPoller-1] 04/25/16 13:12:28 DEBUG Fetcher:453 - Leader
> >>>> for partition kafkaPOCTopic-0 unavailable for fetching offset, wait
> >>>> for metadata refresh
> >>>>
> >>>> I also saw this behavior due to below error where i might have
> >>>> killed
> >> the
> >>>> only ISR available for the topic at that time.
> >>>>
> >>>> "Error while fetching metadata with correlation id 242 :
> >>>> {kafkaPOCTopic=LEADER_NOT_AVAILABLE}"
> >>>>
> >>>> Do you see any such error in your logs?
> >>>>
> >>>> Regards,
> >>>> Vinay Sharma
> >>>>
> >>>>
> >>>> On Mon, Apr 25, 2016 at 9:38 AM, Fumo, Vincent <
> >>>> Vincent_Fumo@cable.comcast.com> wrote:
> >>>>
> >>>>>
> >>>>>
> >>>>> My code is very straightforward. I create a producer, and then
> >>>>> call it
> >> to
> >>>>> send messages. Here is the factory method::
> >>>>>
> >>>>> public Producer<String, String> createProducer() {
> >>>>>
> >>>>>  Properties props = new Properties();
> >>>>>
> >>>>>  props.put("bootstrap.servers", "cmp-arch-kafka-01d.cc.com:9092");
> >>>>>  props.put("key.serializer",
> >>>>> "org.apache.kafka.common.serialization.StringSerializer");
> >>>>>  props.put("value.serializer",
> >>>>> "org.apache.kafka.common.serialization.StringSerializer");
> >>>>>  props.put("acks", "all");
> >>>>>  props.put("retries", "0");
> >>>>>  props.put("batch.size", "1638");
> >>>>>  props.put("linger.ms", "1");
> >>>>>  props.put("buffer.memory", "33554432");
> >>>>> props.put("compression.type", "gzip");  props.put("client.id",
> >>>>> "sds-merdevl");
> >>>>>
> >>>>>  return new KafkaProducer<>(props); }
> >>>>>
> >>>>> That is injected into a single instance of KafkaNotifier::
> >>>>>
> >>>>> public class KafkaNotifier implements MessageNotifier {  private
> >>>>> static Logger logger =
> >>>>> LoggerFactory.getLogger(KafkaNotifier.class);
> >>>>>
> >>>>>  private Producer<String, String> producer;  private String
> >>>>> topicName;
> >>>>>
> >>>>>  @Override
> >>>>>  public void sendNotificationMessage(DataObjectModificationMessage
> >>>>> message) {
> >>>>>
> >>>>>      String json = message.asJSON();
> >>>>>      String key = message.getObject().getId().toString();
> >>>>>
> >>>>>      producer.send(new ProducerRecord<>(topicName, key, json));
> >>>>>
> >>>>>      if (logger.isDebugEnabled()) {
> >>>>>          logger.debug("Sent Kafka message for object with id={}",
> >> key);
> >>>>>      }
> >>>>>  }
> >>>>>
> >>>>>  @Required
> >>>>>  public void setProducer(Producer<String, String> producer)
{
> >>>>>      this.producer = producer;
> >>>>>  }
> >>>>>
> >>>>>  @Required
> >>>>>  public void setTopicName(String topicName) {
> >>>>>      this.topicName = topicName;
> >>>>>  }
> >>>>> }
> >>>>>
> >>>>> Here is the topic config info :
> >>>>>
> >>>>> ./kafka-topics.sh -zookeeper mer-arch-zk-01d.cc.com:2181
> >>>>> --describe --topic s.notifications.dev
> >>>>> Topic:s.notifications.dev       PartitionCount:1
> >>>>> ReplicationFactor:1     Configs:retention.ms
> >>>>> =172800000,cleanup.policy=compact
> >>>>>      Topic: s.notifications.dev      Partition: 0    Leader: 0
> >>>>> Replicas: 0     Isr: 0
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Apr 22, 2016, at 6:27 PM, vinay sharma
> >>>>>> <vinsharma.tech@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> 2 producer's to same topic should not be a problem. There can
be
> >> multiple
> >>>>>> producers and consumers of same kafka topic.
> >>>>>>
> >>>>>> I am not sure what can be wrong here. I can this at my end If
you
> >>>>>> can
> >>>>> share
> >>>>>> producer code and any config of topic ot broker that you changed
> >>>>>> and
> >> is
> >>>>> not
> >>>>>> default.
> >>>>>>
> >>>>>> Please also check that you are not creating producer every time
> >>>>>> you
> >> send
> >>>>> a
> >>>>>> message but reusing producer once created to send multiple messages.
> >>>>> Before
> >>>>>> a send and after creation a producers sends this request to
fetch
> >>>>> metadata.
> >>>>>> I ran a test to publish messages from 2 producers to a topic
with
> >>>>>> 3 partitions on a 3 broker 1 zookeeper kafka setup. I ran test
> >>>>>> for more
> >>>>> than
> >>>>>> a minute and saw just once for both producers before their 1st
send.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Vinay Sharma
> >>>>>> On Apr 22, 2016 3:15 PM, "Fumo, Vincent" <
> >> Vincent_Fumo@cable.comcast.com
> >>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi. I've not set that value. My producer properties are
as follows
> :
> >>>>>>>
> >>>>>>> acks=all
> >>>>>>> retries=0
> >>>>>>> bath.size=1638
> >>>>>>> linger.ms=1
> >>>>>>> buffer.memory=33554432
> >>>>>>> compression.type=gzip
> >>>>>>> client.id=sds-merdevl
> >>>>>>>
> >>>>>>> I have this running on two hosts with the same config. I
thought
> >>>>>>> that having the same client.id on each would just consolidate
> >>>>>>> the
> >> tracking
> >>>>>>> (same logical name). You don't think there is an issue with
2
> >> producers
> >>>>> to
> >>>>>>> the same topic?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Apr 22, 2016, at 3:05 PM, vinay sharma
> >>>>>>>> <vinsharma.tech@gmail.com
> >>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Generally a proactive metadata refresh request is sent
by
> >>>>>>>> producer
> >> and
> >>>>>>>> consumer every 5 minutes but this interval can be overriden
> >>>>>>>> with
> >>>>>>> property "
> >>>>>>>> metadata.max.age.ms" which has default value 300000
i.e 5
> minutes.
> >>>>>>> Check if
> >>>>>>>> you have set this property very low in your producer?
> >>>>>>>>
> >>>>>>>> On Fri, Apr 22, 2016 at 11:46 AM, Fumo, Vincent <
> >>>>>>>> Vincent_Fumo@cable.comcast.com> wrote:
> >>>>>>>>
> >>>>>>>>> I'm testing a kafka install and using the java client.
I have
> >>>>>>>>> a
> >> topic
> >>>>>>> set
> >>>>>>>>> up and it appears to work great, but after a while
I noticed
> >>>>>>>>> my log starting to fill up with what appears to
be some kind
> >>>>>>>>> of loop for
> >>>>>>> metadata
> >>>>>>>>> updates.
> >>>>>>>>>
> >>>>>>>>> example::
> >>>>>>>>>
> >>>>>>>>> 2016-04-22 15:43:55,139 DEBUG s=s-root_out  env="md"
> >>>>>>>>> [kafka-producer-network-thread | sds-merdevl]
> >>>>>>>>> org.apache.kafka.clients.Metadata: Updated cluster
metadata
> >>>>>>>>> version
> >>>>>>> 6196 to
> >>>>>>>>> Cluster(nodes = [Node(0, cmp-arch-kafka-01d.cc.com,
9092)],
> >>>>> partitions
> >>>>>>> =
> >>>>>>>>> [Partition(topic = s.notifications.dev, partition
= 0, leader
> >>>>>>>>> = 0,
> >>>>>>> replicas
> >>>>>>>>> = [0,], isr = [0,]])
> >>>>>>>>> 2016-04-22 15:43:55,240 DEBUG s=s-root_out  env="md"
> >>>>>>>>> [kafka-producer-network-thread | sds-merdevl]
> >>>>>>>>> org.apache.kafka.clients.Metadata: Updated cluster
metadata
> >>>>>>>>> version
> >>>>>>> 6197 to
> >>>>>>>>> Cluster(nodes = [Node(0, cmp-arch-kafka-01d.cc.com,
9092)],
> >>>>> partitions
> >>>>>>> =
> >>>>>>>>> [Partition(topic = s.notifications.dev, partition
= 0, leader
> >>>>>>>>> = 0,
> >>>>>>> replicas
> >>>>>>>>> = [0,], isr = [0,]])
> >>>>>>>>> 2016-04-22 15:43:55,341 DEBUG s=s-root_out  env="md"
> >>>>>>>>> [kafka-producer-network-thread | sds-merdevl]
> >>>>>>>>> org.apache.kafka.clients.Metadata: Updated cluster
metadata
> >>>>>>>>> version
> >>>>>>> 6198 to
> >>>>>>>>> Cluster(nodes = [Node(0, cmp-arch-kafka-01d.cc.com,
9092)],
> >>>>> partitions
> >>>>>>> =
> >>>>>>>>> [Partition(topic = s.notifications.dev, partition
= 0, leader
> >>>>>>>>> = 0,
> >>>>>>> replicas
> >>>>>>>>> = [0,], isr = [0,]])
> >>>>>>>>>
> >>>>>>>>> etc.
> >>>>>>>>>
> >>>>>>>>> It hasn't stopped..
> >>>>>>>>>
> >>>>>>>>> I'm curious about what's going on here. Can anyone
help?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message