kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: Kafka Consumer poll
Date Wed, 03 Aug 2016 00:20:53 GMT
Also keep in mind that unfortunately KafkaConsumer.poll(..) will deadlock regardless of the
timeout if connection to the broker can not be established and won't react to thread interrupts.
This essentially means that the only way to exit is to kill jvm. This is all because Kafka
fetches topic metadata synchronously before timeout takes effect.
While it is my understanding that the reason for it is there is a background thread attempting
to reconnect in the event of temporary broker outage, it doesn't help if you accidentally
specified wrong broker url.

Oleg

> On Aug 2, 2016, at 10:27, Kamal C <kamaltarget@gmail.com> wrote:
> 
> See the answers inline.
> 
>> On Tue, Aug 2, 2016 at 12:23 AM, sat <sathish.alwar@gmail.com> wrote:
>> 
>> Hi,
>> 
>> I am new to Kafka. We are planning to use Kafka messaging for our
>> application. I was playing with Kafka 0.9.0.1 version and i have following
>> queries. Sorry for asking basic questions.
>> 
>> 
>> 1) I have instantiated Kafka Consumer and invoked
>> consumer.poll(Long.MAX_VALUE). Although i have specified timeout as
>> Long.MAX_VALUE, i observe my consumer to fetch records whenever the
>> publisher publishes a message to a topic. This makes me wonder whether
>> Kafka Consumer is push or pull mechanism. Please help us understand the
>> logic of consumer.poll(timeout).
> 
> Fetches the data from the topic, waiting up to the specified wait time *if
> necessary *for a record to become available.
> Kafka Consumer by design is pull mechanism.
> 
> Take a look into Kafka Consumer java docs[1]. It's explained in detail.
> 
> 
>> 2) What are the pros and cons of poll for long timeout vs short timeout.
>> 
>> Short Timeout
> 
> Pros:
> - On shutdown, if no data available in the topic -- Shutdown will be quick
> 
> Cons:
> - Number of network trips will be high
> 
> 
>> 
>> Thanks and Regards
>> A.SathishKumar
> 
> [1]:
> https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
> 
> -- Kamal

Mime
View raw message