kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marina <ppi...@yahoo.com.INVALID>
Subject what do 'soTimeout', 'bufferSize' and 'minBytes' mean for SimpleConsumer?
Date Mon, 08 Jun 2015 20:46:26 GMT
I'm using Kafka SimpleConsumer . Could somebody clarify the meaning of a few config parameters for
the SimpleConsumer and FetchREquestBuilder? My questions are below:
-- Q1: in the signature of the SimpleConsumer constructor I see the Int 'soTimeout' parameter
what is the meaning of this timeout? Is this a timeout to connect to the Kafka broker? Timeout
on getting a response from any [or specific??] request to Kafka (like FetchRequest)? Something
    (val host: String,
    val port: Int,
    val soTimeout: Int,
    val bufferSize: Int,
    val clientId: String) 

-- Q2: also, SimpleConsumer constructor takes Int 'bufferSize' parameter. What is the meaning
of it? Is this how many bytes SimpleConsumer will read when a fetchRequest is issued? 
-- When building FetchREquest via FetchRequestBuilder (see below), I also need to specify
FetchRequest req= newFetchRequestBuilder ()
    .addFetch(topic, partition, offset, fetchSizeInBytes)
Looking at the source code of the FetchRequestBuilder , I think (I'm not a Scala pro) those
translate into the below method calls - and there the final parameter passed into the FetchRequest
is called 'minBytes', hinting that this is not the exact fetch size, possibly….
class FetchRequestBuilder(): 
    def addFetch(topic: String, partition: Int, offset: Long, fetchSize: Int)
    def build() = {
    val fetchRequest= FetchRequest(versionId, correlationId.getAndIncrement, clientId, replicaId,
maxWait, minBytes, requestMap.toMap)
FetchRequest(versionId: Short = FetchRequest.CurrentVersion,
    correlationId: Int = FetchRequest.DefaultCorrelationId,
    clientId: String = ConsumerConfig.DefaultClientId,
    replicaId: Int = Request.OrdinaryConsumerId,
    maxWait: Int = FetchRequest.DefaultMaxWait,
    minBytes: Int = FetchRequest.DefaultMinBytes,
So, my final question is: 
-- Q3: how do 'bufferSize' and 'fetchSize/minBytes' relate? What exactly do they define? Do
I have to make
sure one is smaller or grater than the other?

View raw message