kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marina <ppi...@yahoo.com.INVALID>
Subject what do 'soTimeout', 'bufferSize' and 'minBytes' mean for SimpleConsumer?
Date Mon, 08 Jun 2015 20:46:26 GMT
Hi, 
I'm using Kafka
0.8.2.1 SimpleConsumer . Could somebody clarify the meaning of a few config parameters for
the SimpleConsumer and FetchREquestBuilder? My questions are below:
 
-- Q1: in the signature of the SimpleConsumer constructor I see the Int 'soTimeout' parameter
-
what is the meaning of this timeout? Is this a timeout to connect to the Kafka broker? Timeout
on getting a response from any [or specific??] request to Kafka (like FetchRequest)? Something
else?
 
kafka.javaapi.consumer.SimpleConsumer
    (val host: String,
    val port: Int,
    val soTimeout: Int,
    val bufferSize: Int,
    val clientId: String) 
 


-- Q2: also, SimpleConsumer constructor takes Int 'bufferSize' parameter. What is the meaning
of it? Is this how many bytes SimpleConsumer will read when a fetchRequest is issued? 
 
-- When building FetchREquest via FetchRequestBuilder (see below), I also need to specify
'fetchSize': 
 
FetchRequest req= newFetchRequestBuilder ()
    .clientId(kafkaGroupId)
    .addFetch(topic, partition, offset, fetchSizeInBytes)
    .build();
 
Looking at the source code of the FetchRequestBuilder , I think (I'm not a Scala pro) those
calls
translate into the below method calls - and there the final parameter passed into the FetchRequest
is called 'minBytes', hinting that this is not the exact fetch size, possibly….
 
class FetchRequestBuilder(): 
    def addFetch(topic: String, partition: Int, offset: Long, fetchSize: Int)
 
    def build() = {
    val fetchRequest= FetchRequest(versionId, correlationId.getAndIncrement, clientId, replicaId,
maxWait, minBytes, requestMap.toMap)
 
FetchRequest(versionId: Short = FetchRequest.CurrentVersion,
    correlationId: Int = FetchRequest.DefaultCorrelationId,
    clientId: String = ConsumerConfig.DefaultClientId,
    replicaId: Int = Request.OrdinaryConsumerId,
    maxWait: Int = FetchRequest.DefaultMaxWait,
    minBytes: Int = FetchRequest.DefaultMinBytes,
    ...)
 
 
So, my final question is: 
-- Q3: how do 'bufferSize' and 'fetchSize/minBytes' relate? What exactly do they define? Do
I have to make
sure one is smaller or grater than the other?
 
Thanks,
Marina

Mime
View raw message