kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gnanasoundari Soundarajan <gnanasoundari.soundara...@in.man.eu>
Subject Kafka Producer Query - how to increase throughput of producer
Date Thu, 31 May 2018 15:22:56 GMT
Hi,

I am new to kafka. I am getting less throughput and high latency in publishing message of
size 100-200 bytes.

I have the producer configured with the following configuration. I am using akka-reactive
kafka to publish messages
Configuration:
kafka {
        producer {
                parallelism = 500
                bootstrap.servers = "x:9092"
                close-timeout = 86400s
                auto.offset.reset = "earliest"
                max.redeliveries = 0
                use-dispatcher = "kafka-dispatcher"
                kafka-clients {
                retries = 0
                acks = 0
                buffer.memory=67108864
                batch.size=8192
                linger.ms = 2
       }

        }
  }


kafka-dispatcher {
  type = Dispatcher
  executor = "fork-join-executor"
  fork-join-executor {
    parallelism-min = 10
    parallelism-factor = 20.0
    parallelism-max = 300
  }

  throughput = 1
}

Traffic Pattern:
Message published to 10 Topic with 150 partitions.
I am sending the request for 10 Topics with 30 concurrent requests from Jmeter.
I am getting the throughput of 3-5 Hits/sec.

Kafka Server Configuration:

Kafka server and Zookeeper on the same machine centos with 4 CPU 100GB Diskspace and 16 GB
RAM.
All are defaults except the below one:

# The number of threads that the server uses for receiving requests from the network and sending
responses to the network
num.network.threads=50

# The number of threads that the server uses for processing requests, which may include disk
I/O
num.io.threads=100

Let me know how to increase the throughput of producer ?

Regards,
Gnana


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message