kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sining Ma <sinin...@aol.com>
Subject Kafka producer sends messages efficiency
Date Mon, 06 May 2013 21:48:28 GMT
I tried some simple performance tests for kafka producer.
I read 1000 lines of text from one file into memory and write these data to a file, regular
socket and using kafka producer.

The results is as follows:
When I use BufferedOutputStream writing the data to a file, it takes 17 milliseconds to write
1000 lines.

		FileOutputStream fos = new FileOutputStream(filename, true);
		BufferedOutputStream bufferdOutput = new BufferedOutputStream(fos);

		long start = System.currentTimeMillis();
		for (String line2: strList) {
		long end = System.currentTimeMillis();

When I use BufferedOutputStream writing the data to a regular socket, it takes 21 milliseconds
to write 1000 lines.
	os = new BufferedOutputStream(new DataOutputStream(socket.getOutputStream()));

		long start = System.currentTimeMillis();
		for (String line: strList) {
		long end = System.currentTimeMillis();

At last, I use kafka producer to send the data. My current configuration is 


It takes 25 milliseconds to write 1000 lines. But if I use sync send, it takes 84 milliseconds.

My question is that is there any method that I can reduce the time and improve the efficiency
for sending messages through kafka producer.

-- Regards	
   Sining Ma

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message