kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sa Li <sal...@gmail.com>
Subject Re: connection error among nodes
Date Mon, 19 Jan 2015 19:05:59 GMT
Hello, Jun

I run such command:

bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test-rep-three 1000000 3000 -1 acks=-1 bootstrap.servers=10.100.98.100:9092,
10.100.98.101:9092 <http://10.100.98.100:9092/>, 10.100.98.102:9092
<http://10.100.98.100:9092/> buffer.memory=67108864 batch.size=8196

This is the perf test to producing data to different hosts. I have such
questions:
1. If I use bootstrap.servers=10.100.98.100:9092, 10.100.98.101:9092
<http://10.100.98.100:9092/>, 10.100.98.102:9092
<http://10.100.98.100:9092/>, does this mean I send data to each of servers
through 9092? However, if I set bootstrap.servers=10.100.98.100:9092 only,
the producer will send data to 100, while replicate the data through other
ports?

here is the netstate I saw:
 netstat -plantue | egrep -i '.98.101|.98.102'
tcp        0      0 10.100.98.100:37512     10.100.98.102:22
ESTABLISHED 1004       7516316     5862/ssh
tcp6       0      0 10.100.98.100:9092      10.100.98.102:56819
ESTABLISHED 0          371522      3852/java
tcp6       0      0 10.100.98.100:3888      10.100.98.101:40052
ESTABLISHED 0          27514       1793/java
tcp6       0    202 10.100.98.100:53592     10.100.98.101:9092
ESTABLISHED 0          497715      3852/java
tcp6       0      0 10.100.98.100:3888      10.100.98.102:32837
ESTABLISHED 0          21701       1793/java
tcp6       0      0 10.100.98.100:9092      10.100.98.101:51053
ESTABLISHED 0          371526      3852/java
tcp6       0    270 10.100.98.100:53591     10.100.98.101:9092
ESTABLISHED 0          497713      3852/java
tcp6       0      0 10.100.98.100:9092      10.100.98.102:57554
ESTABLISHED 0          7491796     3852/java
tcp6       0      0 10.100.98.100:9092      10.100.98.101:51055
ESTABLISHED 0          601277      3852/java
tcp6       0      0 10.100.98.100:9092      10.100.98.102:56824
ESTABLISHED 0          614795      3852/java
tcp6       0      0 10.100.98.100:48226     10.100.98.102:9092
ESTABLISHED 0          3659949     3852/java
tcp6       0      0 10.100.98.100:9092      10.100.98.101:51054
ESTABLISHED 0          601275      3852/java
tcp6       0      0 10.100.98.100:48225     10.100.98.102:9092
ESTABLISHED 0          3803556     3852/java
tcp6       0      0 10.100.98.100:53593     10.100.98.101:9092
ESTABLISHED 0          638462      3852/java
tcp6       0    236 10.100.98.100:48228     10.100.98.102:9092
ESTABLISHED 0          3936260     3852/java
tcp6       0      0 10.100.98.100:9092      10.100.98.102:56827
ESTABLISHED 0          601276      3852/java
tcp6       0      0 10.100.98.100:9092      10.100.98.101:51052
ESTABLISHED 0          614796      3852/java
tcp6       0    230 10.100.98.100:53594     10.100.98.101:9092
ESTABLISHED 0          637547      3852/java
tcp6       0      0 10.100.98.100:9092      10.100.98.102:56826
ESTABLISHED 0          601274      3852/java
tcp6       0      0 10.100.98.100:2181      10.100.98.102:34162
ESTABLISHED 0          7491795     1793/java
tcp6       0    230 10.100.98.100:48227     10.100.98.102:9092
ESTABLISHED 0          3121735     3852/java
tcp6       0      0 10.100.98.100:9092      10.100.98.102:56825
ESTABLISHED 0          497716      3852/java

2. I still see the error of "server disconnect" during the perf test, but
back to normal and print the results like

11212 records sent, 2239.7 records/sec (6.41 MB/sec), 7287.1 ms avg
latency, 15005.0 max latency.
11620 records sent, 2313.8 records/sec (6.62 MB/sec), 7248.9 ms avg
latency, 14807.0 max latency.
11522 records sent, 2293.4 records/sec (6.56 MB/sec), 7110.9 ms avg
latency, 14551.0 max latency.
11058 records sent, 2200.2 records/sec (6.29 MB/sec), 7176.8 ms avg
latency, 14774.0 max latency.

But by monitoring the connection between nodes, we didn't see any
disconnections, but we are not sure if the kafka servers disconnection took
place or not, how to check it up?

thanks

AL


On Sun, Jan 18, 2015 at 10:21 AM, Jun Rao <jun@confluent.io> wrote:

> Any issue with the network?
>
> Thanks,
>
> Jun
>
> On Wed, Jan 7, 2015 at 1:59 PM, Sa Li <salicn@gmail.com> wrote:
>
> > Things bother me, sometimes, the errors won't pop out, sometimes it
> comes,
> > why?
> >
> > On Wed, Jan 7, 2015 at 1:49 PM, Sa Li <salicn@gmail.com> wrote:
> >
> > >
> > > Hi, Experts
> > >
> > > Our cluster is a 3 nodes cluster, I simply test producer locally, see
> > >
> > > bin/kafka-run-class.sh
> org.apache.kafka.clients.tools.ProducerPerformance
> > > test-rep-three 1000000 3000 -1 acks=1 bootstrap.servers=
> > 10.100.98.100:9092
> > > buffer.memory=67108864 batch.size=8196
> > >
> > > But I got such error, I do think this is critical issue, it just
> > > temporally lose the connection and get back, what is the reason for
> this?
> > >
> > > [2015-01-07 21:44:14,180] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >         at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > >         at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > >         at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,190] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >         at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > >         at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > >         at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,200] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >         at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > >         at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > >         at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,210] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >         at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > >         at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > >         at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,220] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >         at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > >         at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > >         at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,230] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >         at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > >         at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > >         at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,240] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >         at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > >         at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > >         at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > >         at java.lang.Thread.run(Thread.java:745)
> > > 80 records sent, 13.6 records/sec (0.04 MB/sec), 31113.8 ms avg
> latency,
> > > 46912.0 max latency.
> > > 116695 records sent, 23339.0 records/sec (66.77 MB/sec), 5157.1 ms avg
> > > latency, 46946.0 max latency.
> > >
> > >
> > > thanks
> > >
> > > --
> > >
> > > Alec Li
> > >
> >
> >
> >
> > --
> >
> > Alec Li
> >
>



-- 

Alec Li

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message