storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabh...@gmail.com>
Subject Re: Apache Storm High Messaging Delay When Passing 5MB Images
Date Wed, 31 Jan 2018 23:36:59 GMT
I'm not clear whether you're saying message transfer for each bolt took
200ms, or summation of 4 or 5 times network transfer latencies were 200 ms.

Why I say I'm not clear is that if it's latter and there's 5 times network
transfers, it is ideal latency in theory, since 1Gbps is 125MBps (1000Mbps,
not 1024Mbps) and 5M/125MBps = 40ms per each transfer. (I'd rather suspect
how it was possible in this case.)

Even there's 4 times network transfers, we may need to take this to
account: the latency calculation above is in theory, and there're many
overheads other than messaging which is not easily seen, so the latency may
not be from messaging overhead.

If it took 200 ms for each transfer that can be a thing to talk about.
Please let me know which is your case.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 2월 1일 (목) 오전 8:11, Wuyang Zhang <wuyang@winlab.rutgers.edu>님이
작성:

> I am playing with Apache Storm for a real-time image processing
> application which requires ultra low latency. In the topology definition, a
> single spout will emit raw images(5MB) in every 1s and a few bolts will
> process them. The processing latency of each bolt is acceptable and the
> overall computing delay can be around 150ms.
>
> *However, I find that the message passing delay between workers on the
> different nodes is really high. The overall such delay on the 5 successive
> bolts is around 200ms.* To calculate this delay, I subtract all the task
> latencies from the end-to-end latency. Moreover, I implement a timer bolt
> and other processing bolts will register in this timer bolt to record the
> timestamp before starting the real processing. By comparing the timestamps
> of the bolts, I find the delay between each bolt is high as I previously
> noticed.
>
> To analyze the source of this high additional delay, I firstly reduce the
> sending interval to 1s and thus there should be no queuing delay due to the
> high computing overheads. Also, from the Storm UI, I find none bolt is in
> high CPU utilization.
>
> Then, I checked the network delay. I am using a 1Gbps network testbed and
> test the network by RTT and bandwidth. The network latency should not be
> that high to send a 5MB image.
>
> Finally, I am thinking about the buffer delay. I find each thread
> maintains its own sending buffer and transfer the data to the worker's
> sending buffer. I am not sure how long it takes before the receiver bolt
> can get this sending message. As suggested by the community, I increase the
> sender/receiver buffer size to 16384, modify STORM_NETTY_MESSAGE_BATCH_SIZE
> to 32768. However, it did not help.
>
> *My question is that how to remove/reduce the messaging overheads between
> bolts?(inter workers)* It is possible to synchronize the communication
> between bolts and have the receiver got the sending messages immediately
> without any delay?
>
> ᐧ
>

Mime
View raw message