storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabh...@gmail.com>
Subject Re: Apache Storm High Messaging Delay When Passing 5MB Images
Date Wed, 31 Jan 2018 23:38:54 GMT
I meant "easily seen" as "not exposed" and we are easy to miss to consider.

2018년 2월 1일 (목) 오전 8:36, Jungtaek Lim <kabhwan@gmail.com>님이 작성:

> I'm not clear whether you're saying message transfer for each bolt took
> 200ms, or summation of 4 or 5 times network transfer latencies were 200 ms.
>
> Why I say I'm not clear is that if it's latter and there's 5 times network
> transfers, it is ideal latency in theory, since 1Gbps is 125MBps (1000Mbps,
> not 1024Mbps) and 5M/125MBps = 40ms per each transfer. (I'd rather suspect
> how it was possible in this case.)
>
> Even there's 4 times network transfers, we may need to take this to
> account: the latency calculation above is in theory, and there're many
> overheads other than messaging which is not easily seen, so the latency may
> not be from messaging overhead.
>
> If it took 200 ms for each transfer that can be a thing to talk about.
> Please let me know which is your case.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 1일 (목) 오전 8:11, Wuyang Zhang <wuyang@winlab.rutgers.edu>님이
작성:
>
>> I am playing with Apache Storm for a real-time image processing
>> application which requires ultra low latency. In the topology definition, a
>> single spout will emit raw images(5MB) in every 1s and a few bolts will
>> process them. The processing latency of each bolt is acceptable and the
>> overall computing delay can be around 150ms.
>>
>> *However, I find that the message passing delay between workers on the
>> different nodes is really high. The overall such delay on the 5 successive
>> bolts is around 200ms.* To calculate this delay, I subtract all the task
>> latencies from the end-to-end latency. Moreover, I implement a timer bolt
>> and other processing bolts will register in this timer bolt to record the
>> timestamp before starting the real processing. By comparing the timestamps
>> of the bolts, I find the delay between each bolt is high as I previously
>> noticed.
>>
>> To analyze the source of this high additional delay, I firstly reduce the
>> sending interval to 1s and thus there should be no queuing delay due to the
>> high computing overheads. Also, from the Storm UI, I find none bolt is in
>> high CPU utilization.
>>
>> Then, I checked the network delay. I am using a 1Gbps network testbed and
>> test the network by RTT and bandwidth. The network latency should not be
>> that high to send a 5MB image.
>>
>> Finally, I am thinking about the buffer delay. I find each thread
>> maintains its own sending buffer and transfer the data to the worker's
>> sending buffer. I am not sure how long it takes before the receiver bolt
>> can get this sending message. As suggested by the community, I increase the
>> sender/receiver buffer size to 16384, modify STORM_NETTY_MESSAGE_BATCH_SIZE
>> to 32768. However, it did not help.
>>
>> *My question is that how to remove/reduce the messaging overheads between
>> bolts?(inter workers)* It is possible to synchronize the communication
>> between bolts and have the receiver got the sending messages immediately
>> without any delay?
>>
>> ᐧ
>>
>

Mime
View raw message