flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangzhijiang999" <wangzhijiang...@aliyun.com>
Subject 答复:答复:how to understand the flink flow control
Date Tue, 11 Aug 2015 01:35:49 GMT
Hi Ufuk,
          Thank you for your instant reply and I have got the basic idea. 
           I am researching flink code now, and  if I have further specific questions
next time, I will send to dev@flink.a.o .
Best wishes,
Zhijiang Wang
------------------------------------------------------------------发件人:Ufuk Celebi
<uce@apache.org>发送时间:2015年8月10日(星期一) 17:13收件人:user <user@flink.apache.org>,wangzhijiang999
<wangzhijiang999@aliyun.com>主 题:Re: 答复:how to understand the flink flow
controlGood to hear. Answers to your questions are inline.I think we should move the discussion
to the dev@flink.a.o list though if there are more questions.On Mon, Aug 10, 2015 at 9:29
AM, wangzhijiang999 <wangzhijiang999@aliyun.com> wrote:Hi Ufuk,       Thank you
for your detail and clear explaination!  I reviewed the code based on your info and got it
clearly.For local transfer: When producer emits results to ResultPartition, it will request
 Buffer from the pool. If there  are no available Buffer, it will wait .Yes, the buffer
request is blocking. The ResultPartition of producer is the InputGate of Consumer, when consumer
read buffer from InputGate and deserializer the buffer, the Buffer will be recycled ,so the
producer can request Buffer again.The producer never requests a buffer from the input gate,
but from its own pool. If the input gate is slow to recycle its buffers, the producer will
become slow as well. But you are right that the producer will not be able to request a buffer
if the input gate never recycles them. So if the consumer slows, the Buffer in the ResultPartition
of producer can not be recycled quickly, resulting in producer has no available Buffer to
emit data.If the producer waits the available Buffer to emit, it can not process elements
and read next data from its InputGate, resulting in slowing the producer.Yes. The task thread
is both consuming and producing data. So if its blocked on the output side, the input is not
consumed, which propagates back pressure further. For remote transfer:     The InputGate
and ResultPartition for each task are separate     1. For producer the Buffer in ResultPartition
will be recycled when producer write them to the channel.     2. For consumer the Buffer
will be recycled when consumer read it from InputGate and deserializer it . The consumer need
to request Buffer when reading data from channel     and put them to InputGate.     3. 
When consumer slows and there are no available Buffer , the consumer will not read data from
channel, so it will affect the producer writing data to channel and  the producer will have
no available Buffer to emit result at last resulting in slowing producer.Yes but there is
some logic involved to stop reading from the actual TCP channel to back pressure the remote
producer. Essentially the selector read event is unsubscribed and on the output side there
is Netty's watermark mechanism, which prevents further data from being written to the TCP

View raw message