flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [flink] azagrebin commented on issue #10180: [FLINK-14631] Account for netty direct allocations in direct memory limit (Netty Shuffle)
Date Mon, 18 Nov 2019 07:39:08 GMT
azagrebin commented on issue #10180: [FLINK-14631] Account for netty direct allocations in
direct memory limit (Netty Shuffle)
URL: https://github.com/apache/flink/pull/10180#issuecomment-554891572
 
 
   @zhijiangW 
   Really thanks for the good insight.
   
   We were thinking about a reasonable default value for the number of arenas before we get
the zero-copy PR in. It seems that allocating even one arena is too much in most of our testing
scenarios. But the artificial tests might be not conclusive enough for that as you just mentioned
it can reach even 1Gb. According tor your experience, what would be that reasonable default
value for the number of arenas? Should we keep the previous one - number of slots?
   
   When you say: `let users config this portion of memory size directly`, do you mean number
of arenas? We keep the number of arenas configurable directly and account for them in shuffle
memory size. Shuffle memory size is also configurable separately. We also check that the configured
shuffle memory size is enough for the configured number of arenas. Did you mean something
like this?
   
   @pnowojski
   sure, I will add the description to the commit

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message