flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7582) Rest client may buffer responses indefinitely under heavy laod
Date Tue, 05 Sep 2017 17:43:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154042#comment-16154042
] 

ASF GitHub Bot commented on FLINK-7582:
---------------------------------------

Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/4645#discussion_r137065666
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java ---
    @@ -159,24 +155,17 @@ public void shutdown(Time timeout) {
     	}
     
     	private <P extends ResponseBody> CompletableFuture<P> submitRequest(String
targetAddress, int targetPort, FullHttpRequest httpRequest, Class<P> responseClass)
{
    -		return CompletableFuture.supplyAsync(() -> bootstrap.connect(targetAddress, targetPort),
executor)
    -			.thenApply((channel) -> {
    -				try {
    -					return channel.sync();
    -				} catch (InterruptedException e) {
    -					throw new FlinkRuntimeException(e);
    -				}
    -			})
    -			.thenApply((ChannelFuture::channel))
    -			.thenCompose(channel -> {
    -				ClientHandler handler = channel.pipeline().get(ClientHandler.class);
    -				CompletableFuture<JsonResponse> future = handler.getJsonFuture();
    -				channel.writeAndFlush(httpRequest);
    -				return future;
    -			}).thenComposeAsync(
    -				(JsonResponse rawResponse) -> parseResponse(rawResponse, responseClass),
    -				executor
    -			);
    +		ChannelFuture connect = bootstrap.connect(targetAddress, targetPort);
    +		Channel channel;
    --- End diff --
    
    Technically yes, this alone would fix the problem at hand.
    
    The sending part was modified as a band-aid to alleviate issues that can occur under heavy
load since we don't have any throttling mechanism for accepting requests. For example, in
one of our tests we had thousands of futures suddenly fail because all ports were in use,
not to mention that you can allocate a virtually infinite number of futures (way faster than
we could ever process them).
    
    I couldn't quickly come up with a good scheme to throttle requests though. Suggestions
are welcome ;)


> Rest client may buffer responses indefinitely under heavy laod
> --------------------------------------------------------------
>
>                 Key: FLINK-7582
>                 URL: https://issues.apache.org/jira/browse/FLINK-7582
>             Project: Flink
>          Issue Type: Bug
>          Components: REST
>    Affects Versions: 1.4.0
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>             Fix For: 1.4.0
>
>
> The RestClient uses an executor for sending requests and parsing responses. Under heavy
load, i.e. lots of requests being sent, the executor may be used exclusively for sending requests.
The responses that are received by the netty threads are thus never parsed and are buffered
in memory, until either requests stop coming in or all memory is used up.
> We should let the netty receiver thread do the parsing as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message