flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (FLINK-8125) Support limiting the number of open FileSystem connections
Date Fri, 24 Nov 2017 12:29:01 GMT

     [ https://issues.apache.org/jira/browse/FLINK-8125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Stephan Ewen resolved FLINK-8125.
    Resolution: Fixed

Fixed in
  - 1.5.0 via d7c2c417213502130b1aeab1868313df178555cc
  - 1.4.1 via a11e2cf0b1f37d3ef22e1978e89928fa374960db

> Support limiting the number of open FileSystem connections
> ----------------------------------------------------------
>                 Key: FLINK-8125
>                 URL: https://issues.apache.org/jira/browse/FLINK-8125
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 1.5.0, 1.4.1
> We need a way to limit the number of streams that Flink FileSystems concurrently open.
> For example, for very small HDFS clusters with few RPC handlers, a large Flink job trying
to build up many connections during a checkpoint causes failures due to rejected connections.

> I propose to add a file system that can wrap another existing file system The file system
may track the progress of streams and close streams that have been inactive for too long,
to avoid locked streams of taking up the complete pool.

This message was sent by Atlassian JIRA

View raw message