flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-1928) BlobManager can be a bottleneck on large deployments
Date Wed, 22 Apr 2015 10:34:58 GMT
Robert Metzger created FLINK-1928:

             Summary: BlobManager can be a bottleneck on large deployments
                 Key: FLINK-1928
                 URL: https://issues.apache.org/jira/browse/FLINK-1928
             Project: Flink
          Issue Type: Improvement
          Components: BlobManager, JobManager, TaskManager
    Affects Versions: 0.9
            Reporter: Robert Metzger
            Priority: Minor

The BlobManager needed more than 2 minutes to distribute 220+ jar files to 50 TaskManagers
in a user's setup.

Serving the files for all the TMs from one JM might will quickly make the network of the JM
a bottleneck when distributing the files.
Potential solutions could be:
- Use a torrent-style distribution mode
- Use HDFS

This message was sent by Atlassian JIRA

View raw message