flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Alexandrov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1789) Allow adding of URLs to the usercode class loader
Date Wed, 01 Apr 2015 15:27:54 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14390800#comment-14390800

Alexander Alexandrov commented on FLINK-1789:

The problem with that approach is that the code loaded by the regular JVM class loader cannot
refer to job specific types (which can be accessed only at the  UserCodeClassLoader level).
Unfortunately, this is the case if we use the classpath entry to generate the dataflows dynamically
at runtime.

My original "hack" was to therefore hardcode a filesystem path next to the list of jars when
initializing the BlobManager, and I wanted to open an issue which makes this configurable
when initializing the execution environment (this is basically the only main feature which
prohibits the use of Emma with "off-the-shelf" Flink).

This, of course, would require that the folders are shared (e.g. via NFS) between client,
master and workers. I think what made Stephan so excited is the idea of using the same URL
mechanism in order to ship the code to all dependent parties (most probably by running a dedicated
HTTP or FTP server on the client).

> Allow adding of URLs to the usercode class loader
> -------------------------------------------------
>                 Key: FLINK-1789
>                 URL: https://issues.apache.org/jira/browse/FLINK-1789
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Runtime
>            Reporter: Timo Walther
>            Assignee: Timo Walther
>            Priority: Minor
> Currently, there is no option to add customs classpath URLs to the FlinkUserCodeClassLoader.
JARs always need to be shipped to the cluster even if they are already present on all nodes.
> It would be great if RemoteEnvironment also accepts valid classpaths URLs and forwards
them to BlobLibraryCacheManager.

This message was sent by Atlassian JIRA

View raw message