flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2954) Not able to pass custom environment variables in cluster to processes that spawning TaskManager
Date Thu, 26 Nov 2015 11:51:11 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028625#comment-15028625
] 

ASF GitHub Bot commented on FLINK-2954:
---------------------------------------

Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1409#discussion_r45970591
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java ---
    @@ -221,4 +210,22 @@ public static void addToEnvironment(Map<String, String> environment,
     	private Utils() {
     		throw new RuntimeException();
     	}
    +
    +	/**
    +	 * Method to extract environment variables from the flinkConfiguration based on the
given prefix String.
    +	 *
    +	 * @param envPrefix Prefix for the environment variables key
    +	 * @param flinkConfiguration The Flink config to get the environment variable defintion
from
    +	 */
    +	public static Map<String, String> getEnvironmentVariables(String envPrefix, org.apache.flink.configuration.Configuration
flinkConfiguration) {
    +		Map<String, String> result  = new HashMap<>();
    +		for(Map.Entry<String, String> entry: flinkConfiguration.toMap().entrySet()) {
    +			if(entry.getKey().startsWith(envPrefix)) {
    +				// remove prefix
    +				String key = entry.getKey().substring(envPrefix.length());
    --- End diff --
    
    What happens if the key is `envPrefix`? There is little validation here.


> Not able to pass custom environment variables in cluster to processes that spawning TaskManager
> -----------------------------------------------------------------------------------------------
>
>                 Key: FLINK-2954
>                 URL: https://issues.apache.org/jira/browse/FLINK-2954
>             Project: Flink
>          Issue Type: Bug
>          Components: Command-line client, Distributed Runtime
>    Affects Versions: 0.10.0
>            Reporter: Jian Jiang
>            Assignee: Robert Metzger
>            Priority: Critical
>             Fix For: 1.0.0
>
>
> There are programs that rely on custom environment variables. In hadoop mapreduce job
we can use -Dmapreduce.map.env and - Dmapreduce.reduce.env to do pass them. Similarly in Spark
> we can use --conf 'spark.executor.XXX=value for XXX'. There is no such feature yet in
Flink.
> This has given Flink a serious disadvantage when customers need such feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message