flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9777) YARN: JM and TM Memory must be specified with Units
Date Thu, 12 Jul 2018 12:47:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541573#comment-16541573
] 

ASF GitHub Bot commented on FLINK-9777:
---------------------------------------

Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6297#discussion_r202020619
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java
---
    @@ -386,10 +386,10 @@ private ClusterSpecification createClusterSpecification(Configuration
configurat
     		}
     
     		// JobManager Memory
    -		final int jobManagerMemoryMB = MemorySize.parse(configuration.getString(JobManagerOptions.JOB_MANAGER_HEAP_MEMORY)).getMebiBytes();
    +		final int jobManagerMemoryMB = MemorySize.parse(configuration.getString(JobManagerOptions.JOB_MANAGER_HEAP_MEMORY),
MemorySize.MemoryUnit.MEGA_BYTES).getMebiBytes();
    --- End diff --
    
    @dawidwys As I understand, backwards compatibility should be prioritized. See https://github.com/apache/flink/pull/5448#issuecomment-394296226
    
    > We need to add an additional MemoryUnit.parse() method that takes the "default" unit,
so that we parse the old heap sizes such that they are in MB if nothing else is specified.


> YARN: JM and TM Memory must be specified with Units 
> ----------------------------------------------------
>
>                 Key: FLINK-9777
>                 URL: https://issues.apache.org/jira/browse/FLINK-9777
>             Project: Flink
>          Issue Type: Bug
>          Components: Documentation, YARN
>    Affects Versions: 1.6.0
>         Environment: commit 9f736d1927c62d220a82931c4f5ffa4955910f27
>            Reporter: Gary Yao
>            Assignee: vinoyang
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.6.0
>
>
> FLINK-6469 breaks backwards compatibility because the JobManager and TaskManager memory
must be specified with units (otherwise bytes are assumed). The command to start a YARN session
as documented ([https://github.com/apache/flink/blob/9f736d1927c62d220a82931c4f5ffa4955910f27/docs/ops/deployment/yarn_setup.md)
|https://github.com/apache/flink/blob/9f736d1927c62d220a82931c4f5ffa4955910f27/docs/ops/deployment/yarn_setup.md]
would not work because 1024 bytes and 4096 bytes are not enough for the heap size. The command
finishes with the following exception:
> {noformat}
> java.lang.reflect.UndeclaredThrowableException
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1854)
> 	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> 	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:802)
> Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy
Yarn session cluster
> 	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:420)
> 	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:599)
> 	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:802)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> 	... 2 more
> Caused by: org.apache.flink.util.FlinkException: Cannot fulfill the minimum memory requirements
with the provided cluster specification. Please increase the memory of the cluster.
> 	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.validateClusterSpecification(AbstractYarnClusterDescriptor.java:453)
> 	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:475)
> 	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:413)
> 	... 7 more
> Caused by: java.lang.IllegalArgumentException
> 	at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123)
> 	at org.apache.flink.runtime.clusterframework.ContaineredTaskManagerParameters.calculateCutoffMB(ContaineredTaskManagerParameters.java:115)
> 	at org.apache.flink.yarn.AbstractYarnClusterDescriptor.validateClusterSpecification(AbstractYarnClusterDescriptor.java:450)
> 	... 9 more
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message