flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nico Kruber (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-5973) check whether the direct memory size is always correctly calculated
Date Tue, 06 Jun 2017 08:49:18 GMT

     [ https://issues.apache.org/jira/browse/FLINK-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nico Kruber updated FLINK-5973:
-------------------------------
    Description: 
A user on the mailing list ran into the problem that the {{directMemorySize}} was incorrectly
set too high which may happen if the following code path gets {{maxMemory}} from 1/4*<physical
mem size>> instead of the calculation, {{taskmanager.sh}} is doing (in his case via
the discouraged {{start-local.sh}} script).

Other code paths may also exhibit this issue, which should be checked.

{code:title=TaskManagerServices#createMemoryManager()}
} else if (memType == MemoryType.OFF_HEAP) {
	// The maximum heap memory has been adjusted according to the fraction
	long maxMemory = EnvironmentInformation.getMaxJvmHeapMemory();
	long directMemorySize = (long) (maxMemory / (1.0 - memoryFraction) * memoryFraction);
	if (preAllocateMemory) {
		LOG.info("Using {} of the maximum memory size for managed off-heap memory ({} MB)." ,
			memoryFraction, directMemorySize >> 20);
	} else {
		LOG.info("Limiting managed memory to {} of the maximum memory size ({} MB)," +
			" memory will be allocated lazily.", memoryFraction, directMemorySize >> 20);
	}
	memorySize = directMemorySize;
} else {
{code}

  was:
A user on the mailing list ran into the problem that the {{directMemorySize}} was incorrectly
set too high which may happen if the following code path gets {{maxMemory}} from 1/4*<physical
mem size>> instead of the calculation, {{taskmanager.sh}} is doing (in his case via
the discouraged {{start-local.sh}} script).

It be the case that other code paths also exhibit this issue, which should be checked.

{code:title=TaskManagerServices#createMemoryManager()}
} else if (memType == MemoryType.OFF_HEAP) {
	// The maximum heap memory has been adjusted according to the fraction
	long maxMemory = EnvironmentInformation.getMaxJvmHeapMemory();
	long directMemorySize = (long) (maxMemory / (1.0 - memoryFraction) * memoryFraction);
	if (preAllocateMemory) {
		LOG.info("Using {} of the maximum memory size for managed off-heap memory ({} MB)." ,
			memoryFraction, directMemorySize >> 20);
	} else {
		LOG.info("Limiting managed memory to {} of the maximum memory size ({} MB)," +
			" memory will be allocated lazily.", memoryFraction, directMemorySize >> 20);
	}
	memorySize = directMemorySize;
} else {
{code}


> check whether the direct memory size is always correctly calculated
> -------------------------------------------------------------------
>
>                 Key: FLINK-5973
>                 URL: https://issues.apache.org/jira/browse/FLINK-5973
>             Project: Flink
>          Issue Type: Bug
>          Components: Local Runtime
>            Reporter: Nico Kruber
>            Priority: Minor
>
> A user on the mailing list ran into the problem that the {{directMemorySize}} was incorrectly
set too high which may happen if the following code path gets {{maxMemory}} from 1/4*<physical
mem size>> instead of the calculation, {{taskmanager.sh}} is doing (in his case via
the discouraged {{start-local.sh}} script).
> Other code paths may also exhibit this issue, which should be checked.
> {code:title=TaskManagerServices#createMemoryManager()}
> } else if (memType == MemoryType.OFF_HEAP) {
> 	// The maximum heap memory has been adjusted according to the fraction
> 	long maxMemory = EnvironmentInformation.getMaxJvmHeapMemory();
> 	long directMemorySize = (long) (maxMemory / (1.0 - memoryFraction) * memoryFraction);
> 	if (preAllocateMemory) {
> 		LOG.info("Using {} of the maximum memory size for managed off-heap memory ({} MB)."
,
> 			memoryFraction, directMemorySize >> 20);
> 	} else {
> 		LOG.info("Limiting managed memory to {} of the maximum memory size ({} MB)," +
> 			" memory will be allocated lazily.", memoryFraction, directMemorySize >> 20);
> 	}
> 	memorySize = directMemorySize;
> } else {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message