flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-11107) [state] Avoid memory stateBackend to create arbitrary folders under HA path when no checkpoint path configured
Date Wed, 12 Dec 2018 06:34:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718511#comment-16718511
] 

ASF GitHub Bot commented on FLINK-11107:
----------------------------------------

Myasuka commented on issue #7281: [FLINK-11107][state] Avoid memory stateBackend to create
arbitrary folders under HA path when no checkpoint path configured
URL: https://github.com/apache/flink/pull/7281#issuecomment-446478434
 
 
   BTW, If we do not create UUID directories for memory state-backend in this situation, job
could still restore from high-availability storage. The only difference is the information
of `Latest Restore` under `Checkpoints` tab of web UI would show the path is `<checkpoint-not-externally-addressable>`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> [state] Avoid memory stateBackend to create arbitrary folders under HA path when no checkpoint
path configured
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-11107
>                 URL: https://issues.apache.org/jira/browse/FLINK-11107
>             Project: Flink
>          Issue Type: Bug
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.6.2, 1.7.0
>            Reporter: Yun Tang
>            Assignee: Yun Tang
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.6.3, 1.7.1
>
>
> Currently, memory state-backend would create a folder named with random UUID under HA
directory if no checkpoint path ever configured. (the code logic locates within {{StateBackendLoader#fromApplicationOrConfigOrDefault}})
However, the default memory state-backend would not only be created on JM side, but also on
each task manager's side, which means many folders with random UUID would be created under
HA directory. It would result in exception like:
> {noformat}
> The directory item limit of /tmp/flink/ha is exceeded: limit=1048576 items=1048576{noformat}
>  If this happens, no new jobs could be submitted only if we clean up those directories
manually.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message