hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8852) DelegationTokenRenewer should be Singleton
Date Wed, 03 Oct 2012 23:21:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468968#comment-13468968

Karthik Kambatla commented on HADOOP-8852:

After spending some time on this, I noticed a couple of other related issues:
- WebHdfsFileSystem and HftpFileSystem have their own implemenations of TokenRenewer that
are not used at all.
- In DelegationTokenRenewer#renew() (see snippet below), we fetch new tokens in case the renew
fails. However, we use only the first token and ignore the rest. Later, if we can't renew
this token, we re-fetch new tokens instead of using the previously fetched ones. Also, we
might not be able to fetch new ones, in which case we give up. Is this valid behavior? Or,
should we be first using the initially fetched tokens, and re-fetch only when we run out of

  Token<?>[] tokens = fs.addDelegationTokens(null, null);
  if (tokens.length == 0) {
    throw new IOException("addDelegationTokens returned no tokens");

Taking the above into consideration along with the goal of further decoupling DelegationTokenRenewer
from actual FS-specific details of renewal/refetching/cancellation, I propose the following.
Please advise appropriately.
# TokenRenewer should be an interface (it is currently a fully abstract class).
# Create a new abstract class TokenManager that holds a list (queue) of tokens, their kind,
renewal period, a TokenRenewer implementation, and an abstract method #manage() to manage
(renew/re-fetch/cancel) the tokens as appropriate.
# Each FileSystem (WebHdfs and Hftp) has its own implemenation (local extended class) of the
# DelegationTokenRenewer allows registering and de-registering TokenManagers. FileSystems
call register() at init(), and de-register() at close().
# DelegationTokenRenewer stores the registered TokenManagers in its DelayQueue. On a TokenManager's
turn, the manage() method is invoked. If the manage fails, the TokenManager is automatically

> DelegationTokenRenewer should be Singleton
> ------------------------------------------
>                 Key: HADOOP-8852
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8852
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.0.0-alpha
>            Reporter: Tom White
>            Assignee: Karthik Kambatla
>         Attachments: hadoop-8852.patch, hadoop-8852.patch, hadoop-8852-v1.patch
> Updated description:
> DelegationTokenRenewer should be Singleton - the instance and renewer threads should
be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop
the renewer, and only register/de-register for token renewal.
> Original issue:
> HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when
they are closed. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message