hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6978) Add JNI support for secure IO operations
Date Sat, 13 Nov 2010 15:52:14 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931690#action_12931690

Devaraj Das commented on HADOOP-6978:

We have noticed that sometimes the C calls like getpwuid_r ends up making direct calls to
the ldap server. It probably is configuration/environment specific, but in Yahoo! the password
entries are maintained by the ldap server. In order to prevent ldap servers from getting overloaded
with password look-ups, we have a daemon called nscd run on all the compute nodes, that caches
the results of such look-ups. The calls such as getpwuid_r should terminate at the local nscd
daemon, but if, for whatever reason, the nscd daemon is down on the node, the calls end up
talking to the ldap server directly. Apparently, nscd is not that stable... 

We have seen the above happening at Yahoo! and in a couple of occasions brought down the ldap
servers. So I was wondering whether we should reduce the number of calls to the getpwuid_r
and such by caching the resolutions {uid,gid}->{username,groupname} in Hadoop.. Thoughts?

> Add JNI support for secure IO operations
> ----------------------------------------
>                 Key: HADOOP-6978
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6978
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io, native, security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>         Attachments: hadoop-6978.txt, hadoop-6978.txt
> In support of MAPREDUCE-2096, we need to add some JNI functionality. In particular, we
need the ability to use fstat() on an open file stream, and to use open() with O_EXCL, O_NOFOLLOW,
and without O_CREAT.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message