samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yi Pan <nickpa...@gmail.com>
Subject Re: Security on YARN
Date Wed, 01 Jul 2015 23:38:49 GMT
Hi, Yan,

Your memory serves as well as mine. :) I remember that Chris and I
discussed this Kerberos ticket expiration issue when we were brain storming
on how to access HDFS data in Samza. At high-level, what happens is that
the Kerberos ticket to access a secured Hadoop cluster is issued to Samza
containers at the job start time, and will expire later. For a long-running
Samza job, it does not work. We will need a way to refresh the Kerberos
ticket periodically, which is not supported yet. Chris probably can chime
in with more details.

-Yi

On Wed, Jul 1, 2015 at 4:08 PM, Yan Fang <yanfang724@gmail.com> wrote:

> Hi Qi,
>
> I think this is caused by the fact that Samza currently does not support
> Yarn with Kerberos. Feel free to open a ticket for this feature.
>
> But if my memory serves, there was an issue mentioned about the Kerberos.
> Seems when the Kerberos ticket expires, Samza will have some issues? Can
> not find the resource. Anyone remember this?
>
> Cheers,
>
> Fang, Yan
> yanfang724@gmail.com
>
> On Wed, Jul 1, 2015 at 3:41 AM, Qi Fu <qfu@talend.com> wrote:
>
> > Hi all,
> >
> >
> > I'm testing Samza on YARN and I have encountered a problem on the
> security
> > setting of YARN (Kerberos). Here is the detail:
> >
> > 1. My cluster is secured by Kerberos, and I deploy my samza job from one
> > of the cluster.
> >
> >
> > 2. My config file is in ~/.samza/conf/(yarn-site.xml, core-site.xml,
> > hdfs-site.xml)
> >
> >
> > 3. The job is deployed successfully, and I can get the info such as:
> >
> >     ClientHelper [INFO] set package url to scheme: "hdfs" port: -1 file:
> > "/user/test/samzatest.tar.gz" for application_1435680272316_0003
> >
> >     ClientHelper [INFO] set package size to 212924524 for
> > application_1435680272316_0003
> >
> >
> >
> >     I think the security setting is correct as it can get the file size
> > from HDFS.
> >
> >
> > 4. But I get the error from YARN job manager as following:
> >
> >
> >     Application application_1435680272316_0003 failed 2 times due to AM
> > Container for appattempt_1435680272316_0003_000002 exited with exitCode:
> > -1000
> >
> > For more detailed output, check application tracking page:
> > http://cdh-namenode:8088/proxy/application_1435680272316_0003/Then,
> click
> > on links to logs of each attempt.
> >
> > Diagnostics: Failed on local exception: java.io.IOException:
> > org.apache.hadoop.security.AccessControlException: Client cannot
> > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is:
> > "talend-cdh-datanode8/62.210.141.237"; destination host is:
> > "talend-cdh-namenode":8020;
> >
> > java.io.IOException: Failed on local exception: java.io.IOException:
> > org.apache.hadoop.security.AccessControlException: Client cannot
> > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is:
> > "cdh-datanode8/62.210.141.237"; destination host is:
> "cdh-namenode":8020;
> >
> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> >
> > ......
> >
> >
> >
> > Anyone knows how to solve this?
> >
> >
> > Qi FU
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message