samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <chen.song...@gmail.com>
Subject Re: Security on YARN
Date Sat, 25 Jul 2015 02:00:51 GMT
Can someone give some context on this? I can volunteer myself and try
working on this.

Chen

On Thu, Jul 2, 2015 at 4:29 AM, Qi Fu <qfu@talend.com> wrote:

> Hi Yi & Yan,
>
> Many thanks for your information. I have created a jira for this:
> https://issues.apache.org/jira/browse/SAMZA-727
> I'm willing to test it if someone can work on this.
>
>
> -Qi
>
> ________________________________________
> From: Yi Pan <nickpan47@gmail.com>
> Sent: Thursday, July 2, 2015 1:38 AM
> To: dev@samza.apache.org
> Subject: Re: Security on YARN
>
> Hi, Yan,
>
> Your memory serves as well as mine. :) I remember that Chris and I
> discussed this Kerberos ticket expiration issue when we were brain storming
> on how to access HDFS data in Samza. At high-level, what happens is that
> the Kerberos ticket to access a secured Hadoop cluster is issued to Samza
> containers at the job start time, and will expire later. For a long-running
> Samza job, it does not work. We will need a way to refresh the Kerberos
> ticket periodically, which is not supported yet. Chris probably can chime
> in with more details.
>
> -Yi
>
> On Wed, Jul 1, 2015 at 4:08 PM, Yan Fang <yanfang724@gmail.com> wrote:
>
> > Hi Qi,
> >
> > I think this is caused by the fact that Samza currently does not support
> > Yarn with Kerberos. Feel free to open a ticket for this feature.
> >
> > But if my memory serves, there was an issue mentioned about the Kerberos.
> > Seems when the Kerberos ticket expires, Samza will have some issues? Can
> > not find the resource. Anyone remember this?
> >
> > Cheers,
> >
> > Fang, Yan
> > yanfang724@gmail.com
> >
> > On Wed, Jul 1, 2015 at 3:41 AM, Qi Fu <qfu@talend.com> wrote:
> >
> > > Hi all,
> > >
> > >
> > > I'm testing Samza on YARN and I have encountered a problem on the
> > security
> > > setting of YARN (Kerberos). Here is the detail:
> > >
> > > 1. My cluster is secured by Kerberos, and I deploy my samza job from
> one
> > > of the cluster.
> > >
> > >
> > > 2. My config file is in ~/.samza/conf/(yarn-site.xml, core-site.xml,
> > > hdfs-site.xml)
> > >
> > >
> > > 3. The job is deployed successfully, and I can get the info such as:
> > >
> > >     ClientHelper [INFO] set package url to scheme: "hdfs" port: -1
> file:
> > > "/user/test/samzatest.tar.gz" for application_1435680272316_0003
> > >
> > >     ClientHelper [INFO] set package size to 212924524 for
> > > application_1435680272316_0003
> > >
> > >
> > >
> > >     I think the security setting is correct as it can get the file size
> > > from HDFS.
> > >
> > >
> > > 4. But I get the error from YARN job manager as following:
> > >
> > >
> > >     Application application_1435680272316_0003 failed 2 times due to AM
> > > Container for appattempt_1435680272316_0003_000002 exited with
> exitCode:
> > > -1000
> > >
> > > For more detailed output, check application tracking page:
> > > http://cdh-namenode:8088/proxy/application_1435680272316_0003/Then,
> > click
> > > on links to logs of each attempt.
> > >
> > > Diagnostics: Failed on local exception: java.io.IOException:
> > > org.apache.hadoop.security.AccessControlException: Client cannot
> > > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is:
> > > "talend-cdh-datanode8/62.210.141.237"; destination host is:
> > > "talend-cdh-namenode":8020;
> > >
> > > java.io.IOException: Failed on local exception: java.io.IOException:
> > > org.apache.hadoop.security.AccessControlException: Client cannot
> > > authenticate via:[TOKEN, KERBEROS]; Host Details : local host is:
> > > "cdh-datanode8/62.210.141.237"; destination host is:
> > "cdh-namenode":8020;
> > >
> > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> > >
> > > ......
> > >
> > >
> > >
> > > Anyone knows how to solve this?
> > >
> > >
> > > Qi FU
> > >
> >
>



-- 
Chen Song

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message