nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Kafka - Hadoop Kerberos issue
Date Thu, 18 May 2017 13:10:14 GMT
Hi Arnaud,

Thanks for following up and providing the additional information.

I can in fact recreate the problem now, and it is specific to using
the PlainSaslServer provided by Kafka. Once you have started the Kafka
processor with that configuration, the PlainSaslServer is being
registered as a provider in a shared JVM class, and is then visible
when PutHDFS goes to access the providers. The Hadoop client code
expects to pass in a null Map to each provider, but the Kafka provider
doesn't check for null which then produces the NullPointerException
you saw.

I am continuing to investigate the best path forward for this
situation and will provide an update when I learn more.

One thing I wanted to mention is that using SASL_SSL with the
PlainSaslServer and sasl.mechanism=PLAIN, is not really Kerberos. Its
basically a simple username and password login over an SSL connection,
as far as I know its not going to a KDC to get a ticket.

This is my understanding from reading section 7.3 here
https://kafka.apache.org/documentation/#security_sasl -
Authenticating using SASL/Kerberos vs. Authenticating using SASL/Plain

The typical Kerberos setup would involve specifying a JAAS file that
uses Krb5LoginModule, like the following:

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  keyTab="/path/to/your.keytab"
  serviceName="kafka"
  principal="someprincipal@SOMEREALM.COM";
};

You then specify this in NiFi's bootstrap.conf like the following:

java.arg.16=-Djava.security.auth.login.config=/path/to/your/jaas.config

In Apache NiFi 1.2.0, this was made slightly easier and there are now
properties in the processors for Kerberos Principal and Kerberos
Keytab, and if you provide those the processor will dynamically create
a JAAS config like above which takes precedence.

Using either of these two approaches that use Krb5LoginModule would
avoid the issues with PutHDFS.

-Bryan


On Thu, May 18, 2017 at 3:33 AM, Arnaud G <greatpatton@gmail.com> wrote:
> mail was sent before completion:
>
>     Security Protocol: SASL_SSL
>     Kerberos Service Name:  myuser (please note that its mandatory to put
> something here or the process fail)
>     SSL Context Service: KafkaSSLcontext
>     sasl.jaas.config: org.apache.kafka.common.security.plain.PlainSaslServer
> require username="kauser" password="XXXXX";
>     sasl.mechanism: PLAIN
>
> This configuration is working fine we can consume the topics without any
> issue.
>
> 4) Here is the configuration of the PutHDFS :
>      Hadoop Configuration Ressources:
> /etc/hadoop/conf/hdfs-site.xml;/etc/hadoop/conf/core-site.xml
>      Kerberos Principal: user1@REALM.COM
>      Kerberos Keytab: /etc/security/keytabs/user1.keytab
>
>
> Additional classpath is empty.
>
> Please note that the Kafka is an external instance (and using SASL/PLAIN)
> and not using the same Kerberos configuration than HDFS. I suppose that if
> we were using the same Kerberos domain everything will be working fine.
>
> A.
>
>
> On Thu, May 18, 2017 at 9:22 AM, Arnaud G <greatpatton@gmail.com> wrote:
>>
>> Hi again,
>>
>> Here are more details:
>>
>> 1) We are redoing the test on a test cluster which is plain 1.2 vanilla
>> (upgraded from 1.1 but without any additional JAR, bootstrap is the package
>> version.
>> 2) The org.apache.kafka.common.security.plain.PlainSaslServer is coming
>> from our JAAS configuration and form understanding is part of the standard
>> 0.10 client library
>> 3) Here is the configuration of our ConsumeKafka_0_10:
>>
>>
>>
>>
>> On Wed, May 17, 2017 at 6:50 PM, Bryan Rosander <brosander@apache.org>
>> wrote:
>>>
>>> Hey guys,
>>>
>>> I also used a flow that has PublishKafka_0_10 and ConsumeKafka_0_10 as
>>> well as PutHDFS and ListHDFS all talking to a secured cluster.  I've tried
>>> various configurations of running vs stopped processors including restarting
>>> nifi in each configuration and can't seem to reproduce.
>>>
>>> It's strange that you're getting
>>> org.apache.kafka.common.security.plain.PlainSaslServer$PlainSaslServerFactory
>>> in the list of Sasl service factories.
>>>
>>> When I put a breakpoint at
>>> org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.<init>(SaslRpcServer.java:380)
>>> I see several factory instances but the closest to that kafka class is
>>> org.apache.hadoop.security.SaslPlainServer$SaslPlainServerFactory.
>>>
>>> Thanks,
>>> Bryan Rosander
>>>
>>> On Wed, May 17, 2017 at 12:28 PM, Bryan Bende <bbende@gmail.com> wrote:
>>>>
>>>> Hi Arnaud,
>>>>
>>>> I created a flow that had PublishKafka_0_10 running with dynamic JAAS
>>>> properties, and also PutHDFS and ListHDFS talking to a kerberized
>>>> HDFS, and I wasn't able to reproduce the above error. I also stopped
>>>> and started all of the processors in different orders, but they
>>>> continued to work.
>>>>
>>>> Is there any pattern for you that causes the problem? For example,
>>>> does it always happen if you start the Kafka processors first and then
>>>> the HDFS processors, and not the other way around?
>>>>
>>>> Also, can you confirm that your PutHDFS processor is not using the
>>>> "Additional Classpath Resources" property, and that there were no
>>>> additional JARs added to NiFi's lib directory?
>>>>
>>>> Just want to ensure nothing else is on the classpath of the PutHDFS
>>>> processor besides what would be expected.
>>>>
>>>> Thanks,
>>>>
>>>> Bryan
>>>>
>>>>
>>>> On Wed, May 17, 2017 at 10:24 AM, Arnaud G <greatpatton@gmail.com>
>>>> wrote:
>>>> > Hi,
>>>> >
>>>> > We didn't fully tested it in 1.1 as we were waiting for the dynamic
>>>> > load of
>>>> > JAAS in the new Kafka consumer/producer processor. We expected that
>>>> > loading
>>>> > a JAAS when starting Nifi may had impact on all Kerberos processes.
>>>> >
>>>> > Thanks,
>>>> >
>>>> > Arnaud
>>>> >
>>>> > On Wed, May 17, 2017 at 2:31 PM, Bryan Bende <bbende@gmail.com>
wrote:
>>>> >>
>>>> >> Hello,
>>>> >>
>>>> >> Thanks for reporting this, we definitely want to figure out what
is
>>>> >> going on here.
>>>> >>
>>>> >> Was this flow working fine before using Apache NiFi 1.1.x (or some
>>>> >> earlier version) and then stopped working when upgrading to 1.2.0?
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >> Bryan
>>>> >>
>>>> >>
>>>> >> On Wed, May 17, 2017 at 4:49 AM, Arnaud G <greatpatton@gmail.com>
>>>> >> wrote:
>>>> >> > Hi!
>>>> >> >
>>>> >> > We are currently facing an issue and we have not yet find a
>>>> >> > potential
>>>> >> > solution.
>>>> >> >
>>>> >> > We are running 1.2 and we have are facing an interaction between
>>>> >> > the
>>>> >> > Kafka
>>>> >> > 10.2 and PutHDFS processors.
>>>> >> >
>>>> >> > We are connecting to an external Kafka server that require
>>>> >> > SASL/PLAIN
>>>> >> > authentication. To enable this we are using the new Kafka processor
>>>> >> > with
>>>> >> > a
>>>> >> > JAAS configuration in the processor to authenticate, and this
is
>>>> >> > working
>>>> >> > fine.
>>>> >> >
>>>> >> > However on an unrelated flow the PutHDFS processor stopped
to work
>>>> >> > as
>>>> >> > the
>>>> >> > processor is now using the JAAS configuration and ignoring
the
>>>> >> > Kerberos
>>>> >> > configuration of the processor. (the setup are completely
>>>> >> > unrelated)
>>>> >> >
>>>> >> > We tried a lot of configuration but we have not yet find a
way to
>>>> >> > allow
>>>> >> > Nifi
>>>> >> > to authenticate to both Kafka 10.2 and HDFS at the same time,
it's
>>>> >> > either
>>>> >> > one or the other.
>>>> >> >
>>>> >> > Are we missing something? Is there a way to ensure that both
>>>> >> > processor
>>>> >> > keep
>>>> >> > their own configuration?
>>>> >> >
>>>> >> > Thanks!
>>>> >> >
>>>> >> >
>>>> >> > For reference PutHDFS failure log:
>>>> >> >
>>>> >> > 2017-05-16 15:27:01,199 ERROR [StandardProcessScheduler Thread-8]
>>>> >> > o.apache.nifi.processors.hadoop.PutHDFS
>>>> >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239]
>>>> >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] failed to
invoke
>>>> >> > @OnScheduled method due to java.lang.RuntimeException: Failed
while
>>>> >> > executing one of processor's OnScheduled task.; processor will
not
>>>> >> > be
>>>> >> > scheduled to run for 30 seconds: java.lang.RuntimeException:
Failed
>>>> >> > while
>>>> >> > executing one of processor's OnScheduled task.
>>>> >> >
>>>> >> > java.lang.RuntimeException: Failed while executing one of
>>>> >> > processor's
>>>> >> > OnScheduled task.
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> >> >
>>>> >> >         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> >> >
>>>> >> >         at java.lang.Thread.run(Thread.java:745)
>>>> >> >
>>>> >> > Caused by: java.util.concurrent.ExecutionException:
>>>> >> > java.lang.reflect.InvocationTargetException
>>>> >> >
>>>> >> >         at
>>>> >> > java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>>> >> >
>>>> >> >         at java.util.concurrent.FutureTask.get(FutureTask.java:206)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
>>>> >> >
>>>> >> >         ... 9 common frames omitted
>>>> >> >
>>>> >> > Caused by: java.lang.reflect.InvocationTargetException: null
>>>> >> >
>>>> >> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> >> > Method)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> >> >
>>>> >> >         at java.lang.reflect.Method.invoke(Method.java:498)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1305)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1301)
>>>> >> >
>>>> >> >         ... 6 common frames omitted
>>>> >> >
>>>> >> > Caused by: java.lang.NullPointerException: null
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.kafka.common.security.plain.PlainSaslServer$PlainSaslServerFactory.getMechanismNames(PlainSaslServer.java:162)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.<init>(SaslRpcServer.java:380)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> > org.apache.hadoop.security.SaslRpcServer.init(SaslRpcServer.java:184)
>>>> >> >
>>>> >> >         at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:577)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:418)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:314)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy(ConfiguredFailoverProxyProvider.java:124)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64)
>>>> >> >
>>>> >> >         at
>>>> >> > org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:181)
>>>> >> >
>>>> >> >         at
>>>> >> > org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
>>>> >> >
>>>> >> >         at
>>>> >> > org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
>>>> >> >
>>>> >> >         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>>> >> >
>>>> >> >         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(AbstractHadoopProcessor.java:304)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor$1.run(AbstractHadoopProcessor.java:301)
>>>> >> >
>>>> >> >         at java.security.AccessController.doPrivileged(Native
>>>> >> > Method)
>>>> >> >
>>>> >> >         at javax.security.auth.Subject.doAs(Subject.java:422)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.getFileSystemAsUser(AbstractHadoopProcessor.java:301)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.resetHDFSResources(AbstractHadoopProcessor.java:268)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> >
>>>> >> > org.apache.nifi.processors.hadoop.AbstractHadoopProcessor.abstractOnScheduled(AbstractHadoopProcessor.java:200)
>>>> >> >
>>>> >> >         at
>>>> >> >
>>>> >> > org.apache.nifi.processors.hadoop.PutHDFS.onScheduled(PutHDFS.java:191)
>>>> >> >
>>>> >> >         ... 16 common frames omitted
>>>> >> >
>>>> >> >
>>>> >
>>>> >
>>>
>>>
>>
>

Mime
View raw message