nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alfredo De Luca <alfredo.del...@gmail.com>
Subject Re: nifi issue with sftp server
Date Mon, 12 Nov 2018 09:10:50 GMT
Hi all. The error was caused neither from NIFI or sftp server but on the
freeipa client side running on the sftp server.
thanks all


On Mon, Nov 5, 2018 at 9:11 AM Alfredo De Luca <alfredo.deluca@gmail.com>
wrote:

> Hi Joe.. have you had any thought about this? I don't know what else to do.
> Cheers
>
> On Tue, Oct 30, 2018 at 8:18 PM Alfredo De Luca <alfredo.deluca@gmail.com>
> wrote:
>
>> Any thought Joe? thank  so much
>>
>> On Tue, Oct 30, 2018 at 2:28 PM Alfredo De Luca <alfredo.deluca@gmail.com>
>> wrote:
>>
>>> I have attached the nifi properties. Also these are more info ...
>>>
>>> - ListSftp is running on primary node of the cluster and with 1
>>> concurrent tasks
>>> - nifi is in cluster with 3 nodes (pod in kubernetes)
>>> ---
>>> *Loading of nifi*
>>> how many threads does nifi show are running?
>>> *150 in average*
>>> ---
>>> How often does the problem happen?  How often does the connection/flow
>>> work?
>>>
>>> *Often, some processors does not list any file on sftp server because of
>>> this problem. Other Processors could list file on server sftp but at
>>> certain time listing fails.*
>>> *In addition sometimes fails also the unique fetchSftp processor*
>>>
>>> SFTP Server
>>> 1 CPU
>>> 2Gb Ram
>>> Load avg. load average: 0.20, 0.27, 0.31
>>>
>>> Cheers
>>>
>>>
>>>
>>> On Mon, Oct 29, 2018 at 5:41 PM Alfredo De Luca <
>>> alfredo.deluca@gmail.com> wrote:
>>>
>>>> Thanks Joe. Appreciated. I will get back to you asap with the info.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Mon, Oct 29, 2018 at 5:26 PM Joe Witt <joe.witt@gmail.com> wrote:
>>>>
>>>>> Alfredo
>>>>>
>>>>> There are quite a few details that will be needed to help folks guess
>>>>> at possible issues.
>>>>>
>>>>> Configuration of the machine running nifi
>>>>>   how many cores?, etc..
>>>>>
>>>>> Configuration of nifi
>>>>>   you shared common properties from nifi.properties which is good.
>>>>>   how many threads does the flow controller have?
>>>>>   how many processors are on the graph?
>>>>>   How many threads does ListSFTP have?
>>>>>   Is this a NiFi cluster or single node?
>>>>>
>>>>> Loading of the system running nifi
>>>>>   cpu load avg
>>>>>   garbage collection behavior in the jVM?
>>>>>
>>>>> Loading of nifi
>>>>>   how many threads does nifi show are running?
>>>>>
>>>>> Similar info for the SFTP server side
>>>>>   System error 4 could mean a lot of things.  Is that system taxed and
>>>>> unable to establish connections at times?
>>>>>
>>>>> How often does the problem happen?  How often does the connection/flow
>>>>> work?
>>>>>
>>>>> Have you tried manually connecting the way nifi would by running
>>>>> commands like 'sftp user@host.sftp.server -vvv' repeatedly to see if
>>>>> you can reproduce the issue?
>>>>>
>>>>> Does a file system get mounted for the sftp user and periodically is
>>>>> this failing?
>>>>>
>>>>> Unfortunately it could be a lot of things so we'll probably need to
>>>>> unpack a lot of info to get there.
>>>>>
>>>>> Thanks
>>>>> On Mon, Oct 29, 2018 at 12:14 PM Alfredo De Luca
>>>>> <alfredo.deluca@gmail.com> wrote:
>>>>> >
>>>>> > hi all. any idea? I am trying everything..but not sure what cause
>>>>> this error.
>>>>> >
>>>>> > Cheers
>>>>> >
>>>>> > On Mon, Oct 29, 2018 at 12:34 PM Alfredo De Luca <
>>>>> alfredo.deluca@gmail.com> wrote:
>>>>> >>
>>>>> >> nifi.swap.in.threads=1
>>>>> >> nifi.swap.out.threads=4
>>>>> >> nifi.cluster.node.protocol.threads=25
>>>>> >> nifi.cluster.node.protocol.max.threads=50
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Mon, Oct 29, 2018 at 12:28 PM Alfredo De Luca <
>>>>> alfredo.deluca@gmail.com> wrote:
>>>>> >>>
>>>>> >>> anyway...this is the error
>>>>> >>> 2018-10-29 11:21:06,768 ERROR [Timer-Driven Process Thread-13]
>>>>> SimpleProcessLogger.java:254
>>>>> ListSFTP[id=766ac418-27ce-335a-97c9-8823f2cf5a96] Failed to perform listing
>>>>> on remote host due to java.io.IOException: Failed to obtain connection
to
>>>>> remote host due to com.jcraft.jsch.JSchException: Auth fail: {}
>>>>> >>>
>>>>> >>>
>>>>> >>> On Mon, Oct 29, 2018 at 12:27 PM Alfredo De Luca <
>>>>> alfredo.deluca@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> Hi Joe.
>>>>> >>>> Sftp server allows 200 connections... not sure about
the nifi. I
>>>>> ll check
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Mon, Oct 29, 2018 at 12:21 PM Joe Witt <joe.witt@gmail.com>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> how many connections does your sftp server allow
at once and how
>>>>> many threads is nifi given for communication?
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On Mon, Oct 29, 2018, 7:10 AM Alfredo De Luca <
>>>>> alfredo.deluca@gmail.com> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Hi all. We have some issue with  our nifi (v.
1.6) where
>>>>> occasionally we have an error saying ...
>>>>> >>>>>> Auth fail.
>>>>> >>>>>>
>>>>> >>>>>> On the sftp server side we get
>>>>> >>>>>>
>>>>> >>>>>> Oct 29 11:48:06 sftp sshd[13845]: pam_sss(sshd:account):
Access
>>>>> denied for user <sftp user>: 4 (System error)
>>>>> >>>>>>
>>>>> >>>>>> Any idea?
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> --
>>>>> >>>>>> Alfredo
>>>>> >>>>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> Alfredo
>>>>> >>>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Alfredo
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Alfredo
>>>>> >>
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Alfredo
>>>>> >
>>>>>
>>>>
>>>>
>>>> --
>>>> *Alfredo*
>>>>
>>>>
>>>
>>> --
>>> *Alfredo*
>>>
>>>
>>
>> --
>> *Alfredo*
>>
>>
>
> --
> *Alfredo*
>
>

-- 
*Alfredo*

Mime
View raw message