nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alfredo De Luca <alfredo.del...@gmail.com>
Subject Re: nifi issue with sftp server
Date Mon, 05 Nov 2018 08:11:30 GMT
Hi Joe.. have you had any thought about this? I don't know what else to do.
Cheers

On Tue, Oct 30, 2018 at 8:18 PM Alfredo De Luca <alfredo.deluca@gmail.com>
wrote:

> Any thought Joe? thank  so much
>
> On Tue, Oct 30, 2018 at 2:28 PM Alfredo De Luca <alfredo.deluca@gmail.com>
> wrote:
>
>> I have attached the nifi properties. Also these are more info ...
>>
>> - ListSftp is running on primary node of the cluster and with 1
>> concurrent tasks
>> - nifi is in cluster with 3 nodes (pod in kubernetes)
>> ---
>> *Loading of nifi*
>> how many threads does nifi show are running?
>> *150 in average*
>> ---
>> How often does the problem happen?  How often does the connection/flow
>> work?
>>
>> *Often, some processors does not list any file on sftp server because of
>> this problem. Other Processors could list file on server sftp but at
>> certain time listing fails.*
>> *In addition sometimes fails also the unique fetchSftp processor*
>>
>> SFTP Server
>> 1 CPU
>> 2Gb Ram
>> Load avg. load average: 0.20, 0.27, 0.31
>>
>> Cheers
>>
>>
>>
>> On Mon, Oct 29, 2018 at 5:41 PM Alfredo De Luca <alfredo.deluca@gmail.com>
>> wrote:
>>
>>> Thanks Joe. Appreciated. I will get back to you asap with the info.
>>>
>>> Cheers
>>>
>>>
>>> On Mon, Oct 29, 2018 at 5:26 PM Joe Witt <joe.witt@gmail.com> wrote:
>>>
>>>> Alfredo
>>>>
>>>> There are quite a few details that will be needed to help folks guess
>>>> at possible issues.
>>>>
>>>> Configuration of the machine running nifi
>>>>   how many cores?, etc..
>>>>
>>>> Configuration of nifi
>>>>   you shared common properties from nifi.properties which is good.
>>>>   how many threads does the flow controller have?
>>>>   how many processors are on the graph?
>>>>   How many threads does ListSFTP have?
>>>>   Is this a NiFi cluster or single node?
>>>>
>>>> Loading of the system running nifi
>>>>   cpu load avg
>>>>   garbage collection behavior in the jVM?
>>>>
>>>> Loading of nifi
>>>>   how many threads does nifi show are running?
>>>>
>>>> Similar info for the SFTP server side
>>>>   System error 4 could mean a lot of things.  Is that system taxed and
>>>> unable to establish connections at times?
>>>>
>>>> How often does the problem happen?  How often does the connection/flow
>>>> work?
>>>>
>>>> Have you tried manually connecting the way nifi would by running
>>>> commands like 'sftp user@host.sftp.server -vvv' repeatedly to see if
>>>> you can reproduce the issue?
>>>>
>>>> Does a file system get mounted for the sftp user and periodically is
>>>> this failing?
>>>>
>>>> Unfortunately it could be a lot of things so we'll probably need to
>>>> unpack a lot of info to get there.
>>>>
>>>> Thanks
>>>> On Mon, Oct 29, 2018 at 12:14 PM Alfredo De Luca
>>>> <alfredo.deluca@gmail.com> wrote:
>>>> >
>>>> > hi all. any idea? I am trying everything..but not sure what cause
>>>> this error.
>>>> >
>>>> > Cheers
>>>> >
>>>> > On Mon, Oct 29, 2018 at 12:34 PM Alfredo De Luca <
>>>> alfredo.deluca@gmail.com> wrote:
>>>> >>
>>>> >> nifi.swap.in.threads=1
>>>> >> nifi.swap.out.threads=4
>>>> >> nifi.cluster.node.protocol.threads=25
>>>> >> nifi.cluster.node.protocol.max.threads=50
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Mon, Oct 29, 2018 at 12:28 PM Alfredo De Luca <
>>>> alfredo.deluca@gmail.com> wrote:
>>>> >>>
>>>> >>> anyway...this is the error
>>>> >>> 2018-10-29 11:21:06,768 ERROR [Timer-Driven Process Thread-13]
>>>> SimpleProcessLogger.java:254
>>>> ListSFTP[id=766ac418-27ce-335a-97c9-8823f2cf5a96] Failed to perform listing
>>>> on remote host due to java.io.IOException: Failed to obtain connection to
>>>> remote host due to com.jcraft.jsch.JSchException: Auth fail: {}
>>>> >>>
>>>> >>>
>>>> >>> On Mon, Oct 29, 2018 at 12:27 PM Alfredo De Luca <
>>>> alfredo.deluca@gmail.com> wrote:
>>>> >>>>
>>>> >>>> Hi Joe.
>>>> >>>> Sftp server allows 200 connections... not sure about the
nifi. I
>>>> ll check
>>>> >>>>
>>>> >>>>
>>>> >>>> On Mon, Oct 29, 2018 at 12:21 PM Joe Witt <joe.witt@gmail.com>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> how many connections does your sftp server allow at
once and how
>>>> many threads is nifi given for communication?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Mon, Oct 29, 2018, 7:10 AM Alfredo De Luca <
>>>> alfredo.deluca@gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> Hi all. We have some issue with  our nifi (v. 1.6)
where
>>>> occasionally we have an error saying ...
>>>> >>>>>> Auth fail.
>>>> >>>>>>
>>>> >>>>>> On the sftp server side we get
>>>> >>>>>>
>>>> >>>>>> Oct 29 11:48:06 sftp sshd[13845]: pam_sss(sshd:account):
Access
>>>> denied for user <sftp user>: 4 (System error)
>>>> >>>>>>
>>>> >>>>>> Any idea?
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> --
>>>> >>>>>> Alfredo
>>>> >>>>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> Alfredo
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Alfredo
>>>> >>>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Alfredo
>>>> >>
>>>> >
>>>> >
>>>> > --
>>>> > Alfredo
>>>> >
>>>>
>>>
>>>
>>> --
>>> *Alfredo*
>>>
>>>
>>
>> --
>> *Alfredo*
>>
>>
>
> --
> *Alfredo*
>
>

-- 
*Alfredo*

Mime
View raw message