nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Braddy <rbra...@softnas.com>
Subject RE: New to NiFi - Remote Process Group failing to connect due to "magic header" not present
Date Sun, 16 Aug 2015 01:43:14 GMT
This issue is resolved.  Turned out to be a duplicate IP address on our internal network causing
unpredictable behavior.  NiFi is working as expected now.

Thanks for the quick replies today.

Rick

From: Rick Braddy
Sent: Saturday, August 15, 2015 5:20 PM
To: users@nifi.apache.org
Subject: RE: New to NiFi - Remote Process Group failing to connect due to "magic header" not
present

Sure.  Attached zip file contains log files on the target node.

I have also observed some occasional Putty disconnects from the sending side’s terminal
connection (Remote Process Group’s VM), which makes me wonder if there may be a networking
issue with it, so may not be a problem with NiFi at all.

One other question I have.  When the network connection stayed up, it was able to transfer
the two 1 GB files and one 10 GB file from source node to target node; however, it appears
these files get “queued” for a long period of time (showing up in the connection between
the Input Connector processor and the PutFile processor.  As these are not “streamed”,
I assume it’s just taking time to copy all that data around and is to be expected.

Is there a better way to “stream” from GetFile ==> Remote Process Group ==> Input
Connetor ==> PutFile (or something equivalent)?

From: Aldrin Piri [mailto:aldrinpiri@gmail.com]
Sent: Saturday, August 15, 2015 4:37 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: New to NiFi - Remote Process Group failing to connect due to "magic header" not
present

Rick,

Timeouts certainly aren't an expected behavior.  Might you have some logs from your remote
receiver that is receiving the files that we could take a look at?

It looks like the connection is functional in part, as one item did at least make the transfer.

Thanks!

On Sat, Aug 15, 2015 at 5:21 PM, Rick Braddy <rbraddy@softnas.com<mailto:rbraddy@softnas.com>>
wrote:
That appeared at first to resolve the connection problem, as I could then see and connect
to the remote input connector via the Remote Process Group and my basic file transfer flow
worked.

However, now there are timeout warnings – assume this is not normal.

[cid:image001.png@01D0D79B.00798110]

From: Aldrin Piri [mailto:aldrinpiri@gmail.com<mailto:aldrinpiri@gmail.com>]
Sent: Saturday, August 15, 2015 4:06 PM

To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: New to NiFi - Remote Process Group failing to connect due to "magic header" not
present

Rick,

Site to Site works by talking to the port that the NiFi web tier is running on and not the
configured "nifi.remote.input.socket.port" which is used after the initial handshaking and
connection.  This is likely why you are receiving the messages about the error of the magic
header.  It is receiving something from that socket, but not the desired input.

Make a remote processing group that points to port 8080 (assuming this was left as the default)
of your other instance and you should be good to go.

Please let us know if that is not the case.

On Sat, Aug 15, 2015 at 4:57 PM, Rick Braddy <rbraddy@softnas.com<mailto:rbraddy@softnas.com>>
wrote:
Hi Aldrin,

Here are the property settings:

# Site to Site properties
nifi.remote.input.socket.port=8081
nifi.remote.input.secure=false

Referencing remote node via http://<IP>:8081/nifi<http://%3cIP%3e:8081/nifi> (not
the UI address, but the separate site-to-site listener, which I see via netstat on target
node)

Rick

From: Aldrin Piri [mailto:aldrinpiri@gmail.com<mailto:aldrinpiri@gmail.com>]
Sent: Saturday, August 15, 2015 3:48 PM
To: users@nifi.apache.org<mailto:users@nifi.apache.org>
Subject: Re: New to NiFi - Remote Process Group failing to connect due to "magic header" not
present

Rick,

Welcome to the community!

We seem to be a little short in the documentation department for how to make use of Remote
Process Groups, but will look to remedy that.

Just to confirm a few settings, both your nodes have a nifi.remote.input.socket.port set and
each has nifi.remote.input.secure set to false within your nifi.properties.

From here, are you referencing the remote node via its UI address?  Out of the box, this would
be <server FQDN/IP>:8080/nifi

Thanks,
Aldrin

On Sat, Aug 15, 2015 at 4:11 PM, Rick Braddy <rbraddy@softnas.com<mailto:rbraddy@softnas.com>>
wrote:
Hi,

I’m new to NiFi, trying to get my first Remote Process Group configured and working between
two CentOS nodes.  For expediency, I have configured site-to-site port to 8081 and set secure
to false (to avoid dealing with SSL certificate setup for now – will get to that later).

Trying to get two nodes to communicate using Remote Process Group.  Google is not finding
any useful examples of how to set this up and get it to work, so kind of fumbling through
it today (learning a lot, but slow going).

I found the nifil-app.log file and why the Remote Process Group is failing to connect to the
second node.  Not sure why the “Magic Header” isn’t right, but connections are being
closed and getting Read Timeouts on the RPG sending node – the receiving NiFi node is closing
the connection because it thinks the sender isn’t a valid NiFi node due to missing/incorrect
magic header.

2015-08-15 13:28:08,939 ERROR [Site-to-Site Worker Thread-27] o.a.nifi.remote.SocketRemoteSiteListener
Unable to communicate with remote instance null due to org.apache.nifi.remote.exception.HandshakeException:
Handshake with nifi://SoftNAS-RGB1:57336 failed because the Magic Header was not present;
closing connection

Not sure where to go from here to resolve this.

Rick



Mime
View raw message