nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karsten Zakarias <...@foss.dk>
Subject RE: Problems getting invokeHTTP/PutSQL to work
Date Wed, 21 Jun 2017 08:54:34 GMT
Hi Andy,

Thanks for your reply. This is how far I’ve gotten.

My invokeHTTP configuration looks like this:
[cid:image001.png@01D2EA7B.79DBF160]

My ContextService looks like this:
[cid:image002.png@01D2EA7B.79DBF160]

As my Apache Nifi is on an EC2 instance I ssh in and tried to locate the Java Runtime Environment
and this is what I found under /usr/lib/jvm. I don’t know if that is the correct location:
[cid:image006.png@01D2EA7B.B4DF8470]

This is the ContextService error:
[cid:image009.jpg@01D2EA7C.C1654710]

Where am I getting it wrong?


Best regards / Med venlig hilsen

Karsten Zakarias
Product Manager, M.Sc.
IT & Digital Business



[cid:image010.jpg@01D2EA7C.C1654710]
Phone:
Mobile:
E-mail:

+45 4820 8514
+45 2761 6920
KZA@foss.dk




[cid:image011.jpg@01D2EA7C.C1654710]

FOSS Analytical A/S - Foss Allé 1 DK-3400 Hillerød DK
________________________________
FOSS provides and supports dedicated, rapid and accurate analytical solutions, which analyze
and control the quality and production of agricultural and food products, for the enhancement
of our customers’ business and better food quality for consumers.


From: Andy LoPresto [mailto:alopresto@apache.org]
Sent: 20. juni 2017 15:37
To: users@nifi.apache.org
Subject: Re: Problems getting invokeHTTP/PutSQL to work

Hi Karsten,

Are you getting a specific error message when using InvokeHTTP or PutSQL? I don’t have an
Airtable account, but it looks like a pretty straightforward REST API over HTTPS. From your
screenshot, I believe you need to configure an SSLContextService in order to validate the
certificate presented by Airtable. You should configure it as shown below. The reason is that
NiFi by default does not trust any certificates. When it tries to make a request to Airtable,
Airtable tries to negotiate an encrypted TLS connection and NiFi attempts to verify the server’s
identity using the presented certificate chain. Without a “truststore” (a collection of
trusted certificates), it fails to do this. Java provides a default truststore to allow these
connections to succeed.

Steps for creating & linking SSLContextService:

1. From the Operate palette, click the gear icon (“Configuration”) and switch to the “Controller
Services” tab. You can also skip directly there from the InvokeHTTP properties tab by opening
the dropdown for “SSL Context Service” and selecting “Create new service…"
2. Create a new StandardSSLContextService. This controller service provides a common interface
to the TLS (nee SSL) settings that various components use.
3. You do not need to populate the first four values (the “keystore” values) unless Airtable
requires client authentication for TLS, which I highly doubt. Instead, populate the following
fields:
            Truststore Filename: <path to your Java Runtime Environment’s “cacerts”
file>
            Truststore Password: “changeit”
            Truststore Type: JKS
            SSL Protocol: TLS
4. Save the controller service and “enable” it by clicking the lightning icon on the far
right.
5. Once this service is enabled, your processor should be valid (as shown by a red “stopped”
icon rather than a yellow “caution” icon).
6. For debugging, I would also recommend the following settings:
            Put Response Body in Attribute: “response” <— This will put the response
body into an attribute so you can view it with a LogAttribute processor or by examining the
flowfile in the queue
            Always Output Response: true <— This will force a flowfile to be generated
even if the HTTP status is one that does not send a response body
7. You provided a custom request header with your API key, but you named it “Authentication”
instead of “Authorization”. You’ll need to change this to “Authorization”.
8. You may need to set the Content Type header to explicitly be “application/json”. The
mime type is correctly set coming out of the AttributesToJSON processor, so I doubt this is
the issue.

There may be an Airtable tool for viewing the incoming requests and determining why the request
body could not be parsed. If not, I suggest setting up a simple HTTP server on your local
device and temporarily pointing the request there (or copying the InvokeHTTP processor and
doing it in parallel) to debug the request. You can also use tools like Paw or Postman to
craft POST calls to Airtable manually until you are successful and compare to the output of
the InvokeHTTP processor — you can enable DEBUG level logging for it by modifying your $NIFI_HOME/conf/logback.xml
file and adding the line:

<logger name=“org.apache.nifi.processors.standard.InvokeHTTP" level="DEBUG"/>

Hope this helps. Please let us know if this solves your issues and if you encounter any further
trouble.


[cid:image005.png@01D2EA7B.79DBF160]


Andy LoPresto
alopresto@apache.org<mailto:alopresto@apache.org>
alopresto.apache@gmail.com<mailto:alopresto.apache@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 20, 2017, at 5:52 AM, Karsten Zakarias <KZA@foss.dk<mailto:KZA@foss.dk>>
wrote:

Hi all,

I’m new to Apache Nifi and having problems writing a POST/PUT request to an external db.
Any advice/links to material and prior threats would be greatly appreciated.

My setup is the following:
-          I have BatchIQ Apache Nifi running on an AWS EC2 instance.
-          A GetFTP processor ingests csv files
-          Each value is extracted using regex and combined into a JSON object using an AttributesToJSON
processor
-          I now want to use a InvokeHTTP to write the json object to an airtable db and a
PutSQL to write to a SQL db.

-          How do I setup InvokeHTTP correctly.

-          How do I configure PutSQL correctly on a remote instance with a database driver.


Thanks a lot in advance!
Karsten.


AttributesToJSON:
<image003.png>

InvokeHTTP:
<image001.png><image002.png>
<image004.png>


PutSQL:
<image006.png>
<image007.png>



Best regards / Med venlig hilsen

Karsten Zakarias
Product Manager, M.Sc.
IT & Digital Business



<image005.jpg>
Phone:
Mobile:
E-mail:

+45 4820 8514
+45 2761 6920
KZA@foss.dk<mailto:KZA@foss.dk>




<image008.jpg>

FOSS Analytical A/S - Foss Allé 1 DK-3400 Hillerød DK
________________________________
FOSS provides and supports dedicated, rapid and accurate analytical solutions, which analyze
and control the quality and production of agricultural and food products, for the enhancement
of our customers’ business and better food quality for consumers.

Mime
View raw message