nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martijn Dekkers" <mart...@dekkers.org.uk>
Subject Re: InvokeHTTP POST
Date Fri, 03 May 2019 06:52:32 GMT
Hi all,

This is a last-ditch attempt to resolve the issue of POSTing multipart/form-data using InvokeHTTP.
Our alternatives would be to either not using NiFi for this use-case - which would be a pity
- or writing our own processor that can properly submit forms. 

Our flow, NiFi 1.9.2:

Issues found so far:

[HandleHTTPRequest] - [ExtractText] - [ReplaceText] - [InvokeHTTP] - [HandleHTTPResponse]

ExtractText pulls the flowfile data into an attribute;

ReplaceText builds a message as per RFC2388 (tried several variations of this, including with/without
newlines in the expression editor):

--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“data_src”${literal(‘\r\n’)}
${literal(‘\r\n’)}
TST${literal(‘\r\n’)}
--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“cmd”${literal(‘\r\n’)}
${literal(‘\r\n’)}
sendResultDetails${literal(‘\r\n’)}
--------------------------555791708927339309556823${literal(‘\r\n’)}
Content-Disposition: form-data; name=“cmd_data”; filename=“${filename}“${literal(‘\r\n’)}
Content-Type: text/plain${literal(‘\r\n’)}${literal(‘\r\n’)}
${cmd_data}
${literal(‘\r\n’)}
--------------------------555791708927339309556823--${literal(‘\r\n’)}

InvokeHTTP POSTs message to httpbin.org for testing:
Set URL, etc.
Set Content-Type to: multipart/form-data; boundary=--------------------------555791708927339309556823
(We also tried setting Content-Type to text/plain and application/octet-stream, and adding
Content-Type: multipart/form-data; boundary=--------------------------555791708927339309556823
to the top of the body, none of these options worked)
Add "Accept: */*" Header

Where we use the Content-Type: multipart/form-data configuration, httpbin.org doesn't see
any data at all. Where we use a different Content-Type, httpdin.org just parses the complete
message body as a standard message body. 

If we build the exact same message with, for example, Postman, we receive valid results. Sending
both NiFi and Postman POSTs yields identical output when captured on a listening netcat port.


The only answer I have managed to find on the question of "How to POST form-data with InvokeHTTP
is "build your own message body". Most of these questions are met with silence. It would be
extremely helpful if anyone could comment on this actually being possible with InvokeHTTP.


Any feedback or assistance would be appreciated.

On Fri, 26 Apr 2019, at 13:27, Martijn Dekkers wrote:
> Hi,
> 
> I am working to implement a flow where we need to POST some data to another system using
multipart/form-data. This system requires that one of the form fields acts as an attached
file. Using curl, this works fine:
> 
> curl -F <field1>=<data1> -F <field2>=<data2> -F <field3>=@[filename]
<URL>
> 
> I created a flowfile that mimics what curl sends, including the boundaries etc. Using
InvokeHTTP POST I receive a web-server error from the receiving end that mentions a missing
parameter.
> 
> Running with curl --trace-ascii or POSTing to a running netcat with both curl and nifi
shows exactly the same results, with the exception of the actual boundary marker (which is
always different) and some minor header stuff (UA, etc.) The payload is simple text. 
> 
> I understand that there is some difference between sending the data and uploading a file,
but am unable to figure out what this is. for example, sending the same data with curl using
-F<field3>=<[filename] (instead of @[filename]) also results in an error. 
> 
> I have read a few different messages about POSTing multipart-form data where files need
to be uploaded, and most appear to end up with writing a scripted processor or invoking shell.
This is particularly undesirable for us given the volume of files we must process, and the
speed at which we should be uploading those files. 
> 
> Is there any way in which to mimic the uploading/attaching of a file, instead of sending
the data as text?
> 
> Thanks
> 
> Martijn

Mime
View raw message