nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <>
Subject Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow
Date Thu, 03 May 2018 20:18:34 GMT

can you share more about the context in which you were told it would
be a bad idea?  There might be an important clarifying point there.
The pattern is certainly useful so we want to get the right messaging
around it.


On Thu, May 3, 2018 at 4:17 PM, Shawn Weeks <> wrote:
> The rest api I'm building is primarily for generating data extracts using
> curl or wget and doesn't have the performance requirements that an Angular
> Rest UI would require. I haven't had any trouble syncing requests and
> responses as the Handle HTTP Request and Response Processors manage that
> with an HTTP Context Map. You need to make sure you don't lose the attribute
> it's using and you can't return multiple responses to the same request as
> the first response closes the request.
> Thanks
> Shawn
> ________________________________
> From: Boris Tyukin <>
> Sent: Thursday, May 3, 2018 2:57:37 PM
> To:
> Subject: Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow
> Shawn, I am not answering your question but I am curious how you would see
> logistics of  HTTP Request and Response Processors. Is your intent to build
> REST api entirely in NiFi? I asked a question here before as I had a similar
> idea but was told it was not a good strategy as it would be difficult to
> sync request/response.
> On Thu, May 3, 2018 at 11:35 AM, Shawn Weeks <>
> wrote:
> I'm building a rest service with the HTTP Request and Response Processors to
> support data extracts from Hive. Since some of the extracts can be quiet
> large using the SelectHiveQL Processor isn't a performant option and instead
> I'm trying to use on demand Hive Temporary Tables to do the heavy lifting
> via CTAS(Create Table as Select). Since GetHDFS doesn't support an incoming
> connection I'm trying to figure out another way to fetch the files Hive
> creates and return them as a download in the web service. Has anyone else
> worked out a good solution for fetching the contents of a directory from
> HDFS as a part of larger flow?
> Thanks
> Shawn

View raw message