nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ed B <bdes...@gmail.com>
Subject Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow
Date Mon, 07 May 2018 09:13:35 GMT
Shawn, Bryan,
If access to the files on HDFS is required, there is PR for that. Not
reviewed yet though:
https://github.com/apache/nifi/pull/2639

Shawn, let me know if that is what you are looking for.

Regards,
Ed

On Fri, May 4, 2018, 9:38 AM Boris Tyukin <boris@boristyukin.com> wrote:

> good to know, thanks Bryan. My use case is similar to Shawn's when
> external consumers can request data from engines like Hive or Impala. I
> will play with it a bit more. I guess another problem would be a pagination
> of the response payload.
>
> On Fri, May 4, 2018 at 9:18 AM, Bryan Bende <bbende@gmail.com> wrote:
>
>> I think the point of the original thread was that building a REST
>> service in NiFi wasn't necessarily meant to be used as the back-end of
>> a full-blown web application that is used by hundreds/thousands of
>> users.
>>
>> However there are still plenty of situations where it is certainly
>> fine, especially for creating a mechanism for system-to-system
>> communication where there was no previous way to access the data
>> easily, and now it can be wrapped in a REST API for some other system
>> to call.
>>
>> On Thu, May 3, 2018 at 9:28 PM, Boris Tyukin <boris@boristyukin.com>
>> wrote:
>> > Hi Joe,
>> >
>> > Here is that thread. I just read it again and I probably asked a pretty
>> > broad question though but I also wanted to create a rest endpoint
>> through
>> > nifi to pull data from persistent storage
>> >
>> >
>> http://apache-nifi.1125220.n5.nabble.com/NiFi-as-a-Web-Services-server-td21938.html
>> >
>> > On Thu, May 3, 2018, 16:18 Joe Witt <joe.witt@gmail.com> wrote:
>> >>
>> >> Boris,
>> >>
>> >> can you share more about the context in which you were told it would
>> >> be a bad idea?  There might be an important clarifying point there.
>> >> The pattern is certainly useful so we want to get the right messaging
>> >> around it.
>> >>
>> >> Thanks
>> >>
>> >> On Thu, May 3, 2018 at 4:17 PM, Shawn Weeks <sweeks@weeksconsulting.us
>> >
>> >> wrote:
>> >> > The rest api I'm building is primarily for generating data extracts
>> >> > using
>> >> > curl or wget and doesn't have the performance requirements that an
>> >> > Angular
>> >> > Rest UI would require. I haven't had any trouble syncing requests and
>> >> > responses as the Handle HTTP Request and Response Processors manage
>> that
>> >> > with an HTTP Context Map. You need to make sure you don't lose the
>> >> > attribute
>> >> > it's using and you can't return multiple responses to the same
>> request
>> >> > as
>> >> > the first response closes the request.
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Shawn
>> >> >
>> >> > ________________________________
>> >> > From: Boris Tyukin <boris@boristyukin.com>
>> >> > Sent: Thursday, May 3, 2018 2:57:37 PM
>> >> > To: users@nifi.apache.org
>> >> > Subject: Re: Fetch Contents of HDFS Directory as a Part of a Larger
>> Flow
>> >> >
>> >> > Shawn, I am not answering your question but I am curious how you
>> would
>> >> > see
>> >> > logistics of  HTTP Request and Response Processors. Is your intent
to
>> >> > build
>> >> > REST api entirely in NiFi? I asked a question here before as I had
a
>> >> > similar
>> >> > idea but was told it was not a good strategy as it would be
>> difficult to
>> >> > sync request/response.
>> >> >
>> >> > On Thu, May 3, 2018 at 11:35 AM, Shawn Weeks <
>> sweeks@weeksconsulting.us>
>> >> > wrote:
>> >> >
>> >> > I'm building a rest service with the HTTP Request and Response
>> >> > Processors to
>> >> > support data extracts from Hive. Since some of the extracts can be
>> quiet
>> >> > large using the SelectHiveQL Processor isn't a performant option and
>> >> > instead
>> >> > I'm trying to use on demand Hive Temporary Tables to do the heavy
>> >> > lifting
>> >> > via CTAS(Create Table as Select). Since GetHDFS doesn't support an
>> >> > incoming
>> >> > connection I'm trying to figure out another way to fetch the files
>> Hive
>> >> > creates and return them as a download in the web service. Has anyone
>> >> > else
>> >> > worked out a good solution for fetching the contents of a directory
>> from
>> >> > HDFS as a part of larger flow?
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Shawn
>> >> >
>> >> >
>>
>
>

Mime
View raw message