nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: Fetch Contents of HDFS Directory as a Part of a Larger Flow
Date Fri, 04 May 2018 13:36:42 GMT
good to know, thanks Bryan. My use case is similar to Shawn's when external
consumers can request data from engines like Hive or Impala. I will play
with it a bit more. I guess another problem would be a pagination of the
response payload.

On Fri, May 4, 2018 at 9:18 AM, Bryan Bende <bbende@gmail.com> wrote:

> I think the point of the original thread was that building a REST
> service in NiFi wasn't necessarily meant to be used as the back-end of
> a full-blown web application that is used by hundreds/thousands of
> users.
>
> However there are still plenty of situations where it is certainly
> fine, especially for creating a mechanism for system-to-system
> communication where there was no previous way to access the data
> easily, and now it can be wrapped in a REST API for some other system
> to call.
>
> On Thu, May 3, 2018 at 9:28 PM, Boris Tyukin <boris@boristyukin.com>
> wrote:
> > Hi Joe,
> >
> > Here is that thread. I just read it again and I probably asked a pretty
> > broad question though but I also wanted to create a rest endpoint through
> > nifi to pull data from persistent storage
> >
> > http://apache-nifi.1125220.n5.nabble.com/NiFi-as-a-Web-
> Services-server-td21938.html
> >
> > On Thu, May 3, 2018, 16:18 Joe Witt <joe.witt@gmail.com> wrote:
> >>
> >> Boris,
> >>
> >> can you share more about the context in which you were told it would
> >> be a bad idea?  There might be an important clarifying point there.
> >> The pattern is certainly useful so we want to get the right messaging
> >> around it.
> >>
> >> Thanks
> >>
> >> On Thu, May 3, 2018 at 4:17 PM, Shawn Weeks <sweeks@weeksconsulting.us>
> >> wrote:
> >> > The rest api I'm building is primarily for generating data extracts
> >> > using
> >> > curl or wget and doesn't have the performance requirements that an
> >> > Angular
> >> > Rest UI would require. I haven't had any trouble syncing requests and
> >> > responses as the Handle HTTP Request and Response Processors manage
> that
> >> > with an HTTP Context Map. You need to make sure you don't lose the
> >> > attribute
> >> > it's using and you can't return multiple responses to the same request
> >> > as
> >> > the first response closes the request.
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Shawn
> >> >
> >> > ________________________________
> >> > From: Boris Tyukin <boris@boristyukin.com>
> >> > Sent: Thursday, May 3, 2018 2:57:37 PM
> >> > To: users@nifi.apache.org
> >> > Subject: Re: Fetch Contents of HDFS Directory as a Part of a Larger
> Flow
> >> >
> >> > Shawn, I am not answering your question but I am curious how you would
> >> > see
> >> > logistics of  HTTP Request and Response Processors. Is your intent to
> >> > build
> >> > REST api entirely in NiFi? I asked a question here before as I had a
> >> > similar
> >> > idea but was told it was not a good strategy as it would be difficult
> to
> >> > sync request/response.
> >> >
> >> > On Thu, May 3, 2018 at 11:35 AM, Shawn Weeks <
> sweeks@weeksconsulting.us>
> >> > wrote:
> >> >
> >> > I'm building a rest service with the HTTP Request and Response
> >> > Processors to
> >> > support data extracts from Hive. Since some of the extracts can be
> quiet
> >> > large using the SelectHiveQL Processor isn't a performant option and
> >> > instead
> >> > I'm trying to use on demand Hive Temporary Tables to do the heavy
> >> > lifting
> >> > via CTAS(Create Table as Select). Since GetHDFS doesn't support an
> >> > incoming
> >> > connection I'm trying to figure out another way to fetch the files
> Hive
> >> > creates and return them as a download in the web service. Has anyone
> >> > else
> >> > worked out a good solution for fetching the contents of a directory
> from
> >> > HDFS as a part of larger flow?
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Shawn
> >> >
> >> >
>

Mime
View raw message