nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From obaidul karim <obaidc...@gmail.com>
Subject Re: JDBC External Table File Target
Date Sun, 31 Jan 2016 17:21:18 GMT
Hi Matthew ,

I didn't read your reply(just now reading :)), however, I have implemented
in the same way as you described :)).

In fact, I followed steps in FetchFile processor to read files directly
from an input list generated by  ListFile processor.
My flow looks like below:

 Flow: An Input Folder > [1]ListFile(Standard Processor) > [2]NzLoad(My
Custom processor) > [3]NzSql (My Custom processor)

Now need your help on below:
1. As I am reading from ListFile, it should be thread safe (a single file
should not be processed by more than 1 threads on NzLoad), right ?
2. How can I make the full flow synchronous. I want it like below:
a. [1]ListFile should list <=N files for each run and then stop until [2] &
[3]c completes
b. [2]NzLoad will process all the <=N files(using multiple thread) and only
then then trigger [3]NzSql
c. [3]NzSql will complete it processing and then trigger [1]ListFile and so
on.

Please share me if you have any alternative idea.

Regards,
Obaid

On Fri, Jan 29, 2016 at 4:22 AM, Matthew Burgess <mattyb149@gmail.com>
wrote:

> I think an approach that could work is the following:
>
> 1) Start with a ListFile processor to locate the filenames of the files
> you want to load into netezza
>
> 2) Send to a ReplaceText processor, where you replace the entire contents
> of the flow file with:
> insert into MYTABLE
> select * FROM EXTERNAL '${filename}'
> USING (<JDBC Options>) ;
>
>
> Note I’m using Expression Language to refer to the “filename” attribute.
> Each incoming flow file will have that attribute set to one of the files
> found by the ListFile processor.
>
>
>
> 3) Send to a PutSQL processor. Your SQL statement(s) are complete at this
> point so you don’t need arguments, parameters, etc.
>
> I haven’t tried this yet but it seems like it should work. If you give it
> a try, please let us know how it works for you :)
>
> Regards,
> Matt
>
>
> On 1/28/16, 3:04 PM, "Joe Percivall" <joepercivall@yahoo.com> wrote:
>
> >Hello Obaid,
> >
> >Sorry no one has gotten back to you sooner, many of the developers are
> working diligently to get 0.5.0 done.
> >
> >I don't know too much about loading SQL tables but when you say "local
> file system filename", is this a file that exits on the target system? If
> so you may just be able to set that command because my understanding is
> that using ExecuteSQL you have an input query which gets sent to the target
> server. Then the target server then runs that command.
> >
> >Can anyone else that has more experience with SQL and loading tables
> chime in?
> >
> >
> >Joe
> >- - - - - -
> >Joseph Percivall
> >linkedin.com/in/Percivall
> >e: joepercivall@yahoo.com
> >
> >
> >
> >On Saturday, January 23, 2016 12:31 AM, obaidul karim <
> obaidcuet@gmail.com> wrote:
> >
> >
> >
> >Hi,
> >
> >I developing a nifi processor for netezza.
> >As per by experience the best way to load data to netezza is using
> netezza JDBC external table.
> >
> >I want to run below command to load a file within NiFi processor.
> >
> >insert into MYTABLE
> >select * FROM EXTERNAL '<local file system filename>'
> >USING (<JDBC Options>) ;
> >
> >My question is which file to usefor <local file system filename>  ?
> >- Is it the flow files ? If yes then how can I get full path of a
> flowfile ?
> >- or can I directly load the file in a spool directory ?
> >
> >
> >Thanks for your help in advance.
> >
> >-Obaid
>
>

Mime
View raw message