nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Kawamura <ijokaruma...@gmail.com>
Subject Re: Transformations using Nifi
Date Fri, 13 Oct 2017 00:28:14 GMT
Hi Aruna,

If you can not upgrade from NiFi 1.2.0, then I think the best bed is using:
ScriptedReader or ScriptedRecordSetWriter for data conversion #1, 2 and 3.
As Russ mentioned, EL might be helpful when you implement the scripted
components.

#4 is a bit harder since it requires a database connection, but doable, I
don't know it works efficiently though..

Alternative approach would be create a temporary table, insert rows as they
are, then perform a insert/update query using the temporary table and
postgresql lookup table such as:
"insert into X select a, b, c, from T inner join L on X.j = T.j where d =
..."
Probably data conversion #1, 2, and 3 can be performed from this query as
well.

Thanks,
Koji

On Thu, Oct 12, 2017 at 11:50 PM, Russell Bateman <russ@windofkeltia.com>
wrote:

> Aruna,
>
> I don't think there is any generalized NiFi training course yet. I started
> writing a book a year ago, but it's a pretty thankless task and takes a lot
> of time. I mostly write custom processors so I tend to see the world
> differently from most "user" users of NiFi. When I answer questions, which
> I don't do too much, I often tell an impractical story that doesn't answer
> the question asked.
>
> Now, I haven't done much database work at all, so I'm not going to be too
> much help to you. However, looking at your questions, particularly the one
> about obtaining the current date when the flow is run (#2) and also #3, I
> would suggest that you study the NiFi Expression Language. This would allow
> you to insert the "now" date into your flow at some point.
>
> The other suggestions I have are to experiment, which I'm sure you're
> doing, and Google hard for help. For this forum, which is filled with very
> nice and helpful people, you'll tend to get a lot more and better help if
> you come in with a very specific question rather than a list of things or a
> general, "help me" sort of plea.
>
> Cheers,
>
> Russ
>
> P.S. NiFi absolutely rocks, which you'll see as soon as you get over your
> initial hump here. But, you're on the right track.
>
> On 10/12/2017 08:16 AM, Aruna Sankaralingam wrote:
>
> Hi,
>
>
>
> Could you someone please help me with these requirements in the email
> below?
>
>
>
> Thanks
>
> Aruna
>
>
>
> *From:* Aruna Sankaralingam [mailto:Aruna.Sankaralingam@Cormac-Corp.com
> <Aruna.Sankaralingam@Cormac-Corp.com>]
> *Sent:* Wednesday, October 11, 2017 11:26 AM
> *To:* users@nifi.apache.org
> *Subject:* Transformations using Nifi
>
>
>
> I am trying to see what kind of transformations that can be done in nifi
> and how.
>
> Now I have a basic flow that takes CSV from the local dir and puts into s3
> and loads into postgres database.
>
> There are 4 columns in my test file 3 of which are string and one is an
> integer field. I would like to do the following before I load the data into
> postgres. If someone can help me on how to go about these, it will be great.
>
> 1.       Convert one of the string columns to upper case
>
> *For converting to upper case, I was told to use the Update Record
> processor but  my version is 1.2.0 and the update record processor is not
> available. *
>
>
>
> 2.       Postgres has an extra column called “Load_Date” in which I would
> like to load the current date with timestamp when the flow is run
>
> 3.       If the integer column has more than 5 digits, I would like to
> take only the first 5 digits and load to the table
>
> 4.       There is a look up table in postgres. I would like to check if
> the first column value is present in the look up table and if yes, proceed
> ahead and if not ignore the record
>
>
>
> I am trying to learn nifi so I would really appreciate any kind of help
> here. Is there any training available online that I can take in order to
> understand and do all these?
>
>
>
>
>

Mime
View raw message