nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Psaltis <psaltis.and...@gmail.com>
Subject Re: Automated Testing of Nifi Flow by injecting data into Processors
Date Wed, 06 Mar 2019 17:29:31 GMT
Hi Alexei,
If you're talking about testing your flow with simulated data as if it's
deployed, basically not unit tests. I've seen this done in several
different ways:
1. Using the GenerateFlowFile processor and adding in custom text. At
first, that may seem obvious, however, if you think about it in that custom
text field you can add in a block of JSON data that then in a subsequent
processor you can manipulate to populate various values and then convert it
to the format you need. From here you can have the data continue on to the
rest of your flow, or if you want to test the reading of data from a type
of source, perhaps File then you can write the data out to test location
and have your regular flow read it in. There are various ways you can take
this to make it as elaborate as necessary.

2. Build a custom processor to do the data generation. Here is an example
for TimeSeries data:
https://medium.com/hashmapinc/its-here-an-apache-nifi-simulator-for-generating-random-and-realistic-time-series-data-d7e463aa5e78

Does this help with idea on how to do it?



On Wed, Mar 6, 2019 at 11:50 AM Zenin, Alexei <alexei.zenin@scotiabank.com>
wrote:

> Hi,
>
> Is there any way in Nifi to test out your whole flow without having the
> source systems online and running. I need this kind of capability to run
> automated integration tests which tests if the flow behaves as expected.
> None of the tools or libraries I have seen accomplish this and the Nifi API
> does not seem to allow to inject data directly into a processor (Nifi's
> TestRunner seems to be able to create FlowFiles but can we inject mock data
> to a processor through the Nifi REST API?). I think this would be a
> critical feature to make this system more production ready and abide by
> best practices for CICD.
>
> Regards,
> Alexei
> To unsubscribe from receiving further commercial electronic messages from
> The Bank of Nova Scotia in Canada, please click here:
> https://unsubscribe.scotiabank.com?entid=BNS&buid=SBNK
>
> To unsubscribe from receiving further commercial electronic messages from
> certain other senders set out in the attached list, please click here:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm
>
> This email may contain confidential information the use of which by an
> unintended recipient is unauthorized. This email may also contain important
> disclosure information for the records of the intended recipient(s).  For
> details please click here:
> http://www.scotiabank.com/email_disclaimer/email_english.html
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de La Banque de Nouvelle-Écosse, veuillez cliquer ici :
> https://desabonnement.banquescotia.com?entid=BNS&buid=SBNK
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de certains autres expéditeurs figurant dans la liste
> ci-jointe, veuillez cliquer ici :
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=fr
>
> Cette transmission peut contenir de l'information confidentielle et son
> utilisation par toute personne autre que la personne à laquelle cette
> transmission est destinée est interdite. Le présent courriel peut aussi
> contenir des renseignements importants pour les dossiers du ou des
> destinataires prévus.  Pour plus de détails, veuillez vous diriger vers:
> http://www.scotiabank.com/email_disclaimer/email_francais.html
>
> Para dejar de recibir mensajes electrónicos comerciales de The Bank of
> Nova Scotia en Canadá, haga clic aquí:
> https://cancelarsuscripcion.scotiabank.com?entid=BNS&buid=SBNK
>
> Para dejar de recibir mensajes electrónicos comerciales de otras entidades
> incluidas en la lista adjunta, haga clic aquí:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=es
>
> Este correo electrónico puede contener información confidencial cuyo uso
> por parte de personas distintas de los destinatarios del mismo está
> prohibido. El mensaje puede tener también datos importantes para los
> archivos   del(de los) destinatario(s) a quien(es) está dirigido.  Para más
> detalles, por favor dirigirse a:
> http://www.scotiabank.com/email_disclaimer/email_espanol.html

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message