nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Piehler <peter.pieh...@arcor.de>
Subject Re: data metrics / data monitoring
Date Tue, 13 Aug 2019 07:51:13 GMT
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Hi <br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">@Edward, @Pierre</div>
    <div class="moz-cite-prefix">thx for your replies. I will read the
      articles and hope I find an inspiration.</div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">Best,<br>
      Peter<br>
    </div>
    <div class="moz-cite-prefix"><br>
    </div>
    <div class="moz-cite-prefix">On 13.08.19 09:11, Pierre Villard
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CABgFv2465KjWZd2zskWrsryW=NyyF=qf6_EusHNRbSbi91s6vA@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">Hi Peter,
        <div><br>
        </div>
        <div>In addition to Edward's answer, you may be interested by
          the below posts:</div>
        <div><a
href="https://pierrevillard.com/2017/05/11/monitoring-nifi-introduction/"
            moz-do-not-send="true">https://pierrevillard.com/2017/05/11/monitoring-nifi-introduction/</a><br>
        </div>
        <div><a
href="https://pierrevillard.com/2018/02/07/fod-paris-jan-18-nifi-registry-and-workflow-monitoring-with-a-use-case/"
            moz-do-not-send="true">https://pierrevillard.com/2018/02/07/fod-paris-jan-18-nifi-registry-and-workflow-monitoring-with-a-use-case/</a><br>
        </div>
        <div><a
href="https://pierrevillard.com/2018/08/29/monitoring-driven-development-with-nifi-1-7/"
            moz-do-not-send="true">https://pierrevillard.com/2018/08/29/monitoring-driven-development-with-nifi-1-7/</a><br>
        </div>
        <div><br>
        </div>
        <div>Hope this helps,</div>
        <div>Pierre</div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">Le mar. 13 août 2019 à 00:36,
          Edward Armes &lt;<a href="mailto:edward.armes@gmail.com"
            moz-do-not-send="true">edward.armes@gmail.com</a>&gt; a
          écrit :<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div dir="auto">
            <div>Hi Peter,</div>
            <div dir="auto"><br>
            </div>
            <div dir="auto">I think this depends on where this data is
              stored. If this data is avaiable as metrics record by
              Nifi, then a reporting task would be the best way forward.
              However if this is data that is recorded in your FlowFiles
              as part of your flow then I think you're looking at either
              collecting in a KeyValue store of sorts and exposing it
              via a Web Server pattern or forwarding the metrics
              contained in the FlowFIle via a message bus, database or
              flow file reciever of some description.</div>
            <div dir="auto"><br>
            </div>
            <div dir="auto">As for displaying your metrics there are a
              lot of options out there that can recieve and processes
              data in various forms and it really depends on what is the
              best fit for your orginisation.</div>
            <div dir="auto"><br>
            </div>
            <div dir="auto">Personaly I would work out how and what you
              use display the data and from there use that to influence
              how you export it out from Nifi.</div>
            <div dir="auto"><br>
            </div>
            <div dir="auto">Edward</div>
            <div dir="auto"><br>
              <div class="gmail_quote" dir="auto">
                <div dir="ltr" class="gmail_attr">On Mon, 12 Aug 2019,
                  22:54 Peter Piehler, &lt;<a
                    href="mailto:peter.piehler@arcor.de" rel="noreferrer
                    noreferrer" target="_blank" moz-do-not-send="true">peter.piehler@arcor.de</a>&gt;
                  wrote:<br>
                </div>
                <blockquote class="gmail_quote" style="margin:0px 0px
                  0px 0.8ex;border-left:1px solid
                  rgb(204,204,204);padding-left:1ex">Hello,<br>
                  <br>
                  does anyone have a tip for me on how I can provide
                  metrics about data <br>
                  processed in nifi in a web UI?<br>
                  <br>
                  I process XML files with nifi. for each file I
                  calculate how many new, <br>
                  modified, unmodified, and deleted records are
                  contained. for each record <br>
                  checks are still made. For example, whether values are
                  in the value range.<br>
                  I would like to create an evaluation which shows me
                  how the data <br>
                  properties are. For example yesterday I had 5 files,
                  one of them with <br>
                  1000 deletions, but the average is only 10 deleted
                  records per file, on <br>
                  average we process 500 files per day.<br>
                  <br>
                  I'm currently looking for ideas on how to do this. I
                  think it would be <br>
                  useful to export this data and then evaluate it in an
                  external <br>
                  application. I am grateful for every hint.<br>
                  <br>
                  Thx,<br>
                  Peter<br>
                  <br>
                </blockquote>
              </div>
            </div>
          </div>
        </blockquote>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>

Mime
View raw message