nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject RE: Nifi Rest API Examples
Date Wed, 02 Sep 2015 22:39:39 GMT
John,
As Matt noted, we store the data in the repository as it is traversing through the system.
There are three key repositories: The FlowFile Repository, the Content Repository, and the
Provenance Repository.
As a Processor writes data to a flowfile, that is streamed directly to the content repository.
When the processorfinishes, it commits the session (essentially marks a transaction as complete).
This triggers the Provenance Repositoryto be updated to include the events that occurred for
that processor and then the FlowFile repositoryis then updated to keep track of where in the
flow the FlowFile is. Finally, the FlowFile can be moved to the nextqueue in the flow.
This way, if power is lost at any point, NiFi is able to resume where it left off.
This, however, glosses over one detail, which is that by default when we update the repositories,
we write the informationto disk but this is often cached by the operating system. If you truly
have a complete loss of power, it is possible to lose those updates tothe repository. This
can be avoided by configuring the repositories in the nifi.properties file to always sync
to disk. This, however,can be a significant hinderance to performance. Simply killing NiFi,
though, will not be problematic, as the operating system will stillbe responsible for flushing
that data to the disk.
This can all get pretty complicated, so if you have more detailed questions, we are happy
to assist.
Thanks-Mark


Date: Wed, 2 Sep 2015 15:22:52 -0700
Subject: Re: Nifi Rest API Examples
From: matt.c.gilman@gmail.com
To: users@nifi.apache.org

John,
The entity objects are for use with the Rest Api. They are populated in the web tier using
NiFi's internal object model. They are then serialized as the entity of the response. Conversely,
they can also be used when consuming the services. A great example of this is in the access
control unit tests [1].
As for the power loss scenarios, NiFi does store the FlowFiles (and content) in their respective
repositories. It will continue where it left off. Another PMC member should be replying here
shortly with more specifics.
Thanks!
Matt
[1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/DfmAccessControlTest.java
On Wed, Sep 2, 2015 at 2:54 PM, John McMahon <jmcmahon@bericotechnologies.com> wrote:

  
    
  
  
    I have been able to run my rest calls and get json data returned. I
    do have a process for creating Json objects from those responses,
    but I see there is the nifi API has object classes for successful
    returns.

    

    For example, the /controller/about call returns a JSON with two
    entries (revision and about) and according to the Rest API doc it
    could also be an AboutEntity that has two properties with the same
    respective names.

    

    But when I look at the About entity, it doesn't have a constructor
    to use a JSON string to set the property values and create an
    instance of the entity.

    

    Is there a recommended approach for getting the response as the
    respective Entity object for the rest call?

    

    

    Also, I've tried some attempts to test Nifi's persistence when a
    data flow is active. But I think the data flow I've created is too
    small or simple as by the time I've disabled power the to mission,
    the data flow has successfully run it's course. Are there any
    recommendations for ways to test what happens to data within a data
    flow when nifi is suddenly terminated?

    

    I know there are four repositories in use, does the data in a data
    flow get stored in any of these repositories when transitioning
    between processors within a data flow? Does Nifi continue where it
    left off when possible?

    

    John

    

    On 9/2/15 11:34 AM, Aldrin Piri wrote:

    
    
      John,
        

        
        The API lives at /nifi-api, so the address you are looking
          for would be http://localhost:8080/nifi-api/controller/about
        

        
        Please let us know if you have any additional questions or
          stumbling blocks. Thanks!
        

        
        Aldrin Piri
          Sent from my mobile device.
        
        

          On Wed, Sep 2, 2015 at 10:16 AM, John
            McMahon <jmcmahon@bericotechnologies.com>
            wrote:

            
               Greetings all,

                

                I have started looking into Apache Nifi. I was
                interested in using the Rest API and after I started my
                instance of Nifi, I wanted to test it.

                

                My Nifi is located at localhost:8080/nifi. I am using
                Rest Web Service Client on Chrome

                

                I set the method to GET and the URL to http://localhost:8080/nifi/controller/about

                

                I think this is a good rest call to use since the
                Authorization is Ready Only and the data shouldn't be
                anything critical.

                

                I get the the following response:

                

                404,
                  Not Found
                Date:
                  Wed, 02 Sep 2015 15:08:44 GMT
                Cache-Control:

                  must-revalidate,no-cache,no-store,
                  must-revalidate,no-cache,no-store
                X-UA-Compatible:
                  IE=edge
                Server:
                  Jetty(9.2.11.v20150529)
                Content-Length:
                  305
                Content-Type:
                  text/html; charset=ISO-8859-1
                
                Time
                  taken (in

                  milliseconds): 52

                  

                 Mind you, I don't have any users entered into
                Nifi's user list. This is currently just a barebones
                Nifi instance.

                

                I don't know why the response contains IE=edge as I do
                not have that software on my Mac.

                

                Is the path I entered for URL incorrect? Then I set the
                URL to http://localhost:8080/controller/about

                

                and it returned a response code of 200, however  in the
                Raw Response I got the following message about mistyping
                and recommending I add "/nifi" to the path:

                

                <!DOCTYPE
                  html>
                <html>
                
                <head>
                <meta
                  http-equiv="Content-Type" content="text/html;
                  charset=utf-8" />
                <link
                  rel="shortcut icon"
                  href="../nifi/images/nifi16.ico"/>
                <title>NiFi</title>
                <link
                  href="/nifi/css/message-pane.css" rel="stylesheet"
                  type="text/css" />
                <link
                  href="/nifi/css/message-page.css" rel="stylesheet"
                  type="text/css" />
                </head>
                
                <body
                  class="message-pane">
                <div
                  class="message-pane-message-box">
                <p
                  class="message-pane-title">
                Did
                  you mean: <a href="/nifi/">/nifi</a>
                </p>
                <p
                  class="message-pane-content">You may have
                  mistyped...</p>
                </div>
                </body>
                </html>

                

              
            
          
          

        
      
    
    

  


 		 	   		  
Mime
View raw message