From dev-return-16578-apmail-nifi-dev-archive=nifi.apache.org@nifi.apache.org Wed Mar 7 04:58:56 2018 Return-Path: X-Original-To: apmail-nifi-dev-archive@minotaur.apache.org Delivered-To: apmail-nifi-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3EADA188BB for ; Wed, 7 Mar 2018 04:58:56 +0000 (UTC) Received: (qmail 73589 invoked by uid 500); 7 Mar 2018 04:58:55 -0000 Delivered-To: apmail-nifi-dev-archive@nifi.apache.org Received: (qmail 73544 invoked by uid 500); 7 Mar 2018 04:58:55 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Delivered-To: moderator for dev@nifi.apache.org Received: (qmail 50499 invoked by uid 99); 7 Mar 2018 04:44:30 -0000 X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.172 X-Spam-Level: ** X-Spam-Status: No, score=2.172 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, NML_ADSP_CUSTOM_MED=1.2, SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.972] autolearn=disabled Date: Tue, 6 Mar 2018 21:44:28 -0700 (MST) From: Bobby To: dev@nifi.apache.org Message-ID: <1520397868613-0.post@n7.nabble.com> Subject: Need opinion, NIFI for real time stream processing?? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Good day, I would like to know the possibilities of NIFI in real time processing; I'm thinking of a simple application of messaging, 1. A processor will accept messages and do a simple mapping of its attributes to database, i.e. if message_body.startsWith("123") then get data from table with ID = 123 and add the value as new attribute 2. A processor will act as a router 3. Two processors will accept act as dispatcher In simple illustration, |RECEIVER| -- |ROUTER| -- |DISPATCHER A| -- |DISPATCHER B| Now my questions are: 1. What is the best practice / proper way to do database mapping with complex criteria?, in my experience, the best way to do this in Java is to fetch the data in specific table (let's say Table A) into a collection (Map) in regular schedule (per 5 mins or so); The reason i'm doing this is to reduce the connection thingy when querying. Imagine if for each message we have to open connection, query, close connection. I've tested this approach and it was much faster, even with connection pooling. The downside is i need space in my memory which grow along with my database size. I use MySQL and haven't tested in memory database, i just want to know the common and fast approach to do this. If there is no other way, i might create a standalone module using spring boot to handle this stuffs and communicating with NIFI processor using REST API 2. Is it OK to use one flow file for one message? 3. Is it possible and recommended to use dependency injection library like Spring? Thank you, Bobby -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/