From dev-return-12980-apmail-nifi-dev-archive=nifi.apache.org@nifi.apache.org Thu Jan 12 19:14:40 2017 Return-Path: X-Original-To: apmail-nifi-dev-archive@minotaur.apache.org Delivered-To: apmail-nifi-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4F69019455 for ; Thu, 12 Jan 2017 19:14:40 +0000 (UTC) Received: (qmail 67315 invoked by uid 500); 12 Jan 2017 19:14:40 -0000 Delivered-To: apmail-nifi-dev-archive@nifi.apache.org Received: (qmail 67267 invoked by uid 500); 12 Jan 2017 19:14:40 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 67254 invoked by uid 99); 12 Jan 2017 19:14:39 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jan 2017 19:14:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 66012180BA9 for ; Thu, 12 Jan 2017 19:14:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.101 X-Spam-Level: X-Spam-Status: No, score=-0.101 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id UTyFt8aXrMsP for ; Thu, 12 Jan 2017 19:14:35 +0000 (UTC) Received: from mail-io0-f170.google.com (mail-io0-f170.google.com [209.85.223.170]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 468B15F370 for ; Thu, 12 Jan 2017 19:14:35 +0000 (UTC) Received: by mail-io0-f170.google.com with SMTP id v96so26338544ioi.0 for ; Thu, 12 Jan 2017 11:14:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-transfer-encoding; bh=mb2uzi5DV0LA3IrxA0SEEaE/OEVSVSyO5mzy7jmRt9Y=; b=Ens9SYgeRFLmbSzof7h+2DKtTTnbv0t5WfX2hJybRMVHYzdR9CmX0GhdBeAAdYDqCp mX39XGieFmHytmjhJGONWtPt+B6VBAwO/23GcP+qD+2HlBgBKAsolo5uSYlf8R5DJvVv TJ139l1o7wzB4PptiwYtTPLtFPpWYQez1toV1MCawF3gtSO57bbkzng9AHdWHLCXfiRT TxnnC+hkgFDsXUrHV1l0UDhJ2nbbnkZc2K52/Ky51VbR1dMB0rVFvEB8xkbsqMp9NHVp KnJkeKAb7aP+uP3vy1HqUG4se3UiGPpFIQLP9Y6T34CfXyqXdllRejX0+EJl+4RyCzOm zPKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-transfer-encoding; bh=mb2uzi5DV0LA3IrxA0SEEaE/OEVSVSyO5mzy7jmRt9Y=; b=Nhr2zaFRHHZw1ujkAaVyDN1DfIR2d3mlucjapbaFMW7Gg2L76dZDwICeThI0dSWP/w 3+q5+d1Vp6nrIO7A7jczn4pbBC2Jo6301oNtfoAeyAJAq8pkP3ZK8h7X8PHwS5r8R3OJ M6XvgE+cAYz99z+usjltvgh84GeOMHCP2dWSbIkW9LZ+L2CxCsymf3yTNZLJu3JYXwg6 6f6FnlGdunPGp/Wzo/+S+peok7HuwzOypLK4V11kJYtXm+mPNGYhuJ6/sDzvMD/iioar fJfJ9kRbuKE2y2NtSV1NFXFZPr0nnHRNeqUKOPM9r5ma7yY2E0KZf9vzNnq/xVamvuen 4EtA== X-Gm-Message-State: AIkVDXLQXecRI7koU+p+d3F2RHMvAEPTbktTig6l71EdgwMHb42oefakYtTE0g0OcXHFbn8F9SKLdBFQ+Mshyg== X-Received: by 10.107.190.71 with SMTP id o68mr5748117iof.95.1484248471400; Thu, 12 Jan 2017 11:14:31 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.140.66 with HTTP; Thu, 12 Jan 2017 11:14:30 -0800 (PST) In-Reply-To: References: <4EB0564F-9C0E-46E8-96E2-7158E618C4AB@mitre.org> From: Joe Witt Date: Thu, 12 Jan 2017 14:14:30 -0500 Message-ID: Subject: Re: [DISCUSS] Run Once scheduling To: dev@nifi.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Naz The green arrow vs red square says "scheduled to execute" vs "not scheduled to execute". For most processors, such as those which take input flow files from a connection, even if they're scheduled to run they're not going to be executed unless there is work to do (data sitting in the queue) and space available (on all destination relationships). Because of this I'm suggesting to consider just leaving them all scheduled to execute even though they won't actually be doing anything most of the time. The stats on each component tell you how many times it was actually invoked and how much data it processed, etc.. So you'll see that they're not doing anything most of the time. You mentioned not wanting to have to do anything manual yet run once would be a manual construct, right? I dont mean to suggest I'm closed off to the idea of a run once concept I just really want to understand your use case better. Thanks Joe On Thu, Jan 12, 2017 at 2:11 PM, Irizarry Jr., Nazario wrot= e: > Correction, that was the processor scheduler=E2=80=99s stopProcessor() me= thod that needs to be invoked so the UI shows that the processor is stopped= . > > Naz Irizarry > MITRE Corp. > 617-893-0074 > > > >> On Jan 12, 2017, at 2:08 PM, Irizarry Jr., Nazario wrote= : >> >> Yes, we found that to keep the UI in sync (make sure it looks stopped af= ter it runs once) the flow controller's stopProcessor() method has to be ca= lled. >> >> Naz Irizarry >> MITRE Corp. >> 617-893-0074 >> >> >> >> On Jan 12, 2017, at 1:41 PM, Brandon DeVries > wrote: >> >> I think answering Joe's question is step one. However, I was curious an= d >> tried something: >> >> public void onTrigger(...){ >> if(!isSheduled()){ >> return; >> } >> FlowFile flowFile =3D session.get() >> if (flowFile =3D=3D null){ >> return; >> } >> session.transfer(flowFile, REL_SUCCESS); >> updateScheduledFalse(); >> } >> >> This processes one FlowFile per "scheduling". I.e., one FlowFile goes >> through, and you need to stop / start to get another. I'm not 100% that >> nothing else would ever call the "built in" updateScheduled* methods, bu= t >> worst case the processor could maintain its own flag. Also, for what it= 's >> worth, calling updateScheduledFalse() doesn't "stop" the processor on th= e >> graph... as Oleg mentions, this (or something like this) could potential= ly >> be visually confusing. >> >> I'm not sure how this fits in a production system, but this + >> GenerateFlowFile and some backpressure seems possibly useful for >> debugging. I know I've faked this behavior with a GenerateFlowFile w/ r= un >> schedule "1 day" or something before... then again, maybe it would be b= est >> to not create something that could be confusing / misused in a productio= n >> system. >> >> Brandon >> >> >> >> >> On Thu, Jan 12, 2017 at 1:02 PM Joe Witt > wrote: >> >> Naz, >> >> Why not just leave all the processes running? If the data only >> arrives periodically that is ok, right? >> >> Thanks >> Joe >> >> On Thu, Jan 12, 2017 at 10:54 AM, Irizarry Jr., Nazario > >> wrote: >> On a project that I am on we have been looking at using NiFi for >> orchestrations that are invoked infrequently. For example, once a month= a >> new data input product becomes available and then one wants to run it >> through a set of processing steps that can be nicely implemented using N= iFi >> processors. However, using the interval or cron scheduling for this >> purpose begins to get cumbersome after a while with the need to start an= d >> manually stop these occasional flows. >> >> It would be fairly easy to add an additional scheduling option - =E2=80= =9CRun >> Once=E2=80=9D for this use case. The behavior would be that when a proc= essor is >> set to run once it automatically stops after it has successfully process= ed >> one input. >> >> What do people think? We are willing to implement this small >> enhancement. >> >> Cheers, >> >> Naz Irizarry >> MITRE Corp. >> 617-893-0074 <(617)%20893-0074> >> >