From dev-return-16681-apmail-nifi-dev-archive=nifi.apache.org@nifi.apache.org Sat Mar 17 21:20:34 2018 Return-Path: X-Original-To: apmail-nifi-dev-archive@minotaur.apache.org Delivered-To: apmail-nifi-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5A61E18FDE for ; Sat, 17 Mar 2018 21:20:33 +0000 (UTC) Received: (qmail 94019 invoked by uid 500); 17 Mar 2018 21:20:33 -0000 Delivered-To: apmail-nifi-dev-archive@nifi.apache.org Received: (qmail 93961 invoked by uid 500); 17 Mar 2018 21:20:33 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 93944 invoked by uid 99); 17 Mar 2018 21:20:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Mar 2018 21:20:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id DA11BC053E for ; Sat, 17 Mar 2018 21:20:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.119 X-Spam-Level: X-Spam-Status: No, score=-0.119 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=interset.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id iBRYyd5ohJT2 for ; Sat, 17 Mar 2018 21:20:30 +0000 (UTC) Received: from mail-ot0-f169.google.com (mail-ot0-f169.google.com [74.125.82.169]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E17725F2EC for ; Sat, 17 Mar 2018 21:20:29 +0000 (UTC) Received: by mail-ot0-f169.google.com with SMTP id m22-v6so13784786otf.10 for ; Sat, 17 Mar 2018 14:20:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=interset.com; s=google; h=user-agent:date:subject:from:to:message-id:thread-topic:references :in-reply-to:mime-version:content-transfer-encoding; bh=F1WohTUKU8wf5J3qPzETNLQ3/kGsg5HOtSzV7volN00=; b=SyLL+sCpn3NxhkqlDj1+0gazKkxfI1HdZhxHrdT/nwhGzzEvtkO41P2D/QhMsy3ye2 0tT3eVpurZCaNh7oPJ/QoO/C+rhjxnNmML++rgxVXNlcZqnK1E9fPxqT2b69Yn3fHA7+ 7sElO3fxJ2VepSCOhbHQSXd+b4H2c+DagyoS4qb8WztXL5kMjlglDnkU3OGB2ruXiAYz neZSGjlPquYiDxqI3FZK4ukb+hUJbgXrCHdSH+oQt8/swawdvJenQi4WCBrN6hwITJ0a C/lH3MtuXbF1S+cjQYLeAJDjgCUGJoI9NFfEM4SKyC9Y8MJ0SHEdH8ms0uLG3NDEqKtz HQzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:user-agent:date:subject:from:to:message-id :thread-topic:references:in-reply-to:mime-version :content-transfer-encoding; bh=F1WohTUKU8wf5J3qPzETNLQ3/kGsg5HOtSzV7volN00=; b=BrJvXgPEcNgDYNTMYXCCfwDxl6ECA5CvPDJqhu9Nk9QU2OhaVKK28B2Q+fqRy4OBix r48kv9gYFba974VtxCqEDgH5VrCol0ws2pxF+gg+ZWOBzH0MRkl00MzLIyPtf8nWq/pG ROYGXiTwBABv6OykSP1Hm4Zyl/X5a10GyYRCLVtJco/sCbvCsQr0U2/1ZXAnMoXTecPL qzuSgcODRUqBS430MdZ1biW83kAIHFHZBfNZza60KxHPQbpz7Hb8nJKlKUj/FS3Oi2d9 uPE1mL8/y41h895UXxxjf6zc5hLejvP9SMrq8Cmc0Ra5SYAAuFsRRlcPagAjQD2CSIJw xxTw== X-Gm-Message-State: AElRT7Ei2OrfOrbXMcJsR3Pvvp2xhgyj7Sh01fe0wAZxviSGjmS9znfD ci0Bi6otwfjqicPuj8k9ti9MljSdZvN6iB5v29IucMa0rZMceod/pKvkiD9EZbItHZB87D1NVmm sDMwaSpSkwsLEDx8fS2ZqGsOUQQvIrA4I6zi2Vla2bUP+uPV/Dc4+Eu0rgURd X-Google-Smtp-Source: AG47ELtqqBokwd3z5HqNmlRejJ4aG38YE1DZdYxTRN/+qVBoAWnyatr6oXaPGsM8suGvPrbzGTUizA== X-Received: by 2002:a9d:3ef6:: with SMTP id b109-v6mr4219314otc.207.1521321622799; Sat, 17 Mar 2018 14:20:22 -0700 (PDT) Received: from [192.168.1.94] ([2600:1700:1960:2390:fcc7:e9b2:4ad5:6764]) by smtp.gmail.com with ESMTPSA id c33sm6207897ote.79.2018.03.17.14.20.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 17 Mar 2018 14:20:22 -0700 (PDT) User-Agent: Microsoft-MacOutlook/f.29.0.171205 Date: Sat, 17 Mar 2018 17:20:20 -0400 Subject: Re: Finding Performance bottleneck issue From: Milan Das To: Message-ID: Thread-Topic: Finding Performance bottleneck issue References: <8E6AF45A-6ECD-4A5C-9C9D-7833FF685A6F@interset.com> In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable Thanks for your help Mike Thomsen & Joe Witt. Performance problem was in the way I was each line from flow input. My logi= c was : io.InputStream->io.BufferedReader-> while reader.readLine() loop. Now I am using guava library. I have changed the code to use =E2=80=9CString inputString =3D CharStreams.toString= (new InputStreamReader(in, "UTF-8"));=E2=80=9D I gained 20 times performance. Regards, Milan Das On 3/16/18, 12:07 PM, "Mike Thomsen" wrote: That seems like a very reasonable use case. You said: =20 > I see that my processor is actually queuing up records at source. =20 Are you saying that the processor isn't able to process them that quick= ly such that you're seeing a big backlog in the input queue? =20 On Fri, Mar 16, 2018 at 11:56 AM, Milan Das wrote: =20 > Hi Mike, > My processor is processing windows Text event as below and creating a= JSON > out of it. > Also I am applying simple JoltTransformer (Simple just Shift and Defa= ult) > to convert to different JSON (no hierarchy) . > > Output have the following: > 1. Original text > 2. Converted JSON > 3. JOLT transformed JSON > 4. Failure > > > Steps in program: > 1. Converting the event to Java Map (using regex: "([^:=3D]*)[:=3D]([^:=3D]= *)") > 2. Map to Json (using Gson) > 3. Jolt transfeormation > > > > Example event: > > Examples of 4626 > User / Device claims information. > > Subject: > Security ID: %1 > Account Name: %2 > Account Domain: %3 > Logon ID: %4 > > Logon Type: %9 > > New Logon: > Security ID: %5 > Account Name: %6 > Account Domain: %7 > Logon ID: %8 > > Event in sequence: %10 of %11 > > User Claims: %12 > > Device Claims: %13 > > The subject fields indicate the account on the local system which > requested the logon. This is most commonly a service such as the Serv= er > service, or a local process such as Winlogon.exe or Services.exe. > > The logon type field indicates the kind of logon that occurred. The m= ost > common types are 2 (interactive) and 3 (network). > > The New Logon fields indicate the account for whom the new logon was > created, i.e. the account that was logged on. > > This event is generated when the Audit User/Device claims subcategory= is > configured and the user=E2=80=99s logon token contains user/device claims > information. The Logon ID field can be used to correlate this event w= ith > the corresponding user logon event as well as to any other security a= udit > events generated during this logon session. > > > > Regards, > Milan Das > > > On 3/16/18, 10:56 AM, "Mike Thomsen" wrote: > > Milan, > > Can you share some details about where you are running into probl= ems? > Like > a basic description of what it's trying to do? > > On Fri, Mar 16, 2018 at 10:39 AM, Milan Das w= rote: > > > I have a custom processor, it works as expected. But I feel the= re is > some > > performance measure need to be done. I see that my processor is > actually > > queuing up records at source. > > > > Is there a run a load test and do performance measure for Cust= om > > Processor? > > > > > > > > Regards, > > > > Milan Das > > > > > > > > =20