nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Bateman <r...@windofkeltia.com>
Subject Re: Adding JVM arguments when running NiFi...
Date Wed, 18 Jan 2017 18:45:47 GMT
Thanks for pointing that out, Mark.

The net result of what I wrote is just that: numeric arguments in order 
of N, then whatever else is there, unsorted. I did nothing, however, to 
handle conflicting values of N. I don't claim the code I attached to the 
JIRA to be the best solution, only an illustration of a solution. In my 
opinion, except for respecting the value of N, this is all using a 3lb 
hammer to crack a peanut, but it has to be done and it's up to the real 
committers to decide how far to run with this. Personally, I'd just use 
the code I wrote, with added handling for duplicate values of N, then 
add the non-numeric arguments and call it good.

For most folk, it will never make any difference.

Russ

P.S. I'm promising a wiki-exposé on how to use the Java Flight Recorder 
and Java Mission Control as soon as I gain authorization and someone 
explains to me how to create the page in the wiki.


On 01/18/2017 08:31 AM, Mark Payne wrote:
> Russ,
>
> Sorry - I didn't see this sooner. That is accurate, that 'N' can be non-numeric. By default,
in bootstrap.conf, there is actually a property named "java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000"
but it is commented out.
>
> I think that if we want to guarantee ordering, we need to do some sort of sorting on
the keys. For example, first comes all numbers in numeric ordering (i.e., - not lexigraphical
ordering - so java.arg.1, java.arg.2, java.arg.3, ..., java.arg.10, java.arg.11, ...), followed
by non-numeric values in lexigraphical ordering. This would make it much easier to guarantee
ordering by simply using numbers but doesn't enforce numbers to be used. We didn't enforce
numbers to be used because historically we've found that people are inclined to always use
a one-up number. So when they want to add a new property, they go searching through the file
looking for the latest number used and then use the next number; then, if they un-comment
a line, that one may conflict resulting in two arguments with the same numbers and one of
them is overridden.
>
> Thanks
> -Mark
>
>
>> On Jan 18, 2017, at 10:22 AM, Russell Bateman <russ@windofkeltia.com> wrote:
>>
>> For my own purposes, I fixed this in 0.7.1 /nifi-bootstrap///src/main/java/org/////apache/nifi/bootstrap/RunNiFi.java/,
but at Joe's suggestion, I created JIRA NIFI-3364 hoping some committer will get to it for
all code branches-forward. The solution I used was made to be as little disruptive to the
existing code as possible. To that JIRA I attached the solution pinched between comments in
a larger test demonstrating it.
>>
>> I had found that I could trick the order in my local environment by trial and error,
however, moving to another host and with the additional need to dominate the order of JMX
remote definitions as well (in order to reach NiFi remote from Java Mission Control), it just
became impossible to achieve it.
>>
>> The code I wrote isn't the only way to fix nor even necessarily the best, but it
works supporting both the ordering of /N/in java.arg.Nas well as (did you know this?) the
use of non-numeric options for the position of /N/.
>>
>> Russ
>>
>> On 12/14/2016 11:05 AM, Russell Bateman wrote:
>>> I've doctored /conf/bootstrap.conf/ to contain these additional lines:
>>>
>>>     java.arg.15=-XX:+UnlockCommercialFeatures
>>>     java.arg.16=-XX:+FlightRecorder
>>>     java.arg.17=-XX:StartFlightRecording=duration=120m,filename=recording.jfr
>>>
>>>
>>> In the end, NiFi's grumpy about this and won't start (from /logs/nifi-bootstrap.log/):
>>>
>>> 2016-12-14 10:39:36,489 ERROR [NiFi logging handler] org.apache.nifi.StdErr Error:
*To use 'FlightRecorder', first unlock using -XX:+UnlockCommercialFeatures.*
>>> 2016-12-14 10:39:36,489 ERROR [NiFi logging handler] org.apache.nifi.StdErr Error:
Could not create the Java Virtual Machine.
>>> 2016-12-14 10:39:36,489 ERROR [NiFi logging handler] org.apache.nifi.StdErr Error:
A fatal exception has occurred. Program will exit.
>>> 2016-12-14 10:39:36,507 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never
started. Will not restart NiFi
>>>
>>> I tried using all options as one (in case the order is disturbed, which it was):
>>>
>>>     java.arg.15=-XX:+UnlockCommercialFeatures-XX:+FlightRecorder-XX:StartFlightRecording=duration=120m,filename=recording.jfr
>>>
>>>
>>> and then I get:
>>>
>>> *2016-12-14 10:50:07,574 ERROR [NiFi logging handler] org.apache.nifi.StdErr
Unrecognized VM option 'UnlockCommercialFeatures -XX:+FlightRecorder -XX:StartFlightRecording=duration=120m,filename=recording.jfr'*
>>> 2016-12-14 10:50:07,574 ERROR [NiFi logging handler] org.apache.nifi.StdErr Error:
Could not create the Java Virtual Machine.
>>> 2016-12-14 10:50:07,574 ERROR [NiFi logging handler] org.apache.nifi.StdErr Error:
A fatal exception has occurred. Program will exit.
>>> 2016-12-14 10:50:07,598 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never
started. Will not restart NiFi
>>>
>>> Here's the second command line from /logs/nifi-bootstrap.log/, which I've wrapped
for convenience in reading (see JVM version here too):
>>>
>>> 2016-12-14 10:56:40,526 INFO [main] org.apache.nifi.bootstrap.Command Working
Directory: /home/russ/dev/nifi/nifi-1.1.0
>>> 2016-12-14 10:56:40,526 INFO [main] org.apache.nifi.bootstrap.Command Command:
*/home/russ/dev/jdk1.8.0_112/bin/java*
>>> -classpath /home/russ/dev/nifi/nifi-1.1.0/./conf:/home/russ/dev/nifi/nifi-1.1.0/./lib/jcl-over-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/logback-core-1.1.3.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/nifi-documentation-1.1.0.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/nifi-nar-utils-1.1.0.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/logback-classic-1.1.3.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/jul-to-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/nifi-runtime-1.1.0.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/slf4j-api-1.7.12.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/nifi-api-1.1.0.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/nifi-framework-api-1.1.0.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/log4j-over-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.1.0/./lib/nifi-properties-1.1.0.jar
>>> -Dorg.apache.jasper.compiler.disablejsr199=true
>>> -Xmx512m
>>> -Xms512m
>>> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9191
>>> *-XX:+UnlockCommercialFeatures
>>> -XX:+FlightRecorder
>>> -XX:StartFlightRecording=duration=120m,filename=recording.jfr*
>>> -Dsun.net.http.allowRestrictedHeaders=true
>>> -Djava.net.preferIPv4Stack=true
>>> -Djava.awt.headless=true
>>> -XX:+UseG1GC
>>> -Djava.protocol.handler.pkgs=sun.net.www.protocol
>>> -Dnifi.properties.file.path=/home/russ/dev/nifi/nifi-1.1.0/./conf/nifi.properties
>>> -Dnifi.bootstrap.listen.port=41736
>>> -Dapp=NiFi
>>> -Dorg.apache.nifi.bootstrap.config.log.dir=/home/russ/dev/nifi/nifi-1.1.0/logs
org.apache.nifi.NiFi
>>>
>>> Last, I frantically tried this:
>>>
>>>     java.arg.15="-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
>>>     -XX:StartFlightRecording=duration=120m,filename=recording.jfr"
>>>
>>>
>>> and then got:
>>>
>>> *2016-12-14 11:01:36,659 ERROR [NiFi logging handler] org.apache.nifi.StdErr
Error: Could not find or load main class "-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
-XX:StartFlightRecording=duration=120m,filename=recording.jfr"*
>>>
>>>
>>> What's the canonical way of adding/inserting my own JVM commands? Alternatively,
is NiFi analyzing and blocking -XX:arguments to the JVM as illegal or unrecognized?
>>>
>>> Thanks,
>>>
>>> Russ


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message