logging-log4j-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Sicker <boa...@gmail.com>
Subject Re: Fast and Furious
Date Wed, 02 Mar 2016 15:01:03 GMT
That's what I thought at first, but he means non-human readable formats
since we all use tools to parse logs as it is (Splunk and ELK are the big
two I know of).

On 2 March 2016 at 02:15, Remko Popma <remko.popma@gmail.com> wrote:

> Re: binary logging, I think he's talking about providing an API to log
> objects directly into byte buffers without turning them into Strings first.
>
> https://issues.apache.org/jira/browse/LOG4J2-1274 and
> https://issues.apache.org/jira/browse/LOG4J2-506
>
> were created with that in mind and should be a good step in that
> direction.
>
> Sent from my iPhone
>
> On 2016/03/02, at 15:11, Gary Gregory <garydgregory@gmail.com> wrote:
>
> Well, I've often wondered about creating a binary format but it seems that
> you could use JSON+ZIP or BSON and get most of the advantages.
>
> Gary
>
> On Tue, Mar 1, 2016 at 9:12 PM, Matt Sicker <boards@gmail.com> wrote:
>
>> One other interesting thing I learned is that improper use of logging is
>> a huge source of performance problems. The GC-free parameterized message
>> factory will help with one aspect of that (I suggested parameterized
>> messages, but he countered with the Object[] that is created), and
>> encouraging users to use a Supplier<String> instead of passing parameters
>> should help as well (especially when those parameters have to be computed).
>> He had some strong criticisms of logging APIs promoting bad practices which
>> stems all the way back to log4j1 and affects pretty much every logging API
>> in Java (some criticisms were actually outdated or didn't consider newer
>> features of the API like markers and the huge amount of filters available).
>>
>> His other big idea was promoting the use of binary logging formats
>> because humans rarely read the raw log files as it is, but it's not like
>> there's a standard way to do that.
>>
>> Now I kinda wonder if he'll find this thread one day and tell me how I
>> misinterpreted him or something. ;)
>>
>> On 1 March 2016 at 22:28, Matt Sicker <boards@gmail.com> wrote:
>>
>>> Alright, I learned some interesting things. I'm going to get us some
>>> tools we can use to try and profile this. Otherwise, he did suggest trying
>>> out this project:
>>> https://github.com/RichardWarburton/honest-profiler
>>>
>>> On 1 March 2016 at 19:31, Matt Sicker <boards@gmail.com> wrote:
>>>
>>>> So far he's said something about using lambdas for lazy evaluation
>>>> (though I don't think that would actually help us at all). I'll try to talk
>>>> to him one-on-one afterward to delve more into this.
>>>>
>>>> On 1 March 2016 at 18:13, Ralph Goers <ralph.goers@dslextreme.com>
>>>> wrote:
>>>>
>>>>> Actually, most of the tests have the commands in the comments right in
>>>>> the class. Just cut and past.
>>>>>
>>>>> Ralph
>>>>>
>>>>> On Mar 1, 2016, at 1:43 PM, Matt Sicker <boards@gmail.com> wrote:
>>>>>
>>>>> I can't even figure out how to execute the simple perf test class.
>>>>> IntelliJ gives me some annotation processing error, and doing it from
the
>>>>> command line is turning into a classpath nightmare to figure out what
jars
>>>>> are needed to execute the test manually.
>>>>>
>>>>> On 1 March 2016 at 11:34, Gary Gregory <garydgregory@gmail.com>
wrote:
>>>>>
>>>>>> Before the talk: Hi, I'm Remko, I help on Apache Log4j, are you
>>>>>> available after the preso to talk about some issue we are seeing?
>>>>>>
>>>>>> Gary
>>>>>> On Mar 1, 2016 8:29 AM, "Matt Sicker" <boards@gmail.com> wrote:
>>>>>>
>>>>>>> I'm attending a JUG meetup tonight with Kirk Pepperdine presenting.
>>>>>>> It's supposed to be a Java performance workshop type of thing,
so if you've
>>>>>>> got a decent way to ask about it, I could see if he can help
figure out
>>>>>>> this regression. I can at least show off the SimplePerfTest and
any
>>>>>>> microbenchmarks we have.
>>>>>>>
>>>>>>> On 28 February 2016 at 11:54, Matt Sicker <boards@gmail.com>
wrote:
>>>>>>>
>>>>>>>> Take a look at the git bisect command. Might help you find
which
>>>>>>>> changes caused the problem.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sunday, 28 February 2016, Gary Gregory <garydgregory@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thank you for digging in Remko. This is will be a nice
theme to
>>>>>>>>> publicize when you get it figured out.
>>>>>>>>>
>>>>>>>>> Gary
>>>>>>>>> On Feb 28, 2016 4:08 AM, "Remko Popma" <remko.popma@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> After removing the potential impact of appenders
and layouts by
>>>>>>>>>> testing with log4j-core\src\test\resources\perf-CountingNoOpAppender.xml
>>>>>>>>>> and org.apache.logging.log4j.core.async.perftest.SimplePerfTest,
I've
>>>>>>>>>> confirmed my initial numbers:
>>>>>>>>>>
>>>>>>>>>> 2.0: 7.5M ops/sec
>>>>>>>>>> 2.1: 6M ops/sec
>>>>>>>>>> 2.2: 6M ops/sec
>>>>>>>>>> 2.3: 6M ops/sec
>>>>>>>>>> 2.4: 4.5M ops/sec
>>>>>>>>>> 2.5: 4M ops/sec
>>>>>>>>>> 2.6: 2M ops/sec
>>>>>>>>>>
>>>>>>>>>> I tried reverting various changes made to AsyncLogger
since 2.0,
>>>>>>>>>> performance improves a little up to 4M ops/sec.
>>>>>>>>>> However, when completely reverting AsyncLogger source
to the 2.0
>>>>>>>>>> version, performance is back to 7.5M ops/sec.
>>>>>>>>>>
>>>>>>>>>> I'll try starting from the 2.0 source and getting
back to 2.6
>>>>>>>>>> functionality without losing performance...
>>>>>>>>>> (Lengthy process...)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Feb 27, 2016 at 12:18 PM, Remko Popma <
>>>>>>>>>> remko.popma@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> This is the PerfTestDriver test class (in log4j-core/test,
>>>>>>>>>>> package ...async.perf).
>>>>>>>>>>> Mainly perf3PlainNoLocation.xml:
>>>>>>>>>>> RollingRandomAccessFileAppender, PatternLayout,
all loggers are
>>>>>>>>>>> AsyncLoggers, logging a simple string without
parameters.
>>>>>>>>>>>
>>>>>>>>>>> Profiling with YourKit did not tell me anything
useful.
>>>>>>>>>>>
>>>>>>>>>>> I'm now eliminating the effect of Layouts/Appenders,
using
>>>>>>>>>>> CountingNoOpAppender, and seeing similar numbers.
So this seems to be
>>>>>>>>>>> mostly an issue in AsyncLogger.
>>>>>>>>>>>
>>>>>>>>>>> I'll let you know when I find out more.
>>>>>>>>>>> There's a lot of trial and error here, so this
may take a
>>>>>>>>>>> while...
>>>>>>>>>>>
>>>>>>>>>>> Remko
>>>>>>>>>>>
>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>
>>>>>>>>>>> On 2016/02/26, at 21:02, Mikael Ståldal <
>>>>>>>>>>> mikael.staldal@magine.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Which components (appenders, layouts) are involved
in the tests?
>>>>>>>>>>> Would it be possible to do some profiling to
see if there is any particular
>>>>>>>>>>> component which is to blame?
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Feb 26, 2016 at 12:51 PM, Remko Popma
<
>>>>>>>>>>> remko.popma@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> To give you some rough impression on concrete
numbers for this
>>>>>>>>>>>> trend:
>>>>>>>>>>>> 2.0: ~6M ops/sec
>>>>>>>>>>>> 2.1-2.2: ~5M ops/sec
>>>>>>>>>>>> 2.3-2.4: ~3-4M ops/sec
>>>>>>>>>>>> 2.5: ~3M ops/sec
>>>>>>>>>>>> 2.6: ~2M ops/sec
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Friday, 26 February 2016, Remko Popma
<remko.popma@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> You're absolutely right. I still have
quite a few unit tests
>>>>>>>>>>>>> to add.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Initial perf testing shows a downward
trend in Async Logger
>>>>>>>>>>>>> performance with every release. (Logging
simple string
>>>>>>>>>>>>> messages without params.) This is worrisome
and I'm focusing on figuring
>>>>>>>>>>>>> that out first: this will likely involve
additional code changes and I'll
>>>>>>>>>>>>> add more tests after that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2016/02/26, at 10:38, Gary Gregory
<garydgregory@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Wow, I love the activity we are seeing
toward 2.6! All the
>>>>>>>>>>>>> perf work on top of an existing sizable
change set. Very exciting indeed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There sure are a lot of changes coming
in. I hope that we all
>>>>>>>>>>>>> can pitch in to make sure most if not
all of these changes get code
>>>>>>>>>>>>> coverage from unit tests. I've not checked
closely, but it seems like we
>>>>>>>>>>>>> may not have good coverage _yet_, or
do I have the wrong impression?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I want to make sure we keep our stability
in tip top shape :-)
>>>>>>>>>>>>> and that we have no regression from previous
releases.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gary
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> E-Mail: garydgregory@gmail.com | ggregory@apache.org
>>>>>>>>>>>>> Java Persistence with Hibernate, Second
Edition
>>>>>>>>>>>>> <http://www.manning.com/bauer3/>
>>>>>>>>>>>>> JUnit in Action, Second Edition
>>>>>>>>>>>>> <http://www.manning.com/tahchiev/>
>>>>>>>>>>>>> Spring Batch in Action <http://www.manning.com/templier/>
>>>>>>>>>>>>> Blog: http://garygregory.wordpress.com
>>>>>>>>>>>>> Home: http://garygregory.com/
>>>>>>>>>>>>> Tweet! http://twitter.com/GaryGregory
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> [image: MagineTV]
>>>>>>>>>>>
>>>>>>>>>>> *Mikael Ståldal*
>>>>>>>>>>> Senior software developer
>>>>>>>>>>>
>>>>>>>>>>> *Magine TV*
>>>>>>>>>>> mikael.staldal@magine.com
>>>>>>>>>>> Grev Turegatan 3  | 114 46 Stockholm, Sweden
 |   www.magine.com
>>>>>>>>>>>
>>>>>>>>>>> Privileged and/or Confidential Information may
be contained in
>>>>>>>>>>> this message. If you are not the addressee indicated
in this message
>>>>>>>>>>> (or responsible for delivery of the message to
such a person),
>>>>>>>>>>> you may not copy or deliver this message to anyone.
In such case,
>>>>>>>>>>> you should destroy this message and kindly notify
the sender by
>>>>>>>>>>> reply email.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Matt Sicker <boards@gmail.com>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Matt Sicker <boards@gmail.com>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Matt Sicker <boards@gmail.com>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Matt Sicker <boards@gmail.com>
>>>>
>>>
>>>
>>>
>>> --
>>> Matt Sicker <boards@gmail.com>
>>>
>>
>>
>>
>> --
>> Matt Sicker <boards@gmail.com>
>>
>
>
>
> --
> E-Mail: garydgregory@gmail.com | ggregory@apache.org
> Java Persistence with Hibernate, Second Edition
> <http://www.manning.com/bauer3/>
> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
> Spring Batch in Action <http://www.manning.com/templier/>
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>
>


-- 
Matt Sicker <boards@gmail.com>

Mime
View raw message