logging-log4j-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Goers <ralph.go...@dslextreme.com>
Subject Re: Make LogEvent implementations Externalizable
Date Fri, 11 Jul 2014 17:21:05 GMT
Sounds good. 

This approach allows for many different serialization formats.  It might be nice to have the
CompactBinaryLayout even be “pluggable” to support things like Hessian,  Protobuf, Thrift,
etc. Of course, that would also require the LogEventBridge to have support as well.

Ralph

On Jul 11, 2014, at 10:14 AM, Scott Harrington <scotth01@sns-usa.com> wrote:

> OK! Agree. So the 2.1 enhancement request will be for a CompactBinaryLayout on the sending
side and corresponding LogEventBridge on the receiving side. I'll open it in JIRA when I get
back from vacation and have a chance to work through an initial implementation, unless someone
else beats me to it.
> 
> Current 'Serializable' form is a bit chatty but is easier (automatic) to maintain. Forget
I ever said anything about Externalizable.
> 
> 
> On Sat, 12 Jul 2014, Remko Popma wrote:
> 
>> Yes that is what I had in mind.
>> 
>> 
>> On Sat, Jul 12, 2014 at 1:53 AM, Scott Harrington <scotth01@sns-usa.com>
>> wrote:
>> 
>>> Yes it did appear toByteArray would do the job on the /sending/ side but
>>> what about the /receiving/ side? We've got to put some bytes on the wire
>>> that will come out as a LogEvent from the readObject() call in
>>> ObjectInputStreamLogEventBridge. You can't have the sender be
>>> Externalizable and the received class be only Serializable.
>>> 
>>> Unless you're proposing we make a totally new LogEventBridge -- i.e. a
>>> 'compact binary' protocol to go alongside the XML and Json versions. If we
>>> go that route then you could achieve even more compression since we know
>>> every object that we're sending is a LogEvent and even Externalizable has
>>> some overhead that we could do away with.
>>> 
>>> 
>>> 
>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>> 
>>> On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma <remko.popma@gmail.com>
>>>> wrote:
>>>> 
>>>> What prevents you from writing a more compact representation of the
>>>>> LogEvent (including all other LogEvent fields) to the byte array in the
>>>>> {{toByteArray(LogEvent)}} method of an ExternalizableLayout?
>>>>> 
>>>>> (Just to clarify, I intended this question literally as it could easily
>>>> be
>>>> that I overlooked something...)
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>> On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington <scotth01@sns-usa.com
>>>>>> 
>>>>> wrote:
>>>>> 
>>>>> I looked at replacing SerializedLayout and/or the toSerializable method
>>>>>> but the real savings comes not from the Message itself but from all
the
>>>>>> other fields of the LogEvent, such as the Level, ContextMap,
>>>>>> ContextStack,
>>>>>> and the class descriptor (including superclasses) of the LogEvent
>>>>>> itself.
>>>>>> 
>>>>>> For big messages, an ExternalizableLayout would be a fine solution,
>>>>>> because writeUTF would give you roughly 50% compression over Java's
>>>>>> default
>>>>>> 2-byte per char String serialization.
>>>>>> 
>>>>>> However for typical log messages the message itself is being dwarfed
on
>>>>>> the wire by class descriptor overhead.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>>>>> 
>>>>>> That is a nice reduction in size!
>>>>>> 
>>>>>>> 
>>>>>>> I also think the ExternalizedLayout idea is a very attractive
option.
>>>>>>> That
>>>>>>> way there is no pressure to include this in any particular release,
we
>>>>>>> can
>>>>>>> release it when we are confident that is ready. I also like the
fact
>>>>>>> that
>>>>>>> it does not replace the current serialization and it can be switched
on
>>>>>>> and
>>>>>>> off in configuration so if there is a bug, users can fall back
to the
>>>>>>> existing plain serialization.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker <boards@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> I would second the ExternalizedLayout. Layouts are the way to
go for
>>>>>>> 
>>>>>>>> compatibility and are simpler. Particularly useful for alternative
>>>>>>>> serialization protocols, too.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 11 July 2014 06:41, Gary Gregory <garydgregory@gmail.com>
wrote:
>>>>>>>> 
>>>>>>>> I understand Ralph ' s concern but now is the time for this
kind of
>>>>>>>> 
>>>>>>>>> change.  Otherwise we will need even more clever solutions
to get
>>>>>>>>> this
>>>>>>>>> 
>>>>>>>>> kind
>>>>>>>> 
>>>>>>>> of size improvement. I'd love to see some performance numbers.
The
>>>>>>>>> size
>>>>>>>>> improvement is not negligible, which is great!
>>>>>>>>> 
>>>>>>>>> Gary
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> <div>-------- Original message --------</div><div>From:
Ralph Goers <
>>>>>>>>> ralph.goers@dslextreme.com> </div><div>Date:07/11/2014
 01:46
>>>>>>>>> (GMT-05:00) </div><div>To: Log4J Users List
<
>>>>>>>>> log4j-user@logging.apache.org> </div><div>Subject:
Re: Make LogEvent
>>>>>>>>> implementations Externalizable </div><div>
>>>>>>>>> </div>I’d be afraid of breaking compatibility
even now.  However, I
>>>>>>>>> think
>>>>>>>>> what you really want to do is to create an ExternalizedLayout
and
>>>>>>>>> then
>>>>>>>>> 
>>>>>>>>> just
>>>>>>>> 
>>>>>>>> use that instead of the default SerializedLayout.  If you
want to
>>>>>>>>> supply
>>>>>>>>> that Layout as a patch to a Jira issue it could be added
at any time.
>>>>>>>>> 
>>>>>>>>> Ralph
>>>>>>>>> 
>>>>>>>>> On Jul 10, 2014, at 9:23 PM, Scott Harrington <scotth01@sns-usa.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Ralph & co:
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I hear you're gearing up for the release.
>>>>>>>>>> 
>>>>>>>>>> Last weekend I scratched an itch of mine relating
to SocketAppender
>>>>>>>>>> ->
>>>>>>>>>> 
>>>>>>>>>> SocketServer bandwidth, and was able to reduce a
500-character
>>>>>>>>> message
>>>>>>>>> 
>>>>>>>>> from
>>>>>>>> 
>>>>>>>> around 1700 bytes to 700 bytes on the wire (it's easy to
improve on
>>>>>>>>> 
>>>>>>>>> Java's
>>>>>>>> 
>>>>>>>> default serialization).
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> I was going to submit an enhancement request with
patch to JIRA but
>>>>>>>>>> 
>>>>>>>>>> instead I went on vacation for two weeks.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> I made RingBufferLogEvent implement Externalizable,
i.e. hand-coded
>>>>>>>>>> 
>>>>>>>>>> writeExternal / readExternal methods. I did NOT have
time to make
>>>>>>>>> an
>>>>>>>>> equivalent change to Log4jLogEvent, or to write up any
performance
>>>>>>>>> tests
>>>>>>>>> 
>>>>>>>>> or
>>>>>>>> 
>>>>>>>> regression tests.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Should I submit what I have for discussion and hopeful
inclusion in
>>>>>>>>>> 
>>>>>>>>>> 2.0?
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Or will it have to wait for 2.1?
>>>>>>>>>> 
>>>>>>>>>> If we wait, then due to the necessary serialVersionUID
change, v2.0
>>>>>>>>>> 
>>>>>>>>>> SocketAppender would not be able to talk to v2.1
SocketServer or
>>>>>>>>> vice
>>>>>>>>> 
>>>>>>>>> versa
>>>>>>>> 
>>>>>>>> (unless ugly duplicate versions are maintained).
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Below is what the added code looks like. I only tested
in
>>>>>>>>>> 
>>>>>>>>>> RingBufferLogEvent but should be similarly usable
in
>>>>>>>>> Log4jLogEvent, and
>>>>>>>>> perhaps we should discuss a RingBufferLogEvent.readResolve
that makes
>>>>>>>>> 
>>>>>>>>> them
>>>>>>>> 
>>>>>>>> all become Log4jLogEvents on the SocketServer (receiving)
end.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> ...
>>>>>>>>>> 
>>>>>>>>>>   public void writeExternal(ObjectOutput out) throws
IOException {
>>>>>>>>>>       getThrownProxy();
>>>>>>>>>>       out.writeByte(1); // wireFormat
>>>>>>>>>>       int presenceMap = (loggerName == null ? 0 :
0x1) |
>>>>>>>>>>           (marker == null ? 0 : 0x2) |
>>>>>>>>>>           (fqcn == null ? 0 : 0x4) |
>>>>>>>>>>           (level == null ? 0 : 0x8) |
>>>>>>>>>>           (message == null ? 0 : (0x10 |
>>>>>>>>>> (isSerializeAsString(message)
>>>>>>>>>> 
>>>>>>>>>> ? 0 : 0x20))) |
>>>>>>>>> 
>>>>>>>>>            (thrownProxy == null ? 0 : 0x40) |
>>>>>>>>>>           (contextMap == null ? 0 : 0x80) |
>>>>>>>>>>           (contextStack == null ? 0 : 0x100 |
>>>>>>>>>> (contextStack.getDepth()
>>>>>>>>>> 
>>>>>>>>>> == 0 ? 0 : 0x200)) |
>>>>>>>>> 
>>>>>>>>>            (threadName == null ? 0 : 0x400) |
>>>>>>>>>>           (location == null ? 0 : 0x800);
>>>>>>>>>>       out.writeShort(presenceMap);
>>>>>>>>>>       if (loggerName != null) {
>>>>>>>>>>           out.writeUTF(loggerName);
>>>>>>>>>>       }
>>>>>>>>>>       if (marker != null) {
>>>>>>>>>>           out.writeObject(marker);
>>>>>>>>>>       }
>>>>>>>>>>       if (fqcn != null) {
>>>>>>>>>>           out.writeUTF(fqcn);
>>>>>>>>>>       }
>>>>>>>>>>       if (level != null) {
>>>>>>>>>>           out.writeUTF(level.name());
>>>>>>>>>>       }
>>>>>>>>>>       if (message != null) {
>>>>>>>>>>           if (isSerializeAsString(message)) {
>>>>>>>>>>               out.writeUTF(message.getFormattedMessage());
>>>>>>>>>>           }
>>>>>>>>>>           else {
>>>>>>>>>>               out.writeObject(message);
>>>>>>>>>>           }
>>>>>>>>>>       }
>>>>>>>>>>       if (thrownProxy != null) {
>>>>>>>>>>           out.writeObject(thrownProxy);
>>>>>>>>>>       }
>>>>>>>>>>       if (contextMap != null) {
>>>>>>>>>>           writeString2StringMap(out, contextMap);
>>>>>>>>>>       }
>>>>>>>>>>       if (contextStack != null && contextStack.getDepth()
!= 0) {
>>>>>>>>>>           out.writeObject(contextStack);
>>>>>>>>>>       }
>>>>>>>>>>       if (threadName != null) {
>>>>>>>>>>           out.writeUTF(threadName);
>>>>>>>>>>       }
>>>>>>>>>>       if (location != null) {
>>>>>>>>>>           out.writeUTF(location.getClassName());
>>>>>>>>>>           out.writeUTF(location.getMethodName());
>>>>>>>>>>           if ((presenceMap & 0x1000) != 0) {
>>>>>>>>>>               out.writeUTF(location.getFileName());
>>>>>>>>>>           }
>>>>>>>>>>           out.writeInt(location.getLineNumber());
>>>>>>>>>>       }
>>>>>>>>>>       out.writeLong(currentTimeMillis);
>>>>>>>>>>       out.writeBoolean(endOfBatch);
>>>>>>>>>>       out.writeBoolean(includeLocation);
>>>>>>>>>>   }
>>>>>>>>>> 
>>>>>>>>>>   public void readExternal(ObjectInput in) throws
IOException,
>>>>>>>>>> 
>>>>>>>>>> ClassNotFoundException {
>>>>>>>>> 
>>>>>>>>>        int wireFormat = in.readByte();
>>>>>>>>>>       if (wireFormat == 1) {
>>>>>>>>>>           int presenceMap = in.readShort();
>>>>>>>>>>           loggerName = (presenceMap & 0x1) ==
0 ? null :
>>>>>>>>>>               in.readUTF();
>>>>>>>>>>           marker = (presenceMap & 0x2) == 0 ?
null :
>>>>>>>>>>               (Marker) in.readObject();
>>>>>>>>>>           fqcn = (presenceMap & 0x4) == 0 ? null
:
>>>>>>>>>>               in.readUTF();
>>>>>>>>>>           level = (presenceMap & 0x8) == 0 ?
null :
>>>>>>>>>>               Level.valueOf(in.readUTF());
>>>>>>>>>>           message = (presenceMap & 0x10) == 0
? null :
>>>>>>>>>>               (presenceMap & 0x20) == 0 ? new
>>>>>>>>>> 
>>>>>>>>>> SimpleMessage(in.readUTF()) : (Message) in.readObject();
>>>>>>>>> 
>>>>>>>>>            thrownProxy = (presenceMap & 0x40) ==
0 ? null :
>>>>>>>>>>               (ThrowableProxy) in.readObject();
>>>>>>>>>>           contextMap = (presenceMap & 0x80) ==
0 ? null :
>>>>>>>>>>               readString2StringMap(in);
>>>>>>>>>>           contextStack = (presenceMap & 0x100)
== 0 ? null :
>>>>>>>>>>               (presenceMap & 0x200) == 0 ?
>>>>>>>>>> ThreadContext.EMPTY_STACK
>>>>>>>>>> :
>>>>>>>>>> 
>>>>>>>>>> (ContextStack) in.readObject();
>>>>>>>>> 
>>>>>>>>>            threadName = (presenceMap & 0x400) ==
0 ? null :
>>>>>>>>>>               in.readUTF();
>>>>>>>>>>           location = (presenceMap & 0x800) ==
0 ? null :
>>>>>>>>>>               new StackTraceElement(in.readUTF(),
in.readUTF(),
>>>>>>>>>> 
>>>>>>>>>> (presenceMap & 0x1000) == 0 ? null : in.readUTF(),
in.readInt());
>>>>>>>>> 
>>>>>>>>>            currentTimeMillis = in.readLong();
>>>>>>>>>>           endOfBatch = in.readBoolean();
>>>>>>>>>>           includeLocation = in.readBoolean();
>>>>>>>>>>       }
>>>>>>>>>>       else {
>>>>>>>>>>           throw new StreamCorruptedException("Unsupported
LogEvent
>>>>>>>>>> 
>>>>>>>>>> wire
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> format " + wireFormat);
>>>>>>>>> 
>>>>>>>>>        }
>>>>>>>>>>   }
>>>>>>>>>> 
>>>>>>>>>>   private static boolean isSerializeAsString(Message
message)
>>>>>>>>>>   {
>>>>>>>>>>       return message instanceof SimpleMessage ||
message instanceof
>>>>>>>>>> 
>>>>>>>>>> ObjectMessage;
>>>>>>>>> 
>>>>>>>>>    }
>>>>>>>>>> 
>>>>>>>>>>   private void writeString2StringMap(ObjectOutput
out, Map<String,
>>>>>>>>>> 
>>>>>>>>>> String> map) throws IOException
>>>>>>>>> 
>>>>>>>>>    {
>>>>>>>>>>       out.writeInt(map.size());
>>>>>>>>>>       for (Map.Entry<String, String> entry
: map.entrySet()) {
>>>>>>>>>>           out.writeUTF(entry.getKey());
>>>>>>>>>>           out.writeUTF(entry.getValue());
>>>>>>>>>>       }
>>>>>>>>>>   }
>>>>>>>>>> 
>>>>>>>>>>   private static Map<String, String> readString2StringMap(
>>>>>>>>>> ObjectInput
>>>>>>>>>> 
>>>>>>>>>> in) throws ClassNotFoundException, IOException
>>>>>>>>> 
>>>>>>>>>    {
>>>>>>>>>>       int size = in.readInt();
>>>>>>>>>>       if (size == 0) {
>>>>>>>>>>           return Collections.emptyMap();
>>>>>>>>>>       }
>>>>>>>>>>       Map<String, String> map = new HashMap<String,
String>(size);
>>>>>>>>>>       while (size-- > 0) {
>>>>>>>>>>           map.put(in.readUTF(), in.readUTF());
>>>>>>>>>>       }
>>>>>>>>>>       return map;
>>>>>>>>>>   }
>>>>>>>>>> 
>>>>>>>>>> ------------------------------------------------------------
>>>>>>>>>> ---------
>>>>>>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>>>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> ------------------------------------------------------------
>>>>>>>>> ---------
>>>>>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> --
>>>>>>>> Matt Sicker <boards@gmail.com>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>> 
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org


Mime
View raw message