logging-log4j-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remko Popma <remko.po...@gmail.com>
Subject Re: Make LogEvent implementations Externalizable
Date Fri, 11 Jul 2014 16:36:45 GMT
On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma <remko.popma@gmail.com> wrote:

> What prevents you from writing a more compact representation of the
> LogEvent (including all other LogEvent fields) to the byte array in the
> {{toByteArray(LogEvent)}} method of an ExternalizableLayout?
>
(Just to clarify, I intended this question literally as it could easily be
that I overlooked something...)


>
>
> On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington <scotth01@sns-usa.com>
> wrote:
>
>> I looked at replacing SerializedLayout and/or the toSerializable method
>> but the real savings comes not from the Message itself but from all the
>> other fields of the LogEvent, such as the Level, ContextMap, ContextStack,
>> and the class descriptor (including superclasses) of the LogEvent itself.
>>
>> For big messages, an ExternalizableLayout would be a fine solution,
>> because writeUTF would give you roughly 50% compression over Java's default
>> 2-byte per char String serialization.
>>
>> However for typical log messages the message itself is being dwarfed on
>> the wire by class descriptor overhead.
>>
>>
>>
>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>
>>  That is a nice reduction in size!
>>>
>>> I also think the ExternalizedLayout idea is a very attractive option.
>>> That
>>> way there is no pressure to include this in any particular release, we
>>> can
>>> release it when we are confident that is ready. I also like the fact that
>>> it does not replace the current serialization and it can be switched on
>>> and
>>> off in configuration so if there is a bug, users can fall back to the
>>> existing plain serialization.
>>>
>>>
>>>
>>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker <boards@gmail.com> wrote:
>>>
>>>  I would second the ExternalizedLayout. Layouts are the way to go for
>>>> compatibility and are simpler. Particularly useful for alternative
>>>> serialization protocols, too.
>>>>
>>>>
>>>> On 11 July 2014 06:41, Gary Gregory <garydgregory@gmail.com> wrote:
>>>>
>>>>  I understand Ralph ' s concern but now is the time for this kind of
>>>>> change.  Otherwise we will need even more clever solutions to get this
>>>>>
>>>> kind
>>>>
>>>>> of size improvement. I'd love to see some performance numbers. The size
>>>>> improvement is not negligible, which is great!
>>>>>
>>>>> Gary
>>>>>
>>>>>
>>>>>
>>>>> <div>-------- Original message --------</div><div>From:
Ralph Goers <
>>>>> ralph.goers@dslextreme.com> </div><div>Date:07/11/2014
 01:46
>>>>>  (GMT-05:00) </div><div>To: Log4J Users List <
>>>>> log4j-user@logging.apache.org> </div><div>Subject: Re:
Make LogEvent
>>>>> implementations Externalizable </div><div>
>>>>> </div>I’d be afraid of breaking compatibility even now.  However,
I
>>>>> think
>>>>> what you really want to do is to create an ExternalizedLayout and then
>>>>>
>>>> just
>>>>
>>>>> use that instead of the default SerializedLayout.  If you want to
>>>>> supply
>>>>> that Layout as a patch to a Jira issue it could be added at any time.
>>>>>
>>>>> Ralph
>>>>>
>>>>> On Jul 10, 2014, at 9:23 PM, Scott Harrington <scotth01@sns-usa.com>
>>>>> wrote:
>>>>>
>>>>>  Ralph & co:
>>>>>>
>>>>>> I hear you're gearing up for the release.
>>>>>>
>>>>>> Last weekend I scratched an itch of mine relating to SocketAppender
->
>>>>>>
>>>>> SocketServer bandwidth, and was able to reduce a 500-character message
>>>>>
>>>> from
>>>>
>>>>> around 1700 bytes to 700 bytes on the wire (it's easy to improve on
>>>>>
>>>> Java's
>>>>
>>>>> default serialization).
>>>>>
>>>>>>
>>>>>> I was going to submit an enhancement request with patch to JIRA but
>>>>>>
>>>>> instead I went on vacation for two weeks.
>>>>>
>>>>>>
>>>>>> I made RingBufferLogEvent implement Externalizable, i.e. hand-coded
>>>>>>
>>>>> writeExternal / readExternal methods. I did NOT have time to make an
>>>>> equivalent change to Log4jLogEvent, or to write up any performance
>>>>> tests
>>>>>
>>>> or
>>>>
>>>>> regression tests.
>>>>>
>>>>>>
>>>>>> Should I submit what I have for discussion and hopeful inclusion
in
>>>>>>
>>>>> 2.0?
>>>>
>>>>>
>>>>>> Or will it have to wait for 2.1?
>>>>>>
>>>>>> If we wait, then due to the necessary serialVersionUID change, v2.0
>>>>>>
>>>>> SocketAppender would not be able to talk to v2.1 SocketServer or vice
>>>>>
>>>> versa
>>>>
>>>>> (unless ugly duplicate versions are maintained).
>>>>>
>>>>>>
>>>>>> Below is what the added code looks like. I only tested in
>>>>>>
>>>>> RingBufferLogEvent but should be similarly usable in Log4jLogEvent, and
>>>>> perhaps we should discuss a RingBufferLogEvent.readResolve that makes
>>>>>
>>>> them
>>>>
>>>>> all become Log4jLogEvents on the SocketServer (receiving) end.
>>>>>
>>>>>>
>>>>>> ...
>>>>>>
>>>>>>    public void writeExternal(ObjectOutput out) throws IOException
{
>>>>>>        getThrownProxy();
>>>>>>        out.writeByte(1); // wireFormat
>>>>>>        int presenceMap = (loggerName == null ? 0 : 0x1) |
>>>>>>            (marker == null ? 0 : 0x2) |
>>>>>>            (fqcn == null ? 0 : 0x4) |
>>>>>>            (level == null ? 0 : 0x8) |
>>>>>>            (message == null ? 0 : (0x10 |
>>>>>> (isSerializeAsString(message)
>>>>>>
>>>>> ? 0 : 0x20))) |
>>>>>
>>>>>>            (thrownProxy == null ? 0 : 0x40) |
>>>>>>            (contextMap == null ? 0 : 0x80) |
>>>>>>            (contextStack == null ? 0 : 0x100 |
>>>>>> (contextStack.getDepth()
>>>>>>
>>>>> == 0 ? 0 : 0x200)) |
>>>>>
>>>>>>            (threadName == null ? 0 : 0x400) |
>>>>>>            (location == null ? 0 : 0x800);
>>>>>>        out.writeShort(presenceMap);
>>>>>>        if (loggerName != null) {
>>>>>>            out.writeUTF(loggerName);
>>>>>>        }
>>>>>>        if (marker != null) {
>>>>>>            out.writeObject(marker);
>>>>>>        }
>>>>>>        if (fqcn != null) {
>>>>>>            out.writeUTF(fqcn);
>>>>>>        }
>>>>>>        if (level != null) {
>>>>>>            out.writeUTF(level.name());
>>>>>>        }
>>>>>>        if (message != null) {
>>>>>>            if (isSerializeAsString(message)) {
>>>>>>                out.writeUTF(message.getFormattedMessage());
>>>>>>            }
>>>>>>            else {
>>>>>>                out.writeObject(message);
>>>>>>            }
>>>>>>        }
>>>>>>        if (thrownProxy != null) {
>>>>>>            out.writeObject(thrownProxy);
>>>>>>        }
>>>>>>        if (contextMap != null) {
>>>>>>            writeString2StringMap(out, contextMap);
>>>>>>        }
>>>>>>        if (contextStack != null && contextStack.getDepth()
!= 0) {
>>>>>>            out.writeObject(contextStack);
>>>>>>        }
>>>>>>        if (threadName != null) {
>>>>>>            out.writeUTF(threadName);
>>>>>>        }
>>>>>>        if (location != null) {
>>>>>>            out.writeUTF(location.getClassName());
>>>>>>            out.writeUTF(location.getMethodName());
>>>>>>            if ((presenceMap & 0x1000) != 0) {
>>>>>>                out.writeUTF(location.getFileName());
>>>>>>            }
>>>>>>            out.writeInt(location.getLineNumber());
>>>>>>        }
>>>>>>        out.writeLong(currentTimeMillis);
>>>>>>        out.writeBoolean(endOfBatch);
>>>>>>        out.writeBoolean(includeLocation);
>>>>>>    }
>>>>>>
>>>>>>    public void readExternal(ObjectInput in) throws IOException,
>>>>>>
>>>>> ClassNotFoundException {
>>>>>
>>>>>>        int wireFormat = in.readByte();
>>>>>>        if (wireFormat == 1) {
>>>>>>            int presenceMap = in.readShort();
>>>>>>            loggerName = (presenceMap & 0x1) == 0 ? null :
>>>>>>                in.readUTF();
>>>>>>            marker = (presenceMap & 0x2) == 0 ? null :
>>>>>>                (Marker) in.readObject();
>>>>>>            fqcn = (presenceMap & 0x4) == 0 ? null :
>>>>>>                in.readUTF();
>>>>>>            level = (presenceMap & 0x8) == 0 ? null :
>>>>>>                Level.valueOf(in.readUTF());
>>>>>>            message = (presenceMap & 0x10) == 0 ? null :
>>>>>>                (presenceMap & 0x20) == 0 ? new
>>>>>>
>>>>> SimpleMessage(in.readUTF()) : (Message) in.readObject();
>>>>>
>>>>>>            thrownProxy = (presenceMap & 0x40) == 0 ? null :
>>>>>>                (ThrowableProxy) in.readObject();
>>>>>>            contextMap = (presenceMap & 0x80) == 0 ? null :
>>>>>>                readString2StringMap(in);
>>>>>>            contextStack = (presenceMap & 0x100) == 0 ? null :
>>>>>>                (presenceMap & 0x200) == 0 ? ThreadContext.EMPTY_STACK
>>>>>> :
>>>>>>
>>>>> (ContextStack) in.readObject();
>>>>>
>>>>>>            threadName = (presenceMap & 0x400) == 0 ? null :
>>>>>>                in.readUTF();
>>>>>>            location = (presenceMap & 0x800) == 0 ? null :
>>>>>>                new StackTraceElement(in.readUTF(), in.readUTF(),
>>>>>>
>>>>> (presenceMap & 0x1000) == 0 ? null : in.readUTF(), in.readInt());
>>>>>
>>>>>>            currentTimeMillis = in.readLong();
>>>>>>            endOfBatch = in.readBoolean();
>>>>>>            includeLocation = in.readBoolean();
>>>>>>        }
>>>>>>        else {
>>>>>>            throw new StreamCorruptedException("Unsupported LogEvent
>>>>>>
>>>>> wire
>>>>
>>>>> format " + wireFormat);
>>>>>
>>>>>>        }
>>>>>>    }
>>>>>>
>>>>>>    private static boolean isSerializeAsString(Message message)
>>>>>>    {
>>>>>>        return message instanceof SimpleMessage || message instanceof
>>>>>>
>>>>> ObjectMessage;
>>>>>
>>>>>>    }
>>>>>>
>>>>>>    private void writeString2StringMap(ObjectOutput out, Map<String,
>>>>>>
>>>>> String> map) throws IOException
>>>>>
>>>>>>    {
>>>>>>        out.writeInt(map.size());
>>>>>>        for (Map.Entry<String, String> entry : map.entrySet())
{
>>>>>>            out.writeUTF(entry.getKey());
>>>>>>            out.writeUTF(entry.getValue());
>>>>>>        }
>>>>>>    }
>>>>>>
>>>>>>    private static Map<String, String> readString2StringMap(
>>>>>> ObjectInput
>>>>>>
>>>>> in) throws ClassNotFoundException, IOException
>>>>>
>>>>>>    {
>>>>>>        int size = in.readInt();
>>>>>>        if (size == 0) {
>>>>>>            return Collections.emptyMap();
>>>>>>        }
>>>>>>        Map<String, String> map = new HashMap<String, String>(size);
>>>>>>        while (size-- > 0) {
>>>>>>            map.put(in.readUTF(), in.readUTF());
>>>>>>        }
>>>>>>        return map;
>>>>>>    }
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Matt Sicker <boards@gmail.com>
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message