logging-log4j-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Harrington <scott...@sns-usa.com>
Subject Re: Make LogEvent implementations Externalizable
Date Fri, 11 Jul 2014 17:14:21 GMT
OK! Agree. So the 2.1 enhancement request will be for a 
CompactBinaryLayout on the sending side and corresponding LogEventBridge 
on the receiving side. I'll open it in JIRA when I get back from vacation 
and have a chance to work through an initial implementation, unless 
someone else beats me to it.

Current 'Serializable' form is a bit chatty but is easier (automatic) to 
maintain. Forget I ever said anything about Externalizable.


On Sat, 12 Jul 2014, Remko Popma wrote:

> Yes that is what I had in mind.
>
>
> On Sat, Jul 12, 2014 at 1:53 AM, Scott Harrington <scotth01@sns-usa.com>
> wrote:
>
>> Yes it did appear toByteArray would do the job on the /sending/ side but
>> what about the /receiving/ side? We've got to put some bytes on the wire
>> that will come out as a LogEvent from the readObject() call in
>> ObjectInputStreamLogEventBridge. You can't have the sender be
>> Externalizable and the received class be only Serializable.
>>
>> Unless you're proposing we make a totally new LogEventBridge -- i.e. a
>> 'compact binary' protocol to go alongside the XML and Json versions. If we
>> go that route then you could achieve even more compression since we know
>> every object that we're sending is a LogEvent and even Externalizable has
>> some overhead that we could do away with.
>>
>>
>>
>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>
>>  On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma <remko.popma@gmail.com>
>>> wrote:
>>>
>>>  What prevents you from writing a more compact representation of the
>>>> LogEvent (including all other LogEvent fields) to the byte array in the
>>>> {{toByteArray(LogEvent)}} method of an ExternalizableLayout?
>>>>
>>>>  (Just to clarify, I intended this question literally as it could easily
>>> be
>>> that I overlooked something...)
>>>
>>>
>>>
>>>>
>>>> On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington <scotth01@sns-usa.com
>>>>>
>>>> wrote:
>>>>
>>>>  I looked at replacing SerializedLayout and/or the toSerializable method
>>>>> but the real savings comes not from the Message itself but from all the
>>>>> other fields of the LogEvent, such as the Level, ContextMap,
>>>>> ContextStack,
>>>>> and the class descriptor (including superclasses) of the LogEvent
>>>>> itself.
>>>>>
>>>>> For big messages, an ExternalizableLayout would be a fine solution,
>>>>> because writeUTF would give you roughly 50% compression over Java's
>>>>> default
>>>>> 2-byte per char String serialization.
>>>>>
>>>>> However for typical log messages the message itself is being dwarfed
on
>>>>> the wire by class descriptor overhead.
>>>>>
>>>>>
>>>>>
>>>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>>>>
>>>>>  That is a nice reduction in size!
>>>>>
>>>>>>
>>>>>> I also think the ExternalizedLayout idea is a very attractive option.
>>>>>> That
>>>>>> way there is no pressure to include this in any particular release,
we
>>>>>> can
>>>>>> release it when we are confident that is ready. I also like the fact
>>>>>> that
>>>>>> it does not replace the current serialization and it can be switched
on
>>>>>> and
>>>>>> off in configuration so if there is a bug, users can fall back to
the
>>>>>> existing plain serialization.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker <boards@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>  I would second the ExternalizedLayout. Layouts are the way to go
for
>>>>>>
>>>>>>> compatibility and are simpler. Particularly useful for alternative
>>>>>>> serialization protocols, too.
>>>>>>>
>>>>>>>
>>>>>>> On 11 July 2014 06:41, Gary Gregory <garydgregory@gmail.com>
wrote:
>>>>>>>
>>>>>>>  I understand Ralph ' s concern but now is the time for this
kind of
>>>>>>>
>>>>>>>> change.  Otherwise we will need even more clever solutions
to get
>>>>>>>> this
>>>>>>>>
>>>>>>>>  kind
>>>>>>>
>>>>>>>  of size improvement. I'd love to see some performance numbers.
The
>>>>>>>> size
>>>>>>>> improvement is not negligible, which is great!
>>>>>>>>
>>>>>>>> Gary
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> <div>-------- Original message --------</div><div>From:
Ralph Goers <
>>>>>>>> ralph.goers@dslextreme.com> </div><div>Date:07/11/2014
 01:46
>>>>>>>>  (GMT-05:00) </div><div>To: Log4J Users List
<
>>>>>>>> log4j-user@logging.apache.org> </div><div>Subject:
Re: Make LogEvent
>>>>>>>> implementations Externalizable </div><div>
>>>>>>>> </div>I’d be afraid of breaking compatibility even
now.  However, I
>>>>>>>> think
>>>>>>>> what you really want to do is to create an ExternalizedLayout
and
>>>>>>>> then
>>>>>>>>
>>>>>>>>  just
>>>>>>>
>>>>>>>  use that instead of the default SerializedLayout.  If you want
to
>>>>>>>> supply
>>>>>>>> that Layout as a patch to a Jira issue it could be added
at any time.
>>>>>>>>
>>>>>>>> Ralph
>>>>>>>>
>>>>>>>> On Jul 10, 2014, at 9:23 PM, Scott Harrington <scotth01@sns-usa.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>  Ralph & co:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I hear you're gearing up for the release.
>>>>>>>>>
>>>>>>>>> Last weekend I scratched an itch of mine relating to
SocketAppender
>>>>>>>>> ->
>>>>>>>>>
>>>>>>>>>  SocketServer bandwidth, and was able to reduce a 500-character
>>>>>>>> message
>>>>>>>>
>>>>>>>>  from
>>>>>>>
>>>>>>>  around 1700 bytes to 700 bytes on the wire (it's easy to improve
on
>>>>>>>>
>>>>>>>>  Java's
>>>>>>>
>>>>>>>  default serialization).
>>>>>>>>
>>>>>>>>
>>>>>>>>> I was going to submit an enhancement request with patch
to JIRA but
>>>>>>>>>
>>>>>>>>>  instead I went on vacation for two weeks.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I made RingBufferLogEvent implement Externalizable, i.e.
hand-coded
>>>>>>>>>
>>>>>>>>>  writeExternal / readExternal methods. I did NOT have
time to make
>>>>>>>> an
>>>>>>>> equivalent change to Log4jLogEvent, or to write up any performance
>>>>>>>> tests
>>>>>>>>
>>>>>>>>  or
>>>>>>>
>>>>>>>  regression tests.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Should I submit what I have for discussion and hopeful
inclusion in
>>>>>>>>>
>>>>>>>>>  2.0?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>  Or will it have to wait for 2.1?
>>>>>>>>>
>>>>>>>>> If we wait, then due to the necessary serialVersionUID
change, v2.0
>>>>>>>>>
>>>>>>>>>  SocketAppender would not be able to talk to v2.1 SocketServer
or
>>>>>>>> vice
>>>>>>>>
>>>>>>>>  versa
>>>>>>>
>>>>>>>  (unless ugly duplicate versions are maintained).
>>>>>>>>
>>>>>>>>
>>>>>>>>> Below is what the added code looks like. I only tested
in
>>>>>>>>>
>>>>>>>>>  RingBufferLogEvent but should be similarly usable in
>>>>>>>> Log4jLogEvent, and
>>>>>>>> perhaps we should discuss a RingBufferLogEvent.readResolve
that makes
>>>>>>>>
>>>>>>>>  them
>>>>>>>
>>>>>>>  all become Log4jLogEvents on the SocketServer (receiving) end.
>>>>>>>>
>>>>>>>>
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>>    public void writeExternal(ObjectOutput out) throws
IOException {
>>>>>>>>>        getThrownProxy();
>>>>>>>>>        out.writeByte(1); // wireFormat
>>>>>>>>>        int presenceMap = (loggerName == null ? 0 : 0x1)
|
>>>>>>>>>            (marker == null ? 0 : 0x2) |
>>>>>>>>>            (fqcn == null ? 0 : 0x4) |
>>>>>>>>>            (level == null ? 0 : 0x8) |
>>>>>>>>>            (message == null ? 0 : (0x10 |
>>>>>>>>> (isSerializeAsString(message)
>>>>>>>>>
>>>>>>>>>  ? 0 : 0x20))) |
>>>>>>>>
>>>>>>>>             (thrownProxy == null ? 0 : 0x40) |
>>>>>>>>>            (contextMap == null ? 0 : 0x80) |
>>>>>>>>>            (contextStack == null ? 0 : 0x100 |
>>>>>>>>> (contextStack.getDepth()
>>>>>>>>>
>>>>>>>>>  == 0 ? 0 : 0x200)) |
>>>>>>>>
>>>>>>>>             (threadName == null ? 0 : 0x400) |
>>>>>>>>>            (location == null ? 0 : 0x800);
>>>>>>>>>        out.writeShort(presenceMap);
>>>>>>>>>        if (loggerName != null) {
>>>>>>>>>            out.writeUTF(loggerName);
>>>>>>>>>        }
>>>>>>>>>        if (marker != null) {
>>>>>>>>>            out.writeObject(marker);
>>>>>>>>>        }
>>>>>>>>>        if (fqcn != null) {
>>>>>>>>>            out.writeUTF(fqcn);
>>>>>>>>>        }
>>>>>>>>>        if (level != null) {
>>>>>>>>>            out.writeUTF(level.name());
>>>>>>>>>        }
>>>>>>>>>        if (message != null) {
>>>>>>>>>            if (isSerializeAsString(message)) {
>>>>>>>>>                out.writeUTF(message.getFormattedMessage());
>>>>>>>>>            }
>>>>>>>>>            else {
>>>>>>>>>                out.writeObject(message);
>>>>>>>>>            }
>>>>>>>>>        }
>>>>>>>>>        if (thrownProxy != null) {
>>>>>>>>>            out.writeObject(thrownProxy);
>>>>>>>>>        }
>>>>>>>>>        if (contextMap != null) {
>>>>>>>>>            writeString2StringMap(out, contextMap);
>>>>>>>>>        }
>>>>>>>>>        if (contextStack != null && contextStack.getDepth()
!= 0) {
>>>>>>>>>            out.writeObject(contextStack);
>>>>>>>>>        }
>>>>>>>>>        if (threadName != null) {
>>>>>>>>>            out.writeUTF(threadName);
>>>>>>>>>        }
>>>>>>>>>        if (location != null) {
>>>>>>>>>            out.writeUTF(location.getClassName());
>>>>>>>>>            out.writeUTF(location.getMethodName());
>>>>>>>>>            if ((presenceMap & 0x1000) != 0) {
>>>>>>>>>                out.writeUTF(location.getFileName());
>>>>>>>>>            }
>>>>>>>>>            out.writeInt(location.getLineNumber());
>>>>>>>>>        }
>>>>>>>>>        out.writeLong(currentTimeMillis);
>>>>>>>>>        out.writeBoolean(endOfBatch);
>>>>>>>>>        out.writeBoolean(includeLocation);
>>>>>>>>>    }
>>>>>>>>>
>>>>>>>>>    public void readExternal(ObjectInput in) throws IOException,
>>>>>>>>>
>>>>>>>>>  ClassNotFoundException {
>>>>>>>>
>>>>>>>>         int wireFormat = in.readByte();
>>>>>>>>>        if (wireFormat == 1) {
>>>>>>>>>            int presenceMap = in.readShort();
>>>>>>>>>            loggerName = (presenceMap & 0x1) == 0
? null :
>>>>>>>>>                in.readUTF();
>>>>>>>>>            marker = (presenceMap & 0x2) == 0 ? null
:
>>>>>>>>>                (Marker) in.readObject();
>>>>>>>>>            fqcn = (presenceMap & 0x4) == 0 ? null
:
>>>>>>>>>                in.readUTF();
>>>>>>>>>            level = (presenceMap & 0x8) == 0 ? null
:
>>>>>>>>>                Level.valueOf(in.readUTF());
>>>>>>>>>            message = (presenceMap & 0x10) == 0 ?
null :
>>>>>>>>>                (presenceMap & 0x20) == 0 ? new
>>>>>>>>>
>>>>>>>>>  SimpleMessage(in.readUTF()) : (Message) in.readObject();
>>>>>>>>
>>>>>>>>             thrownProxy = (presenceMap & 0x40) == 0 ?
null :
>>>>>>>>>                (ThrowableProxy) in.readObject();
>>>>>>>>>            contextMap = (presenceMap & 0x80) == 0
? null :
>>>>>>>>>                readString2StringMap(in);
>>>>>>>>>            contextStack = (presenceMap & 0x100) ==
0 ? null :
>>>>>>>>>                (presenceMap & 0x200) == 0 ?
>>>>>>>>> ThreadContext.EMPTY_STACK
>>>>>>>>> :
>>>>>>>>>
>>>>>>>>>  (ContextStack) in.readObject();
>>>>>>>>
>>>>>>>>             threadName = (presenceMap & 0x400) == 0 ?
null :
>>>>>>>>>                in.readUTF();
>>>>>>>>>            location = (presenceMap & 0x800) == 0
? null :
>>>>>>>>>                new StackTraceElement(in.readUTF(), in.readUTF(),
>>>>>>>>>
>>>>>>>>>  (presenceMap & 0x1000) == 0 ? null : in.readUTF(),
in.readInt());
>>>>>>>>
>>>>>>>>             currentTimeMillis = in.readLong();
>>>>>>>>>            endOfBatch = in.readBoolean();
>>>>>>>>>            includeLocation = in.readBoolean();
>>>>>>>>>        }
>>>>>>>>>        else {
>>>>>>>>>            throw new StreamCorruptedException("Unsupported
LogEvent
>>>>>>>>>
>>>>>>>>>  wire
>>>>>>>>
>>>>>>>
>>>>>>>  format " + wireFormat);
>>>>>>>>
>>>>>>>>         }
>>>>>>>>>    }
>>>>>>>>>
>>>>>>>>>    private static boolean isSerializeAsString(Message
message)
>>>>>>>>>    {
>>>>>>>>>        return message instanceof SimpleMessage || message
instanceof
>>>>>>>>>
>>>>>>>>>  ObjectMessage;
>>>>>>>>
>>>>>>>>     }
>>>>>>>>>
>>>>>>>>>    private void writeString2StringMap(ObjectOutput out,
Map<String,
>>>>>>>>>
>>>>>>>>>  String> map) throws IOException
>>>>>>>>
>>>>>>>>     {
>>>>>>>>>        out.writeInt(map.size());
>>>>>>>>>        for (Map.Entry<String, String> entry : map.entrySet())
{
>>>>>>>>>            out.writeUTF(entry.getKey());
>>>>>>>>>            out.writeUTF(entry.getValue());
>>>>>>>>>        }
>>>>>>>>>    }
>>>>>>>>>
>>>>>>>>>    private static Map<String, String> readString2StringMap(
>>>>>>>>> ObjectInput
>>>>>>>>>
>>>>>>>>>  in) throws ClassNotFoundException, IOException
>>>>>>>>
>>>>>>>>     {
>>>>>>>>>        int size = in.readInt();
>>>>>>>>>        if (size == 0) {
>>>>>>>>>            return Collections.emptyMap();
>>>>>>>>>        }
>>>>>>>>>        Map<String, String> map = new HashMap<String,
String>(size);
>>>>>>>>>        while (size-- > 0) {
>>>>>>>>>            map.put(in.readUTF(), in.readUTF());
>>>>>>>>>        }
>>>>>>>>>        return map;
>>>>>>>>>    }
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>>>>>>>> ---------
>>>>>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ------------------------------------------------------------
>>>>>>>> ---------
>>>>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Matt Sicker <boards@gmail.com>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>
>

Mime
View raw message