logging-log4j-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remko Popma <remko.po...@gmail.com>
Subject Re: Make LogEvent implementations Externalizable
Date Fri, 11 Jul 2014 17:03:15 GMT
Yes that is what I had in mind.


On Sat, Jul 12, 2014 at 1:53 AM, Scott Harrington <scotth01@sns-usa.com>
wrote:

> Yes it did appear toByteArray would do the job on the /sending/ side but
> what about the /receiving/ side? We've got to put some bytes on the wire
> that will come out as a LogEvent from the readObject() call in
> ObjectInputStreamLogEventBridge. You can't have the sender be
> Externalizable and the received class be only Serializable.
>
> Unless you're proposing we make a totally new LogEventBridge -- i.e. a
> 'compact binary' protocol to go alongside the XML and Json versions. If we
> go that route then you could achieve even more compression since we know
> every object that we're sending is a LogEvent and even Externalizable has
> some overhead that we could do away with.
>
>
>
> On Sat, 12 Jul 2014, Remko Popma wrote:
>
>  On Sat, Jul 12, 2014 at 12:52 AM, Remko Popma <remko.popma@gmail.com>
>> wrote:
>>
>>  What prevents you from writing a more compact representation of the
>>> LogEvent (including all other LogEvent fields) to the byte array in the
>>> {{toByteArray(LogEvent)}} method of an ExternalizableLayout?
>>>
>>>  (Just to clarify, I intended this question literally as it could easily
>> be
>> that I overlooked something...)
>>
>>
>>
>>>
>>> On Sat, Jul 12, 2014 at 12:45 AM, Scott Harrington <scotth01@sns-usa.com
>>> >
>>> wrote:
>>>
>>>  I looked at replacing SerializedLayout and/or the toSerializable method
>>>> but the real savings comes not from the Message itself but from all the
>>>> other fields of the LogEvent, such as the Level, ContextMap,
>>>> ContextStack,
>>>> and the class descriptor (including superclasses) of the LogEvent
>>>> itself.
>>>>
>>>> For big messages, an ExternalizableLayout would be a fine solution,
>>>> because writeUTF would give you roughly 50% compression over Java's
>>>> default
>>>> 2-byte per char String serialization.
>>>>
>>>> However for typical log messages the message itself is being dwarfed on
>>>> the wire by class descriptor overhead.
>>>>
>>>>
>>>>
>>>> On Sat, 12 Jul 2014, Remko Popma wrote:
>>>>
>>>>  That is a nice reduction in size!
>>>>
>>>>>
>>>>> I also think the ExternalizedLayout idea is a very attractive option.
>>>>> That
>>>>> way there is no pressure to include this in any particular release, we
>>>>> can
>>>>> release it when we are confident that is ready. I also like the fact
>>>>> that
>>>>> it does not replace the current serialization and it can be switched
on
>>>>> and
>>>>> off in configuration so if there is a bug, users can fall back to the
>>>>> existing plain serialization.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 11, 2014 at 10:00 PM, Matt Sicker <boards@gmail.com>
>>>>> wrote:
>>>>>
>>>>>  I would second the ExternalizedLayout. Layouts are the way to go for
>>>>>
>>>>>> compatibility and are simpler. Particularly useful for alternative
>>>>>> serialization protocols, too.
>>>>>>
>>>>>>
>>>>>> On 11 July 2014 06:41, Gary Gregory <garydgregory@gmail.com>
wrote:
>>>>>>
>>>>>>  I understand Ralph ' s concern but now is the time for this kind
of
>>>>>>
>>>>>>> change.  Otherwise we will need even more clever solutions to
get
>>>>>>> this
>>>>>>>
>>>>>>>  kind
>>>>>>
>>>>>>  of size improvement. I'd love to see some performance numbers. The
>>>>>>> size
>>>>>>> improvement is not negligible, which is great!
>>>>>>>
>>>>>>> Gary
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> <div>-------- Original message --------</div><div>From:
Ralph Goers <
>>>>>>> ralph.goers@dslextreme.com> </div><div>Date:07/11/2014
 01:46
>>>>>>>  (GMT-05:00) </div><div>To: Log4J Users List <
>>>>>>> log4j-user@logging.apache.org> </div><div>Subject:
Re: Make LogEvent
>>>>>>> implementations Externalizable </div><div>
>>>>>>> </div>I’d be afraid of breaking compatibility even now.
 However, I
>>>>>>> think
>>>>>>> what you really want to do is to create an ExternalizedLayout
and
>>>>>>> then
>>>>>>>
>>>>>>>  just
>>>>>>
>>>>>>  use that instead of the default SerializedLayout.  If you want to
>>>>>>> supply
>>>>>>> that Layout as a patch to a Jira issue it could be added at any
time.
>>>>>>>
>>>>>>> Ralph
>>>>>>>
>>>>>>> On Jul 10, 2014, at 9:23 PM, Scott Harrington <scotth01@sns-usa.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  Ralph & co:
>>>>>>>
>>>>>>>>
>>>>>>>> I hear you're gearing up for the release.
>>>>>>>>
>>>>>>>> Last weekend I scratched an itch of mine relating to SocketAppender
>>>>>>>> ->
>>>>>>>>
>>>>>>>>  SocketServer bandwidth, and was able to reduce a 500-character
>>>>>>> message
>>>>>>>
>>>>>>>  from
>>>>>>
>>>>>>  around 1700 bytes to 700 bytes on the wire (it's easy to improve
on
>>>>>>>
>>>>>>>  Java's
>>>>>>
>>>>>>  default serialization).
>>>>>>>
>>>>>>>
>>>>>>>> I was going to submit an enhancement request with patch to
JIRA but
>>>>>>>>
>>>>>>>>  instead I went on vacation for two weeks.
>>>>>>>
>>>>>>>
>>>>>>>> I made RingBufferLogEvent implement Externalizable, i.e.
hand-coded
>>>>>>>>
>>>>>>>>  writeExternal / readExternal methods. I did NOT have time
to make
>>>>>>> an
>>>>>>> equivalent change to Log4jLogEvent, or to write up any performance
>>>>>>> tests
>>>>>>>
>>>>>>>  or
>>>>>>
>>>>>>  regression tests.
>>>>>>>
>>>>>>>
>>>>>>>> Should I submit what I have for discussion and hopeful inclusion
in
>>>>>>>>
>>>>>>>>  2.0?
>>>>>>>
>>>>>>
>>>>>>
>>>>>>>  Or will it have to wait for 2.1?
>>>>>>>>
>>>>>>>> If we wait, then due to the necessary serialVersionUID change,
v2.0
>>>>>>>>
>>>>>>>>  SocketAppender would not be able to talk to v2.1 SocketServer
or
>>>>>>> vice
>>>>>>>
>>>>>>>  versa
>>>>>>
>>>>>>  (unless ugly duplicate versions are maintained).
>>>>>>>
>>>>>>>
>>>>>>>> Below is what the added code looks like. I only tested in
>>>>>>>>
>>>>>>>>  RingBufferLogEvent but should be similarly usable in
>>>>>>> Log4jLogEvent, and
>>>>>>> perhaps we should discuss a RingBufferLogEvent.readResolve that
makes
>>>>>>>
>>>>>>>  them
>>>>>>
>>>>>>  all become Log4jLogEvents on the SocketServer (receiving) end.
>>>>>>>
>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>>    public void writeExternal(ObjectOutput out) throws IOException
{
>>>>>>>>        getThrownProxy();
>>>>>>>>        out.writeByte(1); // wireFormat
>>>>>>>>        int presenceMap = (loggerName == null ? 0 : 0x1) |
>>>>>>>>            (marker == null ? 0 : 0x2) |
>>>>>>>>            (fqcn == null ? 0 : 0x4) |
>>>>>>>>            (level == null ? 0 : 0x8) |
>>>>>>>>            (message == null ? 0 : (0x10 |
>>>>>>>> (isSerializeAsString(message)
>>>>>>>>
>>>>>>>>  ? 0 : 0x20))) |
>>>>>>>
>>>>>>>             (thrownProxy == null ? 0 : 0x40) |
>>>>>>>>            (contextMap == null ? 0 : 0x80) |
>>>>>>>>            (contextStack == null ? 0 : 0x100 |
>>>>>>>> (contextStack.getDepth()
>>>>>>>>
>>>>>>>>  == 0 ? 0 : 0x200)) |
>>>>>>>
>>>>>>>             (threadName == null ? 0 : 0x400) |
>>>>>>>>            (location == null ? 0 : 0x800);
>>>>>>>>        out.writeShort(presenceMap);
>>>>>>>>        if (loggerName != null) {
>>>>>>>>            out.writeUTF(loggerName);
>>>>>>>>        }
>>>>>>>>        if (marker != null) {
>>>>>>>>            out.writeObject(marker);
>>>>>>>>        }
>>>>>>>>        if (fqcn != null) {
>>>>>>>>            out.writeUTF(fqcn);
>>>>>>>>        }
>>>>>>>>        if (level != null) {
>>>>>>>>            out.writeUTF(level.name());
>>>>>>>>        }
>>>>>>>>        if (message != null) {
>>>>>>>>            if (isSerializeAsString(message)) {
>>>>>>>>                out.writeUTF(message.getFormattedMessage());
>>>>>>>>            }
>>>>>>>>            else {
>>>>>>>>                out.writeObject(message);
>>>>>>>>            }
>>>>>>>>        }
>>>>>>>>        if (thrownProxy != null) {
>>>>>>>>            out.writeObject(thrownProxy);
>>>>>>>>        }
>>>>>>>>        if (contextMap != null) {
>>>>>>>>            writeString2StringMap(out, contextMap);
>>>>>>>>        }
>>>>>>>>        if (contextStack != null && contextStack.getDepth()
!= 0) {
>>>>>>>>            out.writeObject(contextStack);
>>>>>>>>        }
>>>>>>>>        if (threadName != null) {
>>>>>>>>            out.writeUTF(threadName);
>>>>>>>>        }
>>>>>>>>        if (location != null) {
>>>>>>>>            out.writeUTF(location.getClassName());
>>>>>>>>            out.writeUTF(location.getMethodName());
>>>>>>>>            if ((presenceMap & 0x1000) != 0) {
>>>>>>>>                out.writeUTF(location.getFileName());
>>>>>>>>            }
>>>>>>>>            out.writeInt(location.getLineNumber());
>>>>>>>>        }
>>>>>>>>        out.writeLong(currentTimeMillis);
>>>>>>>>        out.writeBoolean(endOfBatch);
>>>>>>>>        out.writeBoolean(includeLocation);
>>>>>>>>    }
>>>>>>>>
>>>>>>>>    public void readExternal(ObjectInput in) throws IOException,
>>>>>>>>
>>>>>>>>  ClassNotFoundException {
>>>>>>>
>>>>>>>         int wireFormat = in.readByte();
>>>>>>>>        if (wireFormat == 1) {
>>>>>>>>            int presenceMap = in.readShort();
>>>>>>>>            loggerName = (presenceMap & 0x1) == 0 ? null
:
>>>>>>>>                in.readUTF();
>>>>>>>>            marker = (presenceMap & 0x2) == 0 ? null :
>>>>>>>>                (Marker) in.readObject();
>>>>>>>>            fqcn = (presenceMap & 0x4) == 0 ? null :
>>>>>>>>                in.readUTF();
>>>>>>>>            level = (presenceMap & 0x8) == 0 ? null :
>>>>>>>>                Level.valueOf(in.readUTF());
>>>>>>>>            message = (presenceMap & 0x10) == 0 ? null
:
>>>>>>>>                (presenceMap & 0x20) == 0 ? new
>>>>>>>>
>>>>>>>>  SimpleMessage(in.readUTF()) : (Message) in.readObject();
>>>>>>>
>>>>>>>             thrownProxy = (presenceMap & 0x40) == 0 ? null
:
>>>>>>>>                (ThrowableProxy) in.readObject();
>>>>>>>>            contextMap = (presenceMap & 0x80) == 0 ? null
:
>>>>>>>>                readString2StringMap(in);
>>>>>>>>            contextStack = (presenceMap & 0x100) == 0
? null :
>>>>>>>>                (presenceMap & 0x200) == 0 ?
>>>>>>>> ThreadContext.EMPTY_STACK
>>>>>>>> :
>>>>>>>>
>>>>>>>>  (ContextStack) in.readObject();
>>>>>>>
>>>>>>>             threadName = (presenceMap & 0x400) == 0 ? null
:
>>>>>>>>                in.readUTF();
>>>>>>>>            location = (presenceMap & 0x800) == 0 ? null
:
>>>>>>>>                new StackTraceElement(in.readUTF(), in.readUTF(),
>>>>>>>>
>>>>>>>>  (presenceMap & 0x1000) == 0 ? null : in.readUTF(), in.readInt());
>>>>>>>
>>>>>>>             currentTimeMillis = in.readLong();
>>>>>>>>            endOfBatch = in.readBoolean();
>>>>>>>>            includeLocation = in.readBoolean();
>>>>>>>>        }
>>>>>>>>        else {
>>>>>>>>            throw new StreamCorruptedException("Unsupported
LogEvent
>>>>>>>>
>>>>>>>>  wire
>>>>>>>
>>>>>>
>>>>>>  format " + wireFormat);
>>>>>>>
>>>>>>>         }
>>>>>>>>    }
>>>>>>>>
>>>>>>>>    private static boolean isSerializeAsString(Message message)
>>>>>>>>    {
>>>>>>>>        return message instanceof SimpleMessage || message
instanceof
>>>>>>>>
>>>>>>>>  ObjectMessage;
>>>>>>>
>>>>>>>     }
>>>>>>>>
>>>>>>>>    private void writeString2StringMap(ObjectOutput out, Map<String,
>>>>>>>>
>>>>>>>>  String> map) throws IOException
>>>>>>>
>>>>>>>     {
>>>>>>>>        out.writeInt(map.size());
>>>>>>>>        for (Map.Entry<String, String> entry : map.entrySet())
{
>>>>>>>>            out.writeUTF(entry.getKey());
>>>>>>>>            out.writeUTF(entry.getValue());
>>>>>>>>        }
>>>>>>>>    }
>>>>>>>>
>>>>>>>>    private static Map<String, String> readString2StringMap(
>>>>>>>> ObjectInput
>>>>>>>>
>>>>>>>>  in) throws ClassNotFoundException, IOException
>>>>>>>
>>>>>>>     {
>>>>>>>>        int size = in.readInt();
>>>>>>>>        if (size == 0) {
>>>>>>>>            return Collections.emptyMap();
>>>>>>>>        }
>>>>>>>>        Map<String, String> map = new HashMap<String,
String>(size);
>>>>>>>>        while (size-- > 0) {
>>>>>>>>            map.put(in.readUTF(), in.readUTF());
>>>>>>>>        }
>>>>>>>>        return map;
>>>>>>>>    }
>>>>>>>>
>>>>>>>> ------------------------------------------------------------
>>>>>>>> ---------
>>>>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ------------------------------------------------------------
>>>>>>> ---------
>>>>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Matt Sicker <boards@gmail.com>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
>>>> For additional commands, e-mail: log4j-user-help@logging.apache.org
>>>>
>>>>
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message