logging-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikael Ståldal <mi...@apache.org>
Subject Re: Revisiting the binary logging format idea
Date Thu, 19 Oct 2017 19:05:49 GMT
And don't forget a corresponding LogEventParser.


On 2017-10-19 18:56, Remko Popma wrote:
> I agree with Gary: this is just another Layout. Should not need another
> repo...
> Prototyping on another branch makes sense.
> 
> On Fri, Oct 20, 2017 at 1:23 AM, Gary Gregory <garydgregory@gmail.com>
> wrote:
> 
>> Bleh, ANOTHER repo? We have so many already... but I see what the big
>> picture is. Would this add a new layout in Core? Maybe we should just start
>> with that... then grow...
>>
>> Gary
>>
>> On Thu, Oct 19, 2017 at 9:57 AM, Matt Sicker <boards@gmail.com> wrote:
>>
>>> I mean in the logging services project. So it'd be a new repo.
>>>
>>> On 19 October 2017 at 10:48, Ralph Goers <ralph.goers@dslextreme.com>
>>> wrote:
>>>
>>>> When you say “another component in the project” do you mean logging
>>>> services project or log4j? I’d prefer to see you do this in a separate
>>> repo
>>>> or at least a branch until we understand what it looks like. If it is
>>> going
>>>> to apply to many things it probably makes sense to be a separate repo.
>>>>
>>>> Ralph
>>>>
>>>>> On Oct 19, 2017, at 8:26 AM, Matt Sicker <boards@gmail.com> wrote:
>>>>>
>>>>> For generic structured records, I'd probably go with Avro or Thrift
>>> since
>>>>> LogEvents have a lot of standard fields with only a few optional
>>> map-like
>>>>> structures. For optimized log appending, the binary format was
>> proposed
>>>> as
>>>>> a way to append quickly and without garbage IIRC.
>>>>>
>>>>> On 19 October 2017 at 10:21, Gary Gregory <garydgregory@gmail.com>
>>>> wrote:
>>>>>
>>>>>> What about BSON?
>>>>>>
>>>>>> Gary
>>>>>>
>>>>>> On Oct 19, 2017 08:41, "Matt Sicker" <boards@gmail.com> wrote:
>>>>>>
>>>>>>> I don't have the ticket on hand, but a few months ago, Remko
>>> suggested
>>>> a
>>>>>>> binary logging format that would allow for super fast appends
of
>>>>>>> log-specific information along with companion files for additional
>>>>>> metadata
>>>>>>> not commonly used in log messages. I've been thinking about this
>>> idea a
>>>>>> bit
>>>>>>> in relation to existing structured layouts (both textual and
>> binary),
>>>>>> and I
>>>>>>> was thinking that it might be a useful format to standardize
on for
>>> all
>>>>>> the
>>>>>>> logging projects.
>>>>>>>
>>>>>>> What I'd like to propose is making another component in the project
>>>> that
>>>>>>> would contain a reference implementation of encoding and decoding
>> the
>>>>>>> format in Java, C++, .NET, and PHP (or as a C binding for PHP).
>>>>>>> Potentially, this format could be inclusive with other logging
>>> projects
>>>>>>> like Logstash, Logback, Splunk, etc.
>>>>>>>
>>>>>>> What do you all think? Is this a good idea? Or would this be
>>>> duplicating
>>>>>>> effort from other standards already?
>>>>>>>
>>>>>>> --
>>>>>>> Matt Sicker <boards@gmail.com>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Matt Sicker <boards@gmail.com>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Matt Sicker <boards@gmail.com>
>>>
>>
> 


Mime
View raw message