metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
Date Wed, 03 Oct 2018 18:21:00 GMT

    [ https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637336#comment-16637336
] 

ASF GitHub Bot commented on METRON-1761:
----------------------------------------

Github user mmiklavc commented on the issue:

    https://github.com/apache/metron/pull/1184
  
    @ottobackwards I see what you're saying. It looks like that could definitely work. Thinking
out loud here, but might that conflate the semantics of our validation a bit? Validate currently
does things like ensure that a timestamp exists on the message, though I don't see why we
couldn't expand it to validations outside of our global Metron context.
    
    One class that might be worth checking out is the unified enrichment topology. This was
changed to include a parallel enricher that handles errors and message results in an EnrichmentResult
class.
    
    1. https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L270
    2. https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java#L63
    
    It looks to me like there might be some possible collaboration opportunity and overlap
with the work you're doing here and the work @merrimanr is doing on this PR - https://github.com/apache/metron/pull/1213#pullrequestreview-161248142
    
    I'm just wondering if we might be able to kill 2 birds with one stone. We probably don't
want to change the MessageParser interface, but maybe we can manage the bulk processing through
a more generalized bridge between the ParserBolt and parser implementations. I haven't dug
too deep into implementation feasibility, but it seems worth considering.


> Allow a grok statement to be applied to each line in a file.
> ------------------------------------------------------------
>
>                 Key: METRON-1761
>                 URL: https://issues.apache.org/jira/browse/METRON-1761
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Laurens Vets
>            Assignee: Otto Fowler
>            Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is to be parsed
separately) and send them to Metron without having to split the content.
> Example content of a log file where a grok statement needs to be applied to each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.000073
0.001048 0.000057 200 200 0 29 "GET http://www.example.com:80/ HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.000086
0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA
TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.001069
0.000028 0.000041 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.001065
0.000015 0.000023 - - 57 502 "- - - " "-" ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message