metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
Date Thu, 06 Sep 2018 17:29:01 GMT

    [ https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606123#comment-16606123
] 

ASF GitHub Bot commented on METRON-1761:
----------------------------------------

Github user mmiklavc commented on a diff in the pull request:

    https://github.com/apache/metron/pull/1184#discussion_r215710648
  
    --- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
---
    @@ -0,0 +1,112 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.metron.parsers;
    +
    +import com.fasterxml.jackson.databind.ObjectMapper;
    +import org.adrianwalker.multilinestring.Multiline;
    +import org.apache.commons.io.IOUtils;
    +import org.json.simple.JSONObject;
    +import org.json.simple.parser.JSONParser;
    +import org.json.simple.parser.ParseException;
    +import org.junit.Assert;
    +import org.junit.Test;
    +
    +import java.io.File;
    +import java.io.FileInputStream;
    +import java.io.FileOutputStream;
    +import java.io.IOException;
    +import java.util.ArrayList;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +public class MultiLineGrokParserTest extends GrokParserTest {
    --- End diff --
    
    Also, can you add some tests around parsing failure scenarios in the batch case?


> Allow a grok statement to be applied to each line in a file.
> ------------------------------------------------------------
>
>                 Key: METRON-1761
>                 URL: https://issues.apache.org/jira/browse/METRON-1761
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Laurens Vets
>            Assignee: Otto Fowler
>            Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is to be parsed
separately) and send them to Metron without having to split the content.
> Example content of a log file where a grok statement needs to be applied to each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.000073
0.001048 0.000057 200 200 0 29 "GET http://www.example.com:80/ HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.000086
0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA
TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.001069
0.000028 0.000041 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80 0.001065
0.000015 0.000023 - - 57 502 "- - - " "-" ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message