metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (METRON-1717) Relocate Storm Profiler Code
Date Mon, 10 Sep 2018 17:57:00 GMT


ASF GitHub Bot commented on METRON-1717:

Github user merrimanr commented on a diff in the pull request:
    --- Diff: metron-analytics/metron-profiler-common/ ---
    @@ -0,0 +1,386 @@
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +# Metron Profiler
    +* [Introduction](#introduction)
    +* [Getting Started](#getting-started)
    +* [Profiles](#profiles)
    +* [Examples](#examples)
    +## Introduction
    +The Profiler is a feature extraction mechanism that can generate a profile describing
the behavior of an entity.  An entity might be a server, user, subnet or application. Once
a profile has been generated defining what normal behavior looks-like, models can be built
that identify anomalous behavior.
    +This is achieved by summarizing the telemetry data consumed by Metron over tumbling windows.
A summary statistic is applied to the data received within a given window.  Collecting these
values across many windows result in a time series that is useful for analysis.
    +Any field contained within a message can be used to generate a profile.  A profile can
even be produced by combining fields that originate in different data sources.  A user has
considerable power to transform the data used in a profile by leveraging the Stellar language.

    +There are three separate ports of the Profiler that share this common code base.
    +* The [Storm Profiler](../metron-profiler-storm/ builds low-latency profiles
over streaming data sets.
    +* The [Spark Profiler](../metron-profiler-spark/ backfills profiles using archived
    +* The [REPL Profiler](../metron-profiler-repl/ allows profiles to be tested
and debugged within the Stellar REPL.
    +## Getting Started
    +1. [Create a profile](../metron-profiler-repl/ using the Stellar
REPL. Validate your profile using mock data, then apply real, live data.
    +1. [Backfill your profile](../metron-profiler-spark/ using
archived telemetry to see how your profile behaves over time.
    +1. [Deploy your profile](../metron-profiler-storm/ to Storm
to maintain a low-latency profile over a streaming data set.
    +1. [Retrieve your profile data](../metron-profiler-client/ using the Stellar
API so that you can build enrichments, alert on abnormalities
    +1. Explore more ways to create [profiles](#more-examples).
    +## Profiles
    +Let's start with a simple example. The following profile maintains a count of the number
of telemetry messages for each IP source address.  A counter is initialized to 0, then incremented
each time a message is received for a give IP source address.  At regular intervals the count
is flushed and stored. Over time this results in a time series describing the amount of telemetry
received for each IP source address.
    --- End diff --
    give > given

> Relocate Storm Profiler Code
> ----------------------------
>                 Key: METRON-1717
>                 URL:
>             Project: Metron
>          Issue Type: Sub-task
>            Reporter: Nick Allen
>            Assignee: Nick Allen
>            Priority: Major
> The Storm port of the Profiler currently lives in metron-analytics/metron-profiler.  This
should be moved to metron-analytics/metron-profiler-storm.  This would mirror the project
names for the Spark port (metron-profiler-spark) and the REPL port (metron-profiler-repl).
> The package name for the Storm port of the Profiler should be changed to org.apache.metron.profiler.storm.
 This would mimic the package name used for Spark; org.apache.metron.profiler.spark.

This message was sent by Atlassian JIRA

View raw message