metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmed Shah <>
Subject Re: Metron - Emailing Alerts
Date Thu, 14 Dec 2017 19:36:57 GMT
Hello All,

Thanks for the feedback everyone.

The scenario we are trying to account for is when no one is looking at the dashboard for long
periods of time.

A scenario like the following:

When no one is actively looking at the Metron dashboard a tripwire activates on a critical
asset. Without delay the IT team is emailed that a tripwire has been activated. Someone from
the group then logs into Metron to explore more and triage.

Wrt to being bombarded with alerts, a possible solution could be extending the rule with a
time attribute like following?:

Rule C: If "payload" attribute contains "critical" from "ip_dst_addr" =" then email every 10 minutes with [Title(xyz) body:( # of alerts in past 10 min, ip_dst_addr,
ip_src_addr,alert info. ext...)

Being bombarded with an email every 10min will be annoying but the spam should motivate someone
to take an action. When the user gets to the dashboard they can then disable "Rule C", explore,
and reactivate Rule C when they solved the issue.

I'm reviewing the profiler doc:

and trying to cross reference with the stellar lang functions here:

Is this how the profiler could work for the scenario above?









Create a profile named "profile":"criticalalert" with the following:

"foreach": "ip_dst_addr"

"groupBy": [ not sure how to define a 10 min cycle using stellar functions ]

"onlyif": {"'critical' in "payload" AND "ip_dst_addr"== ""},

"init": { "countAlerts" : 0.0},

"update"{"countAlerts": "countAlerts" + 1},

"result": {emailAddress: "", title:"countAlerts", body:ip_dst_addr ext...}

The expectation is we receive the "result" in topicForEmailOut from Kafka.

We use Nifi to connect the kafkatopic to PutEmail  processor.

Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University -<>

Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University -<>

From: Otto Fowler <>
Sent: December 13, 2017 5:24 PM
To:; Simon Elliston Ball
Subject: Re: Metron - Emailing Alerts

We could also filter out of enrichment to a different topology based on
field like Simon has said so that the rules are run on a filtered set etc.

also s/Ever/Either/

On December 13, 2017 at 17:03:15, Otto Fowler (

While summary of _any_ metron data ( perhaps by query etc ) would be good,
let us not lose sight of the OP’s issue.  Ever with summary|digest or one
at a time, they are looking for sending mails to certain people based on

A pseudo path may be

INDEXING -> New Topology or ?? -> evaluate rules -> bin matches to batches
per destination -> create digest from bin’s and send on batch size or
timeout ( as the bulk writer does )

I’m sure there is something wrong with this, but it is easier to frame it
in the way we do it now, and then work from there for me.

On December 13, 2017 at 16:55:35, Simon Elliston Ball ( wrote:

That makes a lot of sense, especially if you wanted the detail in the email
as well. We could definitely use some good "reporting of alerts”
functionality that would make something like that work. What do people


> On 13 Dec 2017, at 21:52, James Sirota <> wrote:
> I think there may be gaps in doing it with the profiler. You can record
stats and counts of different alert types, and maybe even alert ids, but
you can't cross-correlate these IDs to the alert body. At least not in the
profiler. I was thinking about emailing something that looks like a
zeppelin report. You would run it in a cron, export to PDF, and send that
out as a summary. It can be a simple list of alerts that match your rule,
or it can have aggregations, graphics, metrics, KPI screens, etc. That
would be the feature that I would want to discuss and flesh out
> Thanks,
> James
> 13.12.2017, 14:26, "Simon Elliston Ball" <>:
>> We can already do that with profiles I would have thought. Create a
profile that only picks alerts and then base your emails only from the
alert events produced by that profile. Would that create the right batching
mechanism (at a cost of possible higher latency than you might get with a
more specific alert batcher?)
>> Simon
>>> On 13 Dec 2017, at 21:23, James Sirota <> wrote:
>>> I agree with Simon. If you email each alert individually you will be
overwhelmed. I think a better idea would be to email alert summaries
periodically, which is more manageable. This is probably a feature worthy
of consideration for Metron.
>>> 13.12.2017, 12:19, "Simon Elliston Ball" <>:
>>>> Metron generates alerts onto a Kafka queue, which can be used to
integrate with Alert management tools, usually some sort of existing alert
aggregation tool.
>>>> An alternative approach common with this is to have a tool like Apache
NiFi attach to the Metron alert feed and send email.
>>>> The solution here would be to have Metron generate alerts (by adding
the is_alert: true flag in the enrichment process) and possibly other flags
like alert_email for example, and then have NiFi use ConsumeKafka and then
filter out the alert only messages in NiFi to use the PutEmail processor
(probably with a ControlRate before it too).
>>>> Something I would caution is that email is not a great way to manage
or send alerts at the volume likely to occur in network monitoring tools. A
spike in network traffic can lead to a very large number of emails, which
tends to then cause you bigger problems. As such we usually find people
want some sort of buffering or aggregation of alerts, hence the use of a an
alert management or ticketing solution in front.
>>>> Simon
>>>>> On 13 Dec 2017, at 19:06, Ahmed Shah <>
>>>>> Hello,
>>>>> Just wondering if Metron has a feature to email alerts based on rules
that a user defines.
>>>>> Example:
>>>>> Rule A: Email the user whenever ip_src_addr=100.2.10.*
>>>>> Rule B: Email the user whenever payload contains "critical"
>>>>> If not, does anyone have any recommendations on where to code these
rules in the Metron stack that uses attributes from the GROK parser?
>>>>> -Ahmed
>>>>> _______________________________________________________________
>>>>> Ahmed Shah (PMP, M. Eng.)
>>>>> Cybersecurity Analyst & Developer
>>>>> GCR - Cybersecurity Operations Center
>>>>> Carleton University -<>
>>> -------------------
>>> Thank you,
>>> James Sirota
>>> PMC- Apache Metron
>>> jsirota AT apache DOT org
> -------------------
> Thank you,
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message