metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Justin Leet (JIRA)" <>
Subject [jira] [Updated] (METRON-1657) Parser aggregation in storm
Date Wed, 05 Sep 2018 15:18:00 GMT


Justin Leet updated METRON-1657:
    Fix Version/s: 0.6.0

> Parser aggregation in storm
> ---------------------------
>                 Key: METRON-1657
>                 URL:
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Justin Leet
>            Assignee: Justin Leet
>            Priority: Major
>             Fix For: 0.6.0
> Currently our parsing solution requires one storm topology per sensor. It has been complained
that this may be wasteful of resources and that, rather than one storm topology per sensor,
it would be advantageous to have multiple sensors in the same topology. The benefit to this
is that it would require fewer storm slots.
> The issue with this is that whenever we've aggregated functionality like this before,
we've run into issues appropriately being able to scale storm (e.g. batch vs random access
indexing in the same topology).  The main point in addressing this is to recommend that parsers
with similar velocities and complexity are grouped together.
> Particularly for a first cut, leave the configuration mostly as-is, while allowing for
comma separated lists of sensors in (e.g. bro,yaf creates a aggregated
parser consisting of those two). 

This message was sent by Atlassian JIRA

View raw message