metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Richardson <kylerichards...@gmail.com>
Subject Re: [DISCUSS] Moving GeoIP management away from MySQL
Date Mon, 16 Jan 2017 17:52:52 GMT
+1 Agree with David's order

-Kyle

On Mon, Jan 16, 2017 at 12:41 PM, David Lyle <dlyle65535@gmail.com> wrote:

> Def agree on the parity point.
>
> I'm a little worried about Supervisor relocations for non-HBase solutions,
> but having much of the work done for us by MaxMind changes my preference to
> (in order)
>
> 1) MM API
> 2) HBase Enrichment
> 3) MapDB should the others prove not feasible
>
>
> -D...
>
>
> On Mon, Jan 16, 2017 at 12:15 PM, Justin Leet <justinjleet@gmail.com>
> wrote:
>
> > I definitely agree on checking out the MaxMind API.  I'll take a look at
> > it, but at first glance it looks like it does include everything we use.
> > Great find, JJ.
> >
> > More details on various people's points:
> >
> > As a note to anyone hopping in, Simon's point on the range lookup vs a
> key
> > lookup is why it becomes a Scan in HBase vs a Get.  As an addendum to
> what
> > Simon mentioned, denormalizing is easy enough and turns it into an easy
> > range lookup.
> >
> > To David's point, the MapDB does require a network hop, but it's once per
> > refresh of the data (Got a relevant callback? Grab new data, load it,
> swap
> > out) instead of (up to) once per message.  I would expect the same to be
> > true of the MaxMind db files.
> >
> > I'd also argue MapDB not really more complex than refreshing the HBase
> > table, because we potentially have to start worrying about things like
> > hashing and/or indices and even just general data represtation. It's
> > definitely correct that the file processing has to occur on either path,
> so
> > it really boils down to handling the callback and reloading the file vs
> > handling some of the standard HBasey things.  I don't think either is an
> > enormous amount of work (and both are almost certainly more work than
> > MaxMind's API)
> >
> > Regarding extensibility, I'd argue for parity with what we have first,
> then
> > build what we need from there.  Does anybody have any disagreement with
> > that approach for right now?
> >
> > Justin
> >
> > On Mon, Jan 16, 2017 at 12:04 PM, David Lyle <dlyle65535@gmail.com>
> wrote:
> >
> > > It is interesting- save us a ton of effort, and has the right license.
> I
> > > think it's worth at least checking out.
> > >
> > > -D...
> > >
> > >
> > > On Mon, Jan 16, 2017 at 12:00 PM, Simon Elliston Ball <
> > > simon@simonellistonball.com> wrote:
> > >
> > > > I like that approach even more. That way we would only have to worry
> > > about
> > > > distributing the database file in binary format to all the supervisor
> > > nodes
> > > > on update.
> > > >
> > > > It would also make it easier for people to switch to the enterprise
> DB
> > > > potentially if they had the license.
> > > >
> > > > One slight issue with this might be for people who wanted to extend
> the
> > > > database. For example, organisations may want to add geo-enrichment
> to
> > > > their own private network addresses based modified versions of the
> geo
> > > > database. Currently we don’t really allow this, since we hard-code
> > > ignoring
> > > > private network classes into the geo enrichment adapter, but I can
> see
> > a
> > > > case where a global org might want to add their own ranges and
> > locations
> > > to
> > > > the data set. Does that make sense to anyone else?
> > > >
> > > > Simon
> > > >
> > > >
> > > > > On 16 Jan 2017, at 16:50, JJ Meyer <jjmeyer0@gmail.com> wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > Can we leverage maxmind's Java client (
> > > > > https://github.com/maxmind/GeoIP2-java/tree/master/src/
> > > > main/java/com/maxmind/geoip2)
> > > > > in this case? I believe it can directly read maxmind file. Plus I
> > think
> > > > it
> > > > > also has some support for caching as well.
> > > > >
> > > > > Thanks,
> > > > > JJ
> > > > >
> > > > > On Mon, Jan 16, 2017 at 10:32 AM, Simon Elliston Ball <
> > > > > simon@simonellistonball.com> wrote:
> > > > >
> > > > >> I like the idea of MapDB, since we can essentially pull an
> instance
> > > into
> > > > >> each supervisor, so it makes a lot of sense for relatively small
> > > scale,
> > > > >> relatively static enrichments in general.
> > > > >>
> > > > >> Generally this feels like a caching problem, and would be for
a
> > simple
> > > > >> key-value lookup. In that case I would agree with David Lyle
on
> > using
> > > > HBase
> > > > >> as a source or truth and relying on caching.
> > > > >>
> > > > >> That said, GeoIP is a different lookup pattern, since it’s
a range
> > > > lookup
> > > > >> then a key lookup (or if we denormalize the MaxMind data, just
a
> > range
> > > > >> lookup) for that kind of thing MapDB with something like the
BTree
> > > > seems a
> > > > >> good fit.
> > > > >>
> > > > >> Simon
> > > > >>
> > > > >>
> > > > >>> On 16 Jan 2017, at 16:28, David Lyle <dlyle65535@gmail.com>
> wrote:
> > > > >>>
> > > > >>> I'm +1 on removing the MySQL dependency, BUT - I'd prefer
to see
> it
> > > as
> > > > an
> > > > >>> HBase enrichment. If our current caching isn't enough to
mitigate
> > the
> > > > >> above
> > > > >>> issues, we have a problem, don't we? Or do we not recommend
HBase
> > > > >>> enrichment for per message enrichment in general?
> > > > >>>
> > > > >>> Also- can you elaborate on how MapDB would not require a
network
> > hop?
> > > > >>> Doesn't this mean we would have to sync the enrichment data
to
> each
> > > > Storm
> > > > >>> supervisor? HDFS could (probably would) have a network hop
too,
> no?
> > > > >>>
> > > > >>> Fwiw -
> > > > >>> "In its place, I've looked at using MapDB, which is a really
easy
> > to
> > > > use
> > > > >>> library for creating Java collections backed by a file (This
is
> > NOT a
> > > > >>> separate installation of anything, it's just a jar that manages
> > > > >> interaction
> > > > >>> with the file system).  Given the slow churn of the GeoIP
files
> (I
> > > > >> believe
> > > > >>> they get updated once a week), we can have a script that
can be
> run
> > > > when
> > > > >>> needed, downloads the MaxMind tar file, builds the MapDB
file
> that
> > > will
> > > > >> be
> > > > >>> used by the bolts, and places it into HDFS.  Finally, we
update a
> > > > config
> > > > >> to
> > > > >>> point to the new file, the bolts get the updated config callback
> > and
> > > > can
> > > > >>> update their db files.  Inside the code, we wrap the MapDB
> portions
> > > to
> > > > >> make
> > > > >>> it transparent to downstream code."
> > > > >>>
> > > > >>> Seems a bit more complex than "refresh the hbase table".
Afaik,
> > > either
> > > > >>> approach would require some sort of translation between GeoIP
> > source
> > > > >> format
> > > > >>> and target format, so that part is a wash imo.
> > > > >>>
> > > > >>> So, I'd really like to see, at least, an attempt to leverage
> HBase
> > > > >>> enrichment.
> > > > >>>
> > > > >>> -D...
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Jan 16, 2017 at 11:02 AM, Casey Stella <
> cestella@gmail.com
> > >
> > > > >> wrote:
> > > > >>>
> > > > >>>> I think that it's a sensible thing to use MapDB for the
geo
> > > > enrichment.
> > > > >>>> Let me state my reasoning:
> > > > >>>>
> > > > >>>>  - An HBase implementation  would necessitate a HBase
scan
> > possibly
> > > > >>>>  hitting HDFS, which is expensive per-message.
> > > > >>>>  - An HBase implementation would necessitate a network
hop and
> > MapDB
> > > > >>>>  would not.
> > > > >>>>
> > > > >>>> I also think this might be the beginning of a more general
> purpose
> > > > >> support
> > > > >>>> in Stellar for locally shipped, read-only MapDB lookups,
which
> > might
> > > > be
> > > > >>>> interesting.
> > > > >>>>
> > > > >>>> In short, all quotes about premature optimization are
sure to
> > apply
> > > to
> > > > >> my
> > > > >>>> reasoning, but I can't help but have my spidey senses
tingle
> when
> > we
> > > > >>>> introduce a scan-per-message architecture.
> > > > >>>>
> > > > >>>> Casey
> > > > >>>>
> > > > >>>> On Mon, Jan 16, 2017 at 10:53 AM, Dima Kovalyov <
> > > > >> Dima.Kovalyov@sstech.us>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Hello Justin,
> > > > >>>>>
> > > > >>>>> Considering that Metron uses hbase tables for storing
> enrichment
> > > and
> > > > >>>>> threatintel feeds, can we use Hbase for geo enrichment
as well?
> > > > >>>>> Or MapDB can be used for enrichment and threatintel
feeds
> instead
> > > of
> > > > >>>> hbase?
> > > > >>>>>
> > > > >>>>> - Dima
> > > > >>>>>
> > > > >>>>> On 01/16/2017 04:17 PM, Justin Leet wrote:
> > > > >>>>>> Hi all,
> > > > >>>>>>
> > > > >>>>>> As a bit of background, right now, GeoIP data
is loaded into
> and
> > > > >>>> managed
> > > > >>>>> by
> > > > >>>>>> MySQL (the connectors are LGPL licensed and we
need to sever
> our
> > > > Maven
> > > > >>>>>> dependency on it before next release). We currently
depend on
> > and
> > > > >>>> install
> > > > >>>>>> an instance of MySQL (in each of the Management
Pack, Ansible,
> > and
> > > > >>>> Docker
> > > > >>>>>> installs). In the topology, we use the JDBCAdapter
to connect
> to
> > > > MySQL
> > > > >>>>> and
> > > > >>>>>> query for a given IP.  Additionally, it's a single
point of
> > > failure
> > > > >> for
> > > > >>>>>> that particular enrichment right now.  If MySQL
is down, geo
> > > > >> enrichment
> > > > >>>>>> can't occur.
> > > > >>>>>>
> > > > >>>>>> I'm proposing that we eliminate the use of MySQL
entirely,
> > through
> > > > all
> > > > >>>>>> installation paths (which, unless I missed some,
includes
> > Ansible,
> > > > the
> > > > >>>>>> Ambari Management Pack, and Docker).  We'd do
this by dropping
> > all
> > > > the
> > > > >>>>>> various MySQL setup and management through the
code, along
> with
> > > all
> > > > >> the
> > > > >>>>>> DDL, etc.  The JDBCAdapter would stay, so that
anybody who
> wants
> > > to
> > > > >>>> setup
> > > > >>>>>> their own databases for enrichments and install
connectors is
> > able
> > > > to
> > > > >>>> do
> > > > >>>>> so.
> > > > >>>>>>
> > > > >>>>>> In its place, I've looked at using MapDB, which
is a really
> easy
> > > to
> > > > >> use
> > > > >>>>>> library for creating Java collections backed
by a file (This
> is
> > > NOT
> > > > a
> > > > >>>>>> separate installation of anything, it's just
a jar that
> manages
> > > > >>>>> interaction
> > > > >>>>>> with the file system).  Given the slow churn
of the GeoIP
> files
> > (I
> > > > >>>>> believe
> > > > >>>>>> they get updated once a week), we can have a
script that can
> be
> > > run
> > > > >>>> when
> > > > >>>>>> needed, downloads the MaxMind tar file, builds
the MapDB file
> > that
> > > > >> will
> > > > >>>>> be
> > > > >>>>>> used by the bolts, and places it into HDFS. 
Finally, we
> update
> > a
> > > > >>>> config
> > > > >>>>> to
> > > > >>>>>> point to the new file, the bolts get the updated
config
> callback
> > > and
> > > > >>>> can
> > > > >>>>>> update their db files.  Inside the code, we wrap
the MapDB
> > > portions
> > > > to
> > > > >>>>> make
> > > > >>>>>> it transparent to downstream code.
> > > > >>>>>>
> > > > >>>>>> The particularly nice parts about using MapDB
are that its
> ease
> > of
> > > > use
> > > > >>>>> plus
> > > > >>>>>> it offers the utilities we need out of the box
to be able to
> > > support
> > > > >>>> the
> > > > >>>>>> operations we need on this (Keep in mind the
GeoIP files use
> IP
> > > > ranges
> > > > >>>>> and
> > > > >>>>>> we need to be able to easily grab the appropriate
range).
> > > > >>>>>>
> > > > >>>>>> The main point of concern I have about this is
that when we
> grab
> > > the
> > > > >>>> HDFS
> > > > >>>>>> file during an update, given that multiple JVMs
can be
> running,
> > we
> > > > >>>> don't
> > > > >>>>>> want them to clobber each other. I believe this
can be avoided
> > by
> > > > >>>> simply
> > > > >>>>>> using each worker's working directory to store
the file (and
> > > > >>>>> appropriately
> > > > >>>>>> ensure threads on the same JVM manage multithreading).
 This
> > > should
> > > > >>>> keep
> > > > >>>>>> the JVMs (and the underlying DB files) entirely
independent.
> > > > >>>>>>
> > > > >>>>>> This script would get called by the various installations
> during
> > > > >>>> startup
> > > > >>>>> to
> > > > >>>>>> do the initial setup.  After install, it can
then be called on
> > > > demand
> > > > >>>> in
> > > > >>>>>> order.
> > > > >>>>>>
> > > > >>>>>> At this point, we should be all set, with everything
running
> and
> > > > >>>>> updatable.
> > > > >>>>>>
> > > > >>>>>> Justin
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message