metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: [DISCUSS] Moving GeoIP management away from MySQL
Date Mon, 16 Jan 2017 21:11:18 GMT
+1 to using the Java API with the MMDB file provided by Maxmind.  This is
what I had thought we were doing when we discussed this a few months back.
I'd rather use the Maxmind tools as-provided instead of engineering
something on top of it.

On Mon, Jan 16, 2017 at 3:59 PM, JJ Meyer <jjmeyer0@gmail.com> wrote:

> Matt, I agree with your points on why we shouldn't just get rid of the
> database just to get rid of a database. But IMO, I think we may be
> reinventing the wheel a little bit by even putting the maxmind data into
> MySQL. Right now we are already downloading a maxmind file. To me it seems
> simpler to push the file to HDFS where we can pick it up and have the
> maxmind client use that instead of importing data into a DB and then
> running a query. Also, I believe the data gets updated weekly. So syncing
> may become easier too.
>
> James, I believe it works with the paid and free versions of geoip. I know
> NiFi uses this client library in their Geo enrichment processor.
>
> Also, if it is decided that using a SQL database is still the best
> solution, I think there is a benefit to using their library. We would just
> have to implement a `DatabaseProvider` that hits some SQL db instead of
> using their standard implementation.
>
> Thanks,
> JJ
>
> On Mon, Jan 16, 2017 at 2:27 PM, James Sirota <jsirota@apache.org> wrote:
>
> > Hi Guys, I just wanted to clarify one point that I think is lost in this
> > tread.  Geo enrichment is NOT a key-value enrichment.  It requires a
> range
> > scan and a join (which is why it's implemented via mySql and not Hbase).
> > To account for this access pattern via a key-value store you would
> > inevitably have to do something funky or in case of Hbase I don't think
> > there is a way to avoid doing a range scan.
> >
> > With respect to mapdb it only has support for Maps, Sets, Lists, Queues.
> > Are we sure it provides enough functionality for us to do this
> enrichment?
> >
> > With respect to the Maxmind client, are we sure we can use it on the
> > mySql-backed version of their DB?  I thought the Maxmind database itself
> is
> > proprietary and is something you have to pay for.  My understanding is
> that
> > the client is designed for that proprietary version.
> >
> > I somewhat agree with Matt's point.  If mySql is a problem because of
> > licensing, the path of least resistance to remove mySql dependencies
> would
> > be to simply switch to postgresql.  We will always have conventional sql
> > databases in our stack because other big data tools use them. Why not
> take
> > advantage of them too?
> >
> > Thanks,
> > James
> >
> > 16.01.2017, 12:27, "Matt Foley" <mattf@apache.org>:
> > > Hi Justin, and team,
> > > Several components of the Hadoop Stack utilize a SQL database, usually
> > for metadata of some sort. Ambari knows this and arranges for them to
> share
> > a single database installation (on or off the cluster), unless they
> > explicitly configure use of different databases (which is allowed for
> sites
> > that desire it). Ambari defaults to using PostgreSQL, altho it’s happy to
> > use MySQL, Oracle, or Microsoft, along with whatever each component
> > historically defined as their default (such as Derby).
> > >
> > > If we want to start with a replacement of current functionality, I
> would
> > suggest switching the default database to PostgreSQL. Replacing fast,
> > efficient, and proven db services with a file-based api library (but no
> > standard way to propagate the underlying storage files) seems to me to be
> > taking a step backwards.
> > >
> > > Sticking with a SQL-based service will surely minimize the amount of
> > code changes needed. And making the SQL either dialect-independent or
> > capable of switching among dialects, then enables us to do what the rest
> of
> > the Hadoop stack does: allow enterprise customers to substitute Oracle or
> > Microsoft enterprise-class databases where they wish. Regarding the
> > drivers, we should study what the other Stack components do; I’m not an
> > expert in those areas.
> > >
> > > Using the same db as the rest of the stack also means administrators
> can
> > be confident they’ve set up adequate backup and recovery processes.
> > > All these are valuable reasons not to roll our own storage system for
> > this enrichment data. IMO, of course.
> > >
> > > Cheers,
> > > --Matt
> > >
> > > On 1/16/17, 9:52 AM, "Kyle Richardson" <kylerichardson2@gmail.com>
> > wrote:
> > >
> > >     +1 Agree with David's order
> > >
> > >     -Kyle
> > >
> > >     On Mon, Jan 16, 2017 at 12:41 PM, David Lyle <dlyle65535@gmail.com
> >
> > wrote:
> > >
> > >     > Def agree on the parity point.
> > >     >
> > >     > I'm a little worried about Supervisor relocations for non-HBase
> > solutions,
> > >     > but having much of the work done for us by MaxMind changes my
> > preference to
> > >     > (in order)
> > >     >
> > >     > 1) MM API
> > >     > 2) HBase Enrichment
> > >     > 3) MapDB should the others prove not feasible
> > >     >
> > >     >
> > >     > -D...
> > >     >
> > >     >
> > >     > On Mon, Jan 16, 2017 at 12:15 PM, Justin Leet <
> > justinjleet@gmail.com>
> > >     > wrote:
> > >     >
> > >     > > I definitely agree on checking out the MaxMind API. I'll take
a
> > look at
> > >     > > it, but at first glance it looks like it does include
> everything
> > we use.
> > >     > > Great find, JJ.
> > >     > >
> > >     > > More details on various people's points:
> > >     > >
> > >     > > As a note to anyone hopping in, Simon's point on the range
> > lookup vs a
> > >     > key
> > >     > > lookup is why it becomes a Scan in HBase vs a Get. As an
> > addendum to
> > >     > what
> > >     > > Simon mentioned, denormalizing is easy enough and turns it into
> > an easy
> > >     > > range lookup.
> > >     > >
> > >     > > To David's point, the MapDB does require a network hop, but
> it's
> > once per
> > >     > > refresh of the data (Got a relevant callback? Grab new data,
> > load it,
> > >     > swap
> > >     > > out) instead of (up to) once per message. I would expect the
> > same to be
> > >     > > true of the MaxMind db files.
> > >     > >
> > >     > > I'd also argue MapDB not really more complex than refreshing
> the
> > HBase
> > >     > > table, because we potentially have to start worrying about
> > things like
> > >     > > hashing and/or indices and even just general data represtation.
> > It's
> > >     > > definitely correct that the file processing has to occur on
> > either path,
> > >     > so
> > >     > > it really boils down to handling the callback and reloading the
> > file vs
> > >     > > handling some of the standard HBasey things. I don't think
> > either is an
> > >     > > enormous amount of work (and both are almost certainly more
> work
> > than
> > >     > > MaxMind's API)
> > >     > >
> > >     > > Regarding extensibility, I'd argue for parity with what we have
> > first,
> > >     > then
> > >     > > build what we need from there. Does anybody have any
> > disagreement with
> > >     > > that approach for right now?
> > >     > >
> > >     > > Justin
> > >     > >
> > >     > > On Mon, Jan 16, 2017 at 12:04 PM, David Lyle <
> > dlyle65535@gmail.com>
> > >     > wrote:
> > >     > >
> > >     > > > It is interesting- save us a ton of effort, and has the
right
> > license.
> > >     > I
> > >     > > > think it's worth at least checking out.
> > >     > > >
> > >     > > > -D...
> > >     > > >
> > >     > > >
> > >     > > > On Mon, Jan 16, 2017 at 12:00 PM, Simon Elliston Ball <
> > >     > > > simon@simonellistonball.com> wrote:
> > >     > > >
> > >     > > > > I like that approach even more. That way we would only
have
> > to worry
> > >     > > > about
> > >     > > > > distributing the database file in binary format to
all the
> > supervisor
> > >     > > > nodes
> > >     > > > > on update.
> > >     > > > >
> > >     > > > > It would also make it easier for people to switch to
the
> > enterprise
> > >     > DB
> > >     > > > > potentially if they had the license.
> > >     > > > >
> > >     > > > > One slight issue with this might be for people who
wanted
> to
> > extend
> > >     > the
> > >     > > > > database. For example, organisations may want to add
> > geo-enrichment
> > >     > to
> > >     > > > > their own private network addresses based modified
versions
> > of the
> > >     > geo
> > >     > > > > database. Currently we don’t really allow this, since
we
> > hard-code
> > >     > > > ignoring
> > >     > > > > private network classes into the geo enrichment adapter,
> but
> > I can
> > >     > see
> > >     > > a
> > >     > > > > case where a global org might want to add their own
ranges
> > and
> > >     > > locations
> > >     > > > to
> > >     > > > > the data set. Does that make sense to anyone else?
> > >     > > > >
> > >     > > > > Simon
> > >     > > > >
> > >     > > > >
> > >     > > > > > On 16 Jan 2017, at 16:50, JJ Meyer <jjmeyer0@gmail.com>
> > wrote:
> > >     > > > > >
> > >     > > > > > Hello all,
> > >     > > > > >
> > >     > > > > > Can we leverage maxmind's Java client (
> > >     > > > > > https://github.com/maxmind/GeoIP2-java/tree/master/src/
> > >     > > > > main/java/com/maxmind/geoip2)
> > >     > > > > > in this case? I believe it can directly read maxmind
> file.
> > Plus I
> > >     > > think
> > >     > > > > it
> > >     > > > > > also has some support for caching as well.
> > >     > > > > >
> > >     > > > > > Thanks,
> > >     > > > > > JJ
> > >     > > > > >
> > >     > > > > > On Mon, Jan 16, 2017 at 10:32 AM, Simon Elliston
Ball <
> > >     > > > > > simon@simonellistonball.com> wrote:
> > >     > > > > >
> > >     > > > > >> I like the idea of MapDB, since we can essentially
pull
> an
> > >     > instance
> > >     > > > into
> > >     > > > > >> each supervisor, so it makes a lot of sense
for
> > relatively small
> > >     > > > scale,
> > >     > > > > >> relatively static enrichments in general.
> > >     > > > > >>
> > >     > > > > >> Generally this feels like a caching problem,
and would
> be
> > for a
> > >     > > simple
> > >     > > > > >> key-value lookup. In that case I would agree
with David
> > Lyle on
> > >     > > using
> > >     > > > > HBase
> > >     > > > > >> as a source or truth and relying on caching.
> > >     > > > > >>
> > >     > > > > >> That said, GeoIP is a different lookup pattern,
since
> > it’s a range
> > >     > > > > lookup
> > >     > > > > >> then a key lookup (or if we denormalize the
MaxMind
> data,
> > just a
> > >     > > range
> > >     > > > > >> lookup) for that kind of thing MapDB with
something like
> > the BTree
> > >     > > > > seems a
> > >     > > > > >> good fit.
> > >     > > > > >>
> > >     > > > > >> Simon
> > >     > > > > >>
> > >     > > > > >>
> > >     > > > > >>> On 16 Jan 2017, at 16:28, David Lyle <
> > dlyle65535@gmail.com>
> > >     > wrote:
> > >     > > > > >>>
> > >     > > > > >>> I'm +1 on removing the MySQL dependency,
BUT - I'd
> > prefer to see
> > >     > it
> > >     > > > as
> > >     > > > > an
> > >     > > > > >>> HBase enrichment. If our current caching
isn't enough
> to
> > mitigate
> > >     > > the
> > >     > > > > >> above
> > >     > > > > >>> issues, we have a problem, don't we? Or
do we not
> > recommend HBase
> > >     > > > > >>> enrichment for per message enrichment
in general?
> > >     > > > > >>>
> > >     > > > > >>> Also- can you elaborate on how MapDB would
not require
> a
> > network
> > >     > > hop?
> > >     > > > > >>> Doesn't this mean we would have to sync
the enrichment
> > data to
> > >     > each
> > >     > > > > Storm
> > >     > > > > >>> supervisor? HDFS could (probably would)
have a network
> > hop too,
> > >     > no?
> > >     > > > > >>>
> > >     > > > > >>> Fwiw -
> > >     > > > > >>> "In its place, I've looked at using MapDB,
which is a
> > really easy
> > >     > > to
> > >     > > > > use
> > >     > > > > >>> library for creating Java collections
backed by a file
> > (This is
> > >     > > NOT a
> > >     > > > > >>> separate installation of anything, it's
just a jar that
> > manages
> > >     > > > > >> interaction
> > >     > > > > >>> with the file system). Given the slow
churn of the
> GeoIP
> > files
> > >     > (I
> > >     > > > > >> believe
> > >     > > > > >>> they get updated once a week), we can
have a script
> that
> > can be
> > >     > run
> > >     > > > > when
> > >     > > > > >>> needed, downloads the MaxMind tar file,
builds the
> MapDB
> > file
> > >     > that
> > >     > > > will
> > >     > > > > >> be
> > >     > > > > >>> used by the bolts, and places it into
HDFS. Finally, we
> > update a
> > >     > > > > config
> > >     > > > > >> to
> > >     > > > > >>> point to the new file, the bolts get the
updated config
> > callback
> > >     > > and
> > >     > > > > can
> > >     > > > > >>> update their db files. Inside the code,
we wrap the
> MapDB
> > >     > portions
> > >     > > > to
> > >     > > > > >> make
> > >     > > > > >>> it transparent to downstream code."
> > >     > > > > >>>
> > >     > > > > >>> Seems a bit more complex than "refresh
the hbase
> table".
> > Afaik,
> > >     > > > either
> > >     > > > > >>> approach would require some sort of translation
between
> > GeoIP
> > >     > > source
> > >     > > > > >> format
> > >     > > > > >>> and target format, so that part is a wash
imo.
> > >     > > > > >>>
> > >     > > > > >>> So, I'd really like to see, at least,
an attempt to
> > leverage
> > >     > HBase
> > >     > > > > >>> enrichment.
> > >     > > > > >>>
> > >     > > > > >>> -D...
> > >     > > > > >>>
> > >     > > > > >>>
> > >     > > > > >>> On Mon, Jan 16, 2017 at 11:02 AM, Casey
Stella <
> > >     > cestella@gmail.com
> > >     > > >
> > >     > > > > >> wrote:
> > >     > > > > >>>
> > >     > > > > >>>> I think that it's a sensible thing
to use MapDB for
> the
> > geo
> > >     > > > > enrichment.
> > >     > > > > >>>> Let me state my reasoning:
> > >     > > > > >>>>
> > >     > > > > >>>> - An HBase implementation would necessitate
a HBase
> scan
> > >     > > possibly
> > >     > > > > >>>> hitting HDFS, which is expensive per-message.
> > >     > > > > >>>> - An HBase implementation would necessitate
a network
> > hop and
> > >     > > MapDB
> > >     > > > > >>>> would not.
> > >     > > > > >>>>
> > >     > > > > >>>> I also think this might be the beginning
of a more
> > general
> > >     > purpose
> > >     > > > > >> support
> > >     > > > > >>>> in Stellar for locally shipped, read-only
MapDB
> > lookups, which
> > >     > > might
> > >     > > > > be
> > >     > > > > >>>> interesting.
> > >     > > > > >>>>
> > >     > > > > >>>> In short, all quotes about premature
optimization are
> > sure to
> > >     > > apply
> > >     > > > to
> > >     > > > > >> my
> > >     > > > > >>>> reasoning, but I can't help but have
my spidey senses
> > tingle
> > >     > when
> > >     > > we
> > >     > > > > >>>> introduce a scan-per-message architecture.
> > >     > > > > >>>>
> > >     > > > > >>>> Casey
> > >     > > > > >>>>
> > >     > > > > >>>> On Mon, Jan 16, 2017 at 10:53 AM,
Dima Kovalyov <
> > >     > > > > >> Dima.Kovalyov@sstech.us>
> > >     > > > > >>>> wrote:
> > >     > > > > >>>>
> > >     > > > > >>>>> Hello Justin,
> > >     > > > > >>>>>
> > >     > > > > >>>>> Considering that Metron uses hbase
tables for storing
> > >     > enrichment
> > >     > > > and
> > >     > > > > >>>>> threatintel feeds, can we use
Hbase for geo
> enrichment
> > as well?
> > >     > > > > >>>>> Or MapDB can be used for enrichment
and threatintel
> > feeds
> > >     > instead
> > >     > > > of
> > >     > > > > >>>> hbase?
> > >     > > > > >>>>>
> > >     > > > > >>>>> - Dima
> > >     > > > > >>>>>
> > >     > > > > >>>>> On 01/16/2017 04:17 PM, Justin
Leet wrote:
> > >     > > > > >>>>>> Hi all,
> > >     > > > > >>>>>>
> > >     > > > > >>>>>> As a bit of background, right
now, GeoIP data is
> > loaded into
> > >     > and
> > >     > > > > >>>> managed
> > >     > > > > >>>>> by
> > >     > > > > >>>>>> MySQL (the connectors are
LGPL licensed and we need
> > to sever
> > >     > our
> > >     > > > > Maven
> > >     > > > > >>>>>> dependency on it before next
release). We currently
> > depend on
> > >     > > and
> > >     > > > > >>>> install
> > >     > > > > >>>>>> an instance of MySQL (in each
of the Management
> Pack,
> > Ansible,
> > >     > > and
> > >     > > > > >>>> Docker
> > >     > > > > >>>>>> installs). In the topology,
we use the JDBCAdapter
> to
> > connect
> > >     > to
> > >     > > > > MySQL
> > >     > > > > >>>>> and
> > >     > > > > >>>>>> query for a given IP. Additionally,
it's a single
> > point of
> > >     > > > failure
> > >     > > > > >> for
> > >     > > > > >>>>>> that particular enrichment
right now. If MySQL is
> > down, geo
> > >     > > > > >> enrichment
> > >     > > > > >>>>>> can't occur.
> > >     > > > > >>>>>>
> > >     > > > > >>>>>> I'm proposing that we eliminate
the use of MySQL
> > entirely,
> > >     > > through
> > >     > > > > all
> > >     > > > > >>>>>> installation paths (which,
unless I missed some,
> > includes
> > >     > > Ansible,
> > >     > > > > the
> > >     > > > > >>>>>> Ambari Management Pack, and
Docker). We'd do this by
> > dropping
> > >     > > all
> > >     > > > > the
> > >     > > > > >>>>>> various MySQL setup and management
through the code,
> > along
> > >     > with
> > >     > > > all
> > >     > > > > >> the
> > >     > > > > >>>>>> DDL, etc. The JDBCAdapter
would stay, so that
> anybody
> > who
> > >     > wants
> > >     > > > to
> > >     > > > > >>>> setup
> > >     > > > > >>>>>> their own databases for enrichments
and install
> > connectors is
> > >     > > able
> > >     > > > > to
> > >     > > > > >>>> do
> > >     > > > > >>>>> so.
> > >     > > > > >>>>>>
> > >     > > > > >>>>>> In its place, I've looked
at using MapDB, which is a
> > really
> > >     > easy
> > >     > > > to
> > >     > > > > >> use
> > >     > > > > >>>>>> library for creating Java
collections backed by a
> > file (This
> > >     > is
> > >     > > > NOT
> > >     > > > > a
> > >     > > > > >>>>>> separate installation of anything,
it's just a jar
> > that
> > >     > manages
> > >     > > > > >>>>> interaction
> > >     > > > > >>>>>> with the file system). Given
the slow churn of the
> > GeoIP
> > >     > files
> > >     > > (I
> > >     > > > > >>>>> believe
> > >     > > > > >>>>>> they get updated once a week),
we can have a script
> > that can
> > >     > be
> > >     > > > run
> > >     > > > > >>>> when
> > >     > > > > >>>>>> needed, downloads the MaxMind
tar file, builds the
> > MapDB file
> > >     > > that
> > >     > > > > >> will
> > >     > > > > >>>>> be
> > >     > > > > >>>>>> used by the bolts, and places
it into HDFS. Finally,
> > we
> > >     > update
> > >     > > a
> > >     > > > > >>>> config
> > >     > > > > >>>>> to
> > >     > > > > >>>>>> point to the new file, the
bolts get the updated
> > config
> > >     > callback
> > >     > > > and
> > >     > > > > >>>> can
> > >     > > > > >>>>>> update their db files. Inside
the code, we wrap the
> > MapDB
> > >     > > > portions
> > >     > > > > to
> > >     > > > > >>>>> make
> > >     > > > > >>>>>> it transparent to downstream
code.
> > >     > > > > >>>>>>
> > >     > > > > >>>>>> The particularly nice parts
about using MapDB are
> > that its
> > >     > ease
> > >     > > of
> > >     > > > > use
> > >     > > > > >>>>> plus
> > >     > > > > >>>>>> it offers the utilities we
need out of the box to be
> > able to
> > >     > > > support
> > >     > > > > >>>> the
> > >     > > > > >>>>>> operations we need on this
(Keep in mind the GeoIP
> > files use
> > >     > IP
> > >     > > > > ranges
> > >     > > > > >>>>> and
> > >     > > > > >>>>>> we need to be able to easily
grab the appropriate
> > range).
> > >     > > > > >>>>>>
> > >     > > > > >>>>>> The main point of concern
I have about this is that
> > when we
> > >     > grab
> > >     > > > the
> > >     > > > > >>>> HDFS
> > >     > > > > >>>>>> file during an update, given
that multiple JVMs can
> be
> > >     > running,
> > >     > > we
> > >     > > > > >>>> don't
> > >     > > > > >>>>>> want them to clobber each
other. I believe this can
> > be avoided
> > >     > > by
> > >     > > > > >>>> simply
> > >     > > > > >>>>>> using each worker's working
directory to store the
> > file (and
> > >     > > > > >>>>> appropriately
> > >     > > > > >>>>>> ensure threads on the same
JVM manage
> > multithreading). This
> > >     > > > should
> > >     > > > > >>>> keep
> > >     > > > > >>>>>> the JVMs (and the underlying
DB files) entirely
> > independent.
> > >     > > > > >>>>>>
> > >     > > > > >>>>>> This script would get called
by the various
> > installations
> > >     > during
> > >     > > > > >>>> startup
> > >     > > > > >>>>> to
> > >     > > > > >>>>>> do the initial setup. After
install, it can then be
> > called on
> > >     > > > > demand
> > >     > > > > >>>> in
> > >     > > > > >>>>>> order.
> > >     > > > > >>>>>>
> > >     > > > > >>>>>> At this point, we should be
all set, with everything
> > running
> > >     > and
> > >     > > > > >>>>> updatable.
> > >     > > > > >>>>>>
> > >     > > > > >>>>>> Justin
> > >     > > > > >>>>>>
> > >     > > > > >>>>>
> > >     > > > > >>>>>
> > >     > > > > >>>>
> > >     > > > > >>
> > >     > > > > >>
> > >     > > > >
> > >     > > > >
> > >     > > >
> > >     > >
> > >     >
> >
> > -------------------
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
>



-- 
Nick Allen <nick@nickallen.org>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message