metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: [DISCUSS] Pcap panel architecture
Date Tue, 08 May 2018 21:17:47 GMT
@Ryan - pulled your branch and experimented with a few things. In doing so,
it dawned on me that by adding the yarn and hadoop classpath, you probably
didn't introduce a new classpath issue, rather you probably just moved onto
the next classpath issue, ie hbase per your exception about hbase jaxb.
Anyhow, I put up a branch with some pom changes worth trying in conjunction
with invoking the rest app startup via "/usr/bin/yarn jar"

https://github.com/mmiklavc/metron/tree/ryan-rest-test

https://github.com/mmiklavc/metron/commit/5ca23580fc6e043fafae2327c80b65b20ca1c0c9

Mike


On Tue, May 8, 2018 at 7:44 AM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:

> That would be a step closer to something more like a micro-service
> architecture. However, I would want to make sure we think about the
> operational complexity, and mpack implications of having another server
> installed and running somewhere on the cluster (also, ssl, kerberos, etc
> etc requirements for that service).
>
> On 8 May 2018 at 14:27, Ryan Merriman <merrimanr@gmail.com> wrote:
>
> > +1 to having metron-api as it's own service and using a gateway type
> > pattern.
> >
> > On Tue, May 8, 2018 at 8:13 AM, Otto Fowler <ottobackwards@gmail.com>
> > wrote:
> >
> > > Why not have metron-api as it’s own service and use a ‘gateway’ type
> > > pattern in rest?
> > >
> > >
> > > On May 8, 2018 at 08:45:33, Ryan Merriman (merrimanr@gmail.com) wrote:
> > >
> > > Moving the yarn classpath command earlier in the classpath now gives
> this
> > > error:
> > >
> > > Caused by: java.lang.NoSuchMethodError:
> > > javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;
> > >
> > > I will experiment with other combinations, I suspect we will need
> > > finer-grain control over the order.
> > >
> > > The grep matches class names inside jar files. I use this all the time
> > and
> > > it's really useful.
> > >
> > > The metron-rest jar is already shaded.
> > >
> > > Reverse engineering the yarn jar command was the next thing I was going
> > to
> > > try. Will let you know how it goes.
> > >
> > > On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
> > > michael.miklavcic@gmail.com> wrote:
> > >
> > > > What order did you add the hadoop or yarn classpath? The "shaded"
> > > package
> > > > stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> > > > .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try
> adding
> > > > those packages earlier on the classpath.
> > > >
> > > > I think that find command needs a "jar tvf", otherwise you're looking
> > > for a
> > > > class name in jar file names.
> > > >
> > > > Have you tried shading the rest jar?
> > > >
> > > > I'd also look at the classpath you get when running "yarn jar" to
> start
> > > the
> > > > existing pcap service, per the instructions in metron-api/README.md.
> > > >
> > > >
> > > > On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman <merrimanr@gmail.com>
> > > wrote:
> > > >
> > > > > To explore the idea of merging metron-api into metron-rest and
> > running
> > > > pcap
> > > > > queries inside our REST application, I created a simple test here:
> > > > > https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test.
> A
> > > > > summary of what's included:
> > > > >
> > > > > - Added pcap as a dependency in the metron-rest pom.xml
> > > > > - Added a pcap query controller endpoint at
> > > > > http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> > > > queryUsingGET
> > > > > - Added a pcap query service that runs a simple, hardcoded query
> > > > >
> > > > > Generate some pcap data using pycapa (
> > > > > https://github.com/apache/metron/tree/master/metron-sensors/pycapa
> )
> > > and
> > > > > the
> > > > > pcap topology (
> > > > > https://github.com/apache/metron/tree/master/metron-
> > > > > platform/metron-pcap-backend#starting-the-topology).
> > > > > After this initial setup there should be data in HDFS at
> > > > > "/apps/metron/pcap". I believe this should be enough to exercise
> the
> > > > > issue. Just hit the endpoint referenced above. I tested this in an
> > > > > already running full dev by building and deploying the metron-rest
> > > jar.
> > > > I
> > > > > did not rebuild full dev with this change but I would still expect
> it
> > > to
> > > > > work. Let me know if it doesn't.
> > > > >
> > > > > The first error I see when I hit this endpoint is:
> > > > >
> > > > > java.lang.NoClassDefFoundError:
> > > > > org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.
> > > > >
> > > > > Here are the things I've tried so far:
> > > > >
> > > > > - Run the REST application with the YARN jar command since this is
> > how
> > > > > all our other YARN/MR-related applications are started (metron-api,
> > > > > MAAS,
> > > > > pcap query, etc). I wouldn't expect this to work since we have
> > > > runtime
> > > > > dependencies on our shaded elasticsearch and parser jars and I'm
> not
> > > > > aware
> > > > > of a way to add additional jars to the classpath with the YARN jar
> > > > > command
> > > > > (is there a way?). Either way I get this error:
> > > > >
> > > > > 18/05/04 19:49:56 WARN reflections.Reflections: could not create
> Dir
> > > > using
> > > > > jarFile from url file:/usr/hdp/2.6.4.0-91/hadoop/lib/ojdbc6.jar.
> > > > skipping.
> > > > > java.lang.NullPointerException
> > > > >
> > > > >
> > > > > - I tried adding `yarn classpath` and `hadoop classpath` to the
> > > > > classpath in /usr/metron/0.4.3/bin/metron-rest.sh (REST start
> > > > > script). I
> > > > > get this error:
> > > > >
> > > > > java.lang.ClassNotFoundException:
> > > > > org.apache.hadoop.hbase.shaded.org.codehaus.jackson.
> > > > > jaxrs.JacksonJaxbJsonProvider
> > > > >
> > > > >
> > > > > - I searched for the class in the previous attempt but could not
> find
> > > > it
> > > > > in full dev:
> > > > >
> > > > > find / -name "*.jar" 2>/dev/null | xargs grep
> > > > > org/apache/hadoop/hbase/shaded/org/codehaus/jackson/
> > > > > jaxrs/JacksonJaxbJsonProvider
> > > > > 2>/dev/null
> > > > >
> > > > >
> > > > > - Further up in the stack trace I see the error happens when
> > > > initiating
> > > > > the org.apache.hadoop.yarn.util.timeline.TimelineUtils class. I
> > > > tried
> > > > > setting "yarn.timeline-service.enabled" in Ambari to false and
> then
> > I
> > > > > get
> > > > > this error:
> > > > >
> > > > > Unable to parse
> > > > > '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework'
> > as
> > > a
> > > > > URI, check the setting for mapreduce.application.framework.path
> > > > >
> > > > >
> > > > > - I've tried adding different hadoop, hbase, yarn and mapreduce
> Maven
> > > > > dependencies without any success
> > > > > - hadoop-yarn-client
> > > > > - hadoop-yarn-common
> > > > > - hadoop-mapreduce-client-core
> > > > > - hadoop-yarn-server-common
> > > > > - hadoop-yarn-api
> > > > > - hbase-server
> > > > >
> > > > > I will keep exploring other possible solutions. Let me know if
> anyone
> > > > has
> > > > > any ideas.
> > > > >
> > > > > On Mon, May 7, 2018 at 9:02 AM, Otto Fowler <
> ottobackwards@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I can imagine a new generic service(s) capability whose job
( pun
> > > > > intended
> > > > > > ) is to
> > > > > > abstract the submittal, tracking, and storage of results to
yarn.
> > > > > >
> > > > > > It would be extended with storage providers, queue provider,
> > > possibly
> > > > > some
> > > > > > set of policies or rather strategies.
> > > > > >
> > > > > > The pcap ‘report’ would be a client to that service, the
> > specializes
> > > > the
> > > > > > service operation for the way we want pcap to work.
> > > > > >
> > > > > > We can then re-use the generic service for other long running
> yarn
> > > > > > things…..
> > > > > >
> > > > > >
> > > > > > On May 7, 2018 at 09:56:51, Otto Fowler (ottobackwards@gmail.com
> )
> > > > wrote:
> > > > > >
> > > > > > RE: Tracking v. users
> > > > > >
> > > > > > The submittal and tracking can associate the submitter with
the
> > yarn
> > > > job
> > > > > > and track that,
> > > > > > regardless of the yarn credentials.
> > > > > >
> > > > > > IE> if all submittals and monitoring are by the same yarn
user (
> > > > Metron )
> > > > > > from a single or
> > > > > > co-operative set of services, that service can maintain the
> > mapping.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On May 7, 2018 at 09:39:52, Ryan Merriman (merrimanr@gmail.com)
> > > wrote:
> > > > > >
> > > > > > Otto, your use case makes sense to me. We'll have to think about
> > how
> > > to
> > > > > > manage the user to job relationships. I'm assuming YARN jobs
will
> > be
> > > > > > submitted as the metron service user so YARN won't keep track
of
> > > this
> > > > for
> > > > > > us. Is that assumption correct? Do you have any ideas for doing
> > > that?
> > > > > >
> > > > > > Mike, I can start a feature branch and experiment with merging
> > > > metron-api
> > > > > > into metron-rest. That should allow us to collaborate on any
> issues
> > > or
> > > > > > challenges. Also, can you expand on your idea to manage external
> > > > > > dependencies as a special module? That seems like a very
> attractive
> > > > > option
> > > > > > to me.
> > > > > >
> > > > > > On Fri, May 4, 2018 at 8:39 AM, Otto Fowler <
> > ottobackwards@gmail.com>
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > From my response on the other thread, but applicable to
the
> > > backend
> > > > > > stuff:
> > > > > > >
> > > > > > > "The PCAP Query seems more like PCAP Report to me. You
are
> > > > generating a
> > > > > > > report based on parameters.
> > > > > > > That report is something that takes some time and external
> > process
> > > to
> > > > > > > generate… ie you have to wait for it.
> > > > > > >
> > > > > > > I can almost imagine a flow where you:
> > > > > > >
> > > > > > > * Are in the AlertUI
> > > > > > > * Ask to generate a PCAP report based on some selected
> > > > > alerts/meta-alert,
> > > > > > > possibly picking from on or more report ‘templates’
> > > > > > > that have query options etc
> > > > > > > * The report request is ‘queued’, that is dispatched
to be be
> > > > > > > executed/generated
> > > > > > > * You as a user have a ‘queue’ of your report results,
and when
> > > the
> > > > > > report
> > > > > > > is done it is queued there
> > > > > > > * We ‘monitor’ the report/queue press through the yarn
rest (
> > > report
> > > > > > > info/meta has the yarn details )
> > > > > > > * You can select the report from your queue and view it
either
> in
> > > a
> > > > new
> > > > > > UI
> > > > > > > or custom component
> > > > > > > * You can then apply a different ‘view’ to the report
or work
> > with
> > > > the
> > > > > > > report data
> > > > > > > * You can print / save etc
> > > > > > > * You can associate the report with the alerts ( again
in the
> > > report
> > > > > info
> > > > > > > ) with…. a ‘case’ or ‘ticket’ or investigation
something or
> other
> > > > > > >
> > > > > > >
> > > > > > > We can introduce extensibility into the report templates,
> report
> > > > views
> > > > > (
> > > > > > > thinks that work with the json data of the report )
> > > > > > >
> > > > > > > Something like that.”
> > > > > > >
> > > > > > > Maybe we can do :
> > > > > > >
> > > > > > > template -> query parameters -> script => yarn
info
> > > > > > > yarn info + query info + alert context + yarn status =>
report
> > > info
> > > > ->
> > > > > > > stored in a user’s ‘report queue’
> > > > > > > report persistence added to report info
> > > > > > > metron-rest -> api to monitor the queue, read results
( page ),
> > > etc
> > > > etc
> > > > > > >
> > > > > > >
> > > > > > > On May 4, 2018 at 09:23:39, Ryan Merriman (merrimanr@gmail.com
> )
> > > > wrote:
> > > > > > >
> > > > > > > I started a separate thread on Pcap UI considerations and
user
> > > > > > > requirements
> > > > > > > at Otto's request. This should help us keep these two related
> but
> > > > > > separate
> > > > > > > discussions focused.
> > > > > > >
> > > > > > > On Fri, May 4, 2018 at 7:19 AM, Michel Sumbul <
> > > > michelsumbul@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > (Youhouuu my first reply on this kind of mail chain^^)
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > If I may, I would like to share my view on the following
3
> > > points.
> > > > > > > >
> > > > > > > > - Backend:
> > > > > > > >
> > > > > > > > The current metron-api is totally seperate, it will
be logic
> > for
> > > me
> > > > > to
> > > > > > > have
> > > > > > > > it at the same place as the others rest api. Especially
when
> > > more
> > > > > > > security
> > > > > > > > will be added, it will not be needed to do the job
twice.
> > > > > > > > The current implementation send back a pcap object
which
> still
> > > need
> > > > > to
> > > > > > > be
> > > > > > > > decoded. In the opensoc, the decoding was done with
tshard on
> > > the
> > > > > > > frontend.
> > > > > > > > It will be good to have this decoding happening directly
on
> the
> > > > > backend
> > > > > > > to
> > > > > > > > not create a load on frontend. An option will be to
install
> > > tshark
> > > > on
> > > > > > > the
> > > > > > > > rest server and to use to convert the pcap to xml
and then
> to a
> > > > json
> > > > > > > that
> > > > > > > > will be send to the frontend.
> > > > > > > >
> > > > > > > > I tried to start directly the map/reduce job to search
over
> all
> > > the
> > > > > > pcap
> > > > > > > > data from the rest server and as Ryan mention it,
we had
> > > trouble. I
> > > > > > will
> > > > > > > > try to find back the error.
> > > > > > > >
> > > > > > > > Then in the POC, what we tried is to use the pcap_query
> script
> > > and
> > > > > this
> > > > > > > > work fine. I just modified it that he sends back directly
the
> > > > job_id
> > > > > of
> > > > > > > > yarn and not waiting that the job is finished. Then
it will
> > > allow
> > > > the
> > > > > > UI
> > > > > > > > and the rest server to know what the status of the
research
> by
> > > > > querying
> > > > > > > the
> > > > > > > > yarn rest api. This will allow the UI and the rest
server to
> be
> > > > async
> > > > > > > > without any blocking phase. What do you think about
that?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Having the job submitted directly from the code of
the rest
> > > server
> > > > > will
> > > > > > > be
> > > > > > > > perfect, but it will need a lot of investigation I
think (but
> > > I'm
> > > > not
> > > > > > > the
> > > > > > > > expert so I might be completely wrong ^^).
> > > > > > > >
> > > > > > > > We know that the pcap_query scritp work fine so why
not
> calling
> > > it?
> > > > > Is
> > > > > > > it
> > > > > > > > that bad? (maybe stupid question, but I really don’t
see a
> lot
> > > of
> > > > > > > drawback)
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > - Front end:
> > > > > > > >
> > > > > > > > Adding the the pcap search to the alert UI is, I think,
the
> > > easiest
> > > > > way
> > > > > > > to
> > > > > > > > move forward. But indeed, it will then be the “Alert
UI and
> > > > > pcapquery”.
> > > > > > > > Maybe the name of the UI should just change to something
like
> > > > > > > “Monitoring &
> > > > > > > > Investigation UI” ?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Is there any roadmap or plan for the different UI?
I mean did
> > > you
> > > > > > > already
> > > > > > > > had discussion on how you see the ui evolving with
the new
> > > feature
> > > > > that
> > > > > > > > will come in the future?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > - Microservices:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > What do you mean exactly by microservices? Is it to
separate
> > all
> > > > the
> > > > > > > > features in different projects? Or something like
having the
> > > > > different
> > > > > > > > components in container like kubernet? (again maybe
stupid
> > > > question,
> > > > > > but
> > > > > > > I
> > > > > > > > don’t clearly understand what you mean J )
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Michel
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
>
>
>
> --
> --
> simon elliston ball
> @sireb
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message