metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: [DISCUSS] Getting to a 1.0 release
Date Mon, 20 Aug 2018 12:50:12 GMT
I completely agree, Mike.  Our docs are either very high level or very low
level (and possibly stale) and, worse, aren't aimed at the actors that
you've stated.
I think that the HBase project does a good job of providing coherent and
useable documentation in their "HBase Book" (see
https://hbase.apache.org/book.html).
It's not actor-specific, but it is coherent advice for the practical
practitioner of HBase (both admin and developer) and speaks with one
voice.  I think Metron's need
is a bit different, but at the minimum some coherent docs that speaks with
one voice and has a coherent pitch about what Metron is used for and what
it isn't used for
is well needed.

On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Apologies for any spelling mishaps as I'm writing from my phone.
>
> I'm for improving our docs. I'd like to see us guide our various profiles
> of user towards the specific documentation for the abstraction levels
> they'll be most interested in working from. I think we should have platform
> docs about how we're a broadly useful, extensible streaming analytics
> platform for cyber security as well as docs that emphasize more narrow and
> specific use cases.
>
> Personally, I think I see 3 potential tiers or classifications of docs.
> These are just observations and ideas I had, not necessarily a prescription
> for organizing docs:
> - Low level tool instructions, eg
>     - how do I run the pcap toplogy and then query with the CLI and UI?
> - Platform docs about building on top of Metron, e.g.
>     - writing custom parsers, enrichment, and threat Intel (imho we should
> start to take a more opinionated view of leveraging Stellar as this
> extension point rather than implementing new parser classes in Java)
>     - using the profiler for constructing outlier analysis use cases
>     - using MAAS for building and deploying models for use in enrichment
> - Docs around more specific use cases that solve specific as opposed to
> more general problems, similar to those we have in the use-cases folder.
>
> I think one of our challenges currently is that our docs could be better
> tailored to the "actors" we've talked about in the past. An individual SOC
> analyst will have a very different set of interests than would a reseller
> that wants to build on top of our platform to expose new modules and
> functionality to those SOC analyst. And we can, and do, currently support
> both.
>
>
> On Sat, Aug 18, 2018, 9:34 AM Nick Allen <nick@nickallen.org> wrote:
>
> > Yes, I imagine just a separate top level directory which would contain
> the
> > docs.
> >
> > We would need someone to survey what doc tools are out there and provide
> > some advice.
> >
> > Maybe we could look around at other open source projects that have done
> > their docs well and emulate them.
> >
> > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson <
> kylerichardson2@gmail.com>
> > wrote:
> >
> > > +1 to separating developer docs and user docs. How should we approach
> > that.
> > > Have a separate doc book? I haven’t had a ton of time to contribute to
> > code
> > > lately but I’d be happy to help write some of these.
> > >
> > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen <nick@nickallen.org> wrote:
> > >
> > > > Personally, I think the state of our docs and web presence is an
> > > inhibitor
> > > > to growing the Metron community.  Unless we can offer concise,
> > compelling
> > > > answers to the basic questions (What can I do with Metron?  Who does
> it
> > > > help? How do I do that?), potential users and contributors are unable
> > to
> > > > see the value of Metron.
> > > >
> > > >
> > > >
> > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen <nick@nickallen.org>
> > wrote:
> > > >
> > > > > I'd like to see us focus on improving our docs before a version
> 1.0.
> > > > > Right now we just stitch together a bunch of READMEs, which is a
> > great
> > > > > stride from where we started, but is not ideal.
> > > > >
> > > > > Our docs should focused on the user and use cases; What can I do
> with
> > > > > Metron?  Who does it help? How do I do that?
> > > > >
> > > > > The docs should be separate from the code base to allow for an
> > > > > organization that is focused on the user rather than the
> > > implementation.
> > > > > This allows the READMEs to focus on the developer and the
> > > implementation,
> > > > > which should make them more digestible too.  The docs should be
> > version
> > > > > controlled and maintained through PRs, just like the code.  We
> should
> > > > take
> > > > > just as much pride in our docs as we do in our code.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > > > > simon@simonellistonball.com> wrote:
> > > > >
> > > > >> Agreed, should we add TDE by default, and get the ranger policies
> on
> > > by
> > > > >> default? That leaves secured in Kafka, which would have to be
> built
> > > into
> > > > >> the consumers and producers to encrypt into the on disk Kafka
> > topics.
> > > > Does
> > > > >> that seem necessary to people? It would have performance
> > implications
> > > > for
> > > > >> sure.
> > > > >>
> > > > >> Simon
> > > > >>
> > > > >> > On 15 Aug 2018, at 21:26, Otto Fowler <ottobackwards@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > Well, I look at it like this.
> > > > >> >
> > > > >> > The Secure Vault was part of the original metron pitch,
and many
> > may
> > > > >> have used that as part of their evaluations.
> > > > >> > “Look, it is going to have a security vault type thing,
it is on
> > the
> > > > >> roadmap”.
> > > > >> >
> > > > >> > Regardless of the implementation, conceptually, security
of data
> > at
> > > > >> rest is important, and is a major outstanding item or the core
> > metron
> > > > >> proposition.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball
(
> > > > >> simon@simonellistonball.com) wrote:
> > > > >> >>
> > > > >> >> That’s going back a way. I always saw that concept
as begin
> about
> > > the
> > > > >> formats, e.g. Orc, and meta data around it plus the data service
> api
> > > to
> > > > get
> > > > >> at it. I’m all for that too, but think it needs more thought
than
> > the
> > > > >> ticket captures.
> > > > >> >>
> > > > >> >> Simon
> > > > >> >>
> > > > >> >> On 15 Aug 2018, at 20:53, Otto Fowler <ottobackwards@gmail.com
> >
> > > > wrote:
> > > > >> >>
> > > > >> >>> https://issues.apache.org/jira/browse/METRON-343
> > > > >> >>>
> > > > >> >>>> On August 15, 2018 at 15:47:24, Simon Elliston
Ball (
> > > > >> simon@simonellistonball.com) wrote:
> > > > >> >>>>
> > > > >> >>>> What would you see as secure? I’ve seen people
use TDE for
> the
> > > HDFS
> > > > >> store, but it’s harder to encrypt storage with solr / es.
> Something
> > I
> > > > was
> > > > >> thinking of doing to follow up on the Knox Feature was to add
> Ranger
> > > > >> integration for securing and auditing configs, and potentially
> > > > extending to
> > > > >> the index destinations. Do you think that would cover the secure
> > > storage
> > > > >> concept?
> > > > >> >>>>
> > > > >> >>>> Simon
> > > > >> >>>>
> > > > >> >>>> > On 15 Aug 2018, at 20:39, Otto Fowler <
> > ottobackwards@gmail.com
> > > >
> > > > >> wrote:
> > > > >> >>>> >
> > > > >> >>>> > Secure storage off the top of my head
> > > > >> >>>> >
> > > > >> >>>> > On August 15, 2018 at 14:49:26, Zeolla@GMail.com
(
> > > > zeolla@gmail.com)
> > > > >> wrote:
> > > > >> >>>> >
> > > > >> >>>> > So, as has been discussed in a few
> > > > >> >>>> > <
> > > > >> >>>> >
> https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a
> > > > >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
> > > > >> >>>> >
> > > > >> >>>> > other
> > > > >> >>>> > <
> > > > >> >>>> >
> https://lists.apache.org/thread.html/427a20c22207e84331b94e8
> > > > >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
> > > > >> >>>> >
> > > > >> >>>> > recent dev list threads, I would like to
discuss what a
> > Metron
> > > > 1.0
> > > > >> release
> > > > >> >>>> > looks like.
> > > > >> >>>> >
> > > > >> >>>> > In order to kick off the conversation,
I would like to
> make a
> > > few
> > > > >> >>>> > suggestions regarding "what 1.0 means to
me," but I'm very
> > > > >> interested to
> > > > >> >>>> > hear everybody else's opinions.
> > > > >> >>>> >
> > > > >> >>>> > In order to go 1.0 I believe we should
have:
> > > > >> >>>> > 1. A clear, supported method of upgrading
from one version
> of
> > > > >> Metron to the
> > > > >> >>>> > next. We have attempted
> > > > >> >>>> > <https://github.com/apache/metron/blob/master/Upgrading.md
> >
> > to
> > > > >> make this
> > > > >> >>>> > easier in the past, but it is currently
not
> > > > >> >>>> > <
> > > > >> >>>> >
> https://github.com/apache/metron/tree/master/metron-deployme
> > > > >> nt/packaging/ambari/metron-mpack#limitations>
> > > > >> >>>> >
> > > > >> >>>> > supported
> > > > >> >>>> > <
> > > > >> >>>> >
> https://github.com/apache/metron/tree/master/metron-deployme
> > > > >> nt/packaging/ambari/elasticsearch-mpack#limitations>
> > > > >> >>>> >
> > > > >> >>>> > .
> > > > >> >>>> > 2. Authentication for all of the UIs and
APIs should be
> > secure
> > > > and
> > > > >> support
> > > > >> >>>> > SSO. I believe this is in progress via
METRON-1663
> > > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1663>.
> > > > >> >>>> > 3. Each of our personas
> > > > >> >>>> > <
> > > > >> >>>> > https://cwiki.apache.org/confluence/display/METRON/Metron+
> > > > >> User+Personas+And+Benefits>
> > > > >> >>>> >
> > > > >> >>>> > should
> > > > >> >>>> > be well documented, understood, and supported.
> > > > >> >>>> > - The current state of documentation is,
in my opinion,
> > > > inadequate
> > > > >> and I
> > > > >> >>>> > admit I am partially to blame for this.
I suggest we
> define a
> > > > >> strict
> > > > >> >>>> > approach for documentation, align to it
(such as perhaps
> > > > migrating
> > > > >> all
> > > > >> >>>> > useful wiki documentation to git), and
enforce it.
> > > > >> >>>> > - I would consider METRON-1699
> > > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1699>
as a
> > > > critical
> > > > >> item for
> > > > >> >>>> > a Security Data Scientist, but it is currently
not clear to
> > me
> > > > >> where the
> > > > >> >>>> > line exists between some of the other personas,
or that
> each
> > > > >> persona has
> > > > >> >>>> > been sufficiently implemented.
> > > > >> >>>> > 4. A performance tuning guide should be
available for all
> of
> > > the
> > > > >> main
> > > > >> >>>> > components, whether as an independent document
or as a part
> > of
> > > a
> > > > >> larger
> > > > >> >>>> > document.
> > > > >> >>>> > 5. Simple data ingest.
> > > > >> >>>> > - Similar to the ongoing conversation for
NiFi integration
> > > > >> >>>> > <
> > > > >> >>>> >
> https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f269
> > > > >> 73f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
> > > > >> >>>> >
> > > > >> >>>> > we should be able to say that we have broken
down the
> > barriers
> > > to
> > > > >> getting
> > > > >> >>>> > data into a Metron cluster in easy and
efficient ways. In
> > > > addition
> > > > >> to
> > > > >> >>>> > NiFi, having support for other popular
tools such as beats
> > > > >> >>>> > <https://www.elastic.co/products/beats>,
fluentd <
> > > > >> https://www.fluentd.org/>,
> > > > >> >>>> >
> > > > >> >>>> > etc.
> > > > >> >>>> > - Parsers should be pluggable, with independent
tests and
> the
> > > > >> ability to
> > > > >> >>>> > make versioned modifications with roll-backs.
> > > > >> >>>> >
> > > > >> >>>> > What else? Are any of these items not necessary
for a 1.0?
> > > > >> >>>> >
> > > > >> >>>> > Jon
> > > > >> >>>> > --
> > > > >> >>>> >
> > > > >> >>>> > Jon
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message