metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: [DISCUSS][PROPOSAL] Acceptance Tests
Date Mon, 06 Mar 2017 18:30:48 GMT
"I don't think acceptance tests should loosely associate with real uses,
but they should
be free to delve into weird non-happy-pathways."

Not following - are you saying they should *tightly* associate with real
uses and additonally include non-happy-path?

On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <cestella@gmail.com> wrote:

> It is absolutely not a naive question, Matt.  We don't have a lot (or any)
> docs about our integration tests; it's more of a "follow the lead" type of
> thing at the moment, but that should be rectified.
>
> The integration tests spin up and down infrastructure in-process, some of
> which are real and some of which are mock versions of the services.  These
> are good for catching some types of bugs, but often things sneak through,
> like:
>
>    - Hbase and storm can't exist in the same JVM, so HBase is mocked in
>    those cases.
>    - The FileSystem that we get for Hadoop is the LocalRawFileSystem, not
>    truly HDFS.  There are differences and we've run into them..hilariously
> at
>    times. ;)
>    - Things done statically in a bolt are shared across all bolts because
>    they all are threads in the same process
>
> It's good, it catches bugs, it lets us debug things easily, it runs with
> every single build automatically via travis.
> It's bad because it's awkward to get the dependencies isolated sufficiently
> for all of these components to get them to play nice in the same JVM.
>
> Acceptance tests would be run against a real cluster, so they would:
>
>    - run against real components, not testing or mock components
>    - run against multiple nodes
>
> I can imagine a world where we can unify the two to a certain degree in
> many cases if we could spin up a docker version of Metron to run as part of
> the build, but I think in the meantime, we should focus on providing both.
>
> I suspect the reference application is possibly inspiring my suggestions
> here, but I think the main difference here is that the reference
> application is intended to be informational from a end-user perspective:
> it's detailing a use-case that users will understand.  I don't think
> acceptance tests should loosely associate with real uses, but they should
> be free to delve into weird non-happy-pathways.
>
> On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley <mattf@apache.org> wrote:
>
> > Automating stuff that now has to be done manually gets a big +1.
> >
> > But, Casey, could you please clarify the relationship between what you
> > plan to do and the current “integration test” framework?  Will this be in
> > the form of additional integration tests? Or a different test framework?
> > Can it be done in the integration test framework, rather than creating
> new
> > mechanism?
> >
> > BTW, if that’s a naïve question, forgive me, but I could find zero
> > documentation for the existing integration test capability, neither wiki
> > pages nor READMEs nor Jiras.  If there are any docs, please point me at
> > them.  Or even archived email threads.
> >
> > There is also something called the “Reference Application”
> > https://cwiki.apache.org/confluence/display/METRON/
> > Metron+Reference+Application which sounds remarkably like what you
> > propose to automate.  Is there / can there / should there be a
> relationship?
> >
> > Thanks,
> > --Matt
> >
> > On 3/3/17, 7:40 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:
> >
> >     +1
> >
> >     I agree with Justin’s points.
> >
> >
> >     On March 3, 2017 at 08:41:37, Justin Leet (justinjleet@gmail.com)
> > wrote:
> >
> >     +1 to both. Having this would especially ease a lot of testing that
> > hits
> >     multiple areas (which there is a fair amount of, given that we're
> > building
> >     pretty quickly).
> >
> >     I do want to point out that adding this type of thing makes the speed
> > of
> >     our builds and tests more important, because they already take up a
> > good
> >     amount of time. There are obviously tickets to optimize these things,
> > but
> >     I would like to make sure we don't pile too much on to every testing
> > cycle
> >     before a PR. Having said that, I think the testing proposed is
> > absolutely
> >     valuable enough to go forward with.
> >
> >     Justin
> >
> >     On Fri, Mar 3, 2017 at 8:33 AM, Casey Stella <cestella@gmail.com>
> > wrote:
> >
> >     > I also propose, once this is done, that we modify the developer
> > bylaws
> >     and
> >     > the github PR script to ensure that PR authors:
> >     >
> >     > - Update the acceptance tests where appropriate
> >     > - Run the tests as a smoketest
> >     >
> >     >
> >     >
> >     > On Fri, Mar 3, 2017 at 8:21 AM, Casey Stella <cestella@gmail.com>
> > wrote:
> >     >
> >     > > Hi All,
> >     > >
> >     > > After doing METRON-744, where I had to walk through a manual test
> > of
> >     > every
> >     > > place that Stellar touched, it occurred to me that we should
> script
> >     this.
> >     > > It also occurred to me that some scripts that are run by the PR
> > author
> >     to
> >     > > ensure no regressions and, eventually maybe, even run on an INFRA
> >     > instance
> >     > > of Jenkins would give all of us some peace of mind.
> >     > >
> >     > > I am certain that this, along with a couple other manual tests
> from
> >     other
> >     > > PRs, could form the basis of a really great regression
> > acceptance-test
> >     > > suite and I'd like to propose that we do that, as a community.
> >     > >
> >     > > What I'd like to see from such a suite has the following
> >     characteristics:
> >     > >
> >     > > - Can be run on any Metron cluster, including but not limited to
> >     > > - Vagrant
> >     > > - AWS
> >     > > - An existing deployment
> >     > > - Can be *deployed* from ansible, but must be able to be deployed
> >     > > manually
> >     > > - With instructions in the readme
> >     > > - Tests should be idempotent and independent
> >     > > - Tear down what you set up
> >     > >
> >     > > I think between the Stellar REPL and the fundamental
> scriptability
> > of
> >     the
> >     > > Hadoop services, we can accomplish these tests with a combination
> > of
> >     > shell
> >     > > scripts and python.
> >     > >
> >     > > I propose we break this into the following parts:
> >     > >
> >     > > - Acceptance Testing Framework with a small smoketest
> >     > > - Baseline Metron Test
> >     > > - Send squid data through the squid topology
> >     > > - Add an threat triage alert
> >     > > - Ensure it gets through to the other side with alerts preserved
> >     > > - + Enrichment
> >     > > - Add an enrichment in the enrichment pipeline to the above
> >     > > - + Profiler
> >     > > - Add a profile with a tick of 1 minute to count per destination
> >     > > address
> >     > > - Base PCap test
> >     > > - Something like the manual test for METRON-743 (
> >     > > https://github.com/apache/incubator-metron/pull/467#
> >     > issue-210285324
> >     > > <https://github.com/apache/incubator-metron/pull/467#
> >     > issue-210285324>
> >     > > )
> >     > >
> >     > > Thoughts?
> >     > >
> >     > >
> >     > > Best,
> >     > >
> >     > > Casey
> >     > >
> >     >
> >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message