metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: [DISCUSS][PROPOSAL] Acceptance Tests
Date Fri, 03 Mar 2017 19:16:13 GMT
Automating stuff that now has to be done manually gets a big +1.

But, Casey, could you please clarify the relationship between what you plan to do and the
current “integration test” framework?  Will this be in the form of additional integration
tests? Or a different test framework?  Can it be done in the integration test framework, rather
than creating new mechanism?

BTW, if that’s a naïve question, forgive me, but I could find zero documentation for the
existing integration test capability, neither wiki pages nor READMEs nor Jiras.  If there
are any docs, please point me at them.  Or even archived email threads.

There is also something called the “Reference Application” https://cwiki.apache.org/confluence/display/METRON/Metron+Reference+Application
which sounds remarkably like what you propose to automate.  Is there / can there / should
there be a relationship?

Thanks,
--Matt

On 3/3/17, 7:40 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:

    +1
    
    I agree with Justin’s points.
    
    
    On March 3, 2017 at 08:41:37, Justin Leet (justinjleet@gmail.com) wrote:
    
    +1 to both. Having this would especially ease a lot of testing that hits
    multiple areas (which there is a fair amount of, given that we're building
    pretty quickly).
    
    I do want to point out that adding this type of thing makes the speed of
    our builds and tests more important, because they already take up a good
    amount of time. There are obviously tickets to optimize these things, but
    I would like to make sure we don't pile too much on to every testing cycle
    before a PR. Having said that, I think the testing proposed is absolutely
    valuable enough to go forward with.
    
    Justin
    
    On Fri, Mar 3, 2017 at 8:33 AM, Casey Stella <cestella@gmail.com> wrote:
    
    > I also propose, once this is done, that we modify the developer bylaws
    and
    > the github PR script to ensure that PR authors:
    >
    > - Update the acceptance tests where appropriate
    > - Run the tests as a smoketest
    >
    >
    >
    > On Fri, Mar 3, 2017 at 8:21 AM, Casey Stella <cestella@gmail.com> wrote:
    >
    > > Hi All,
    > >
    > > After doing METRON-744, where I had to walk through a manual test of
    > every
    > > place that Stellar touched, it occurred to me that we should script
    this.
    > > It also occurred to me that some scripts that are run by the PR author
    to
    > > ensure no regressions and, eventually maybe, even run on an INFRA
    > instance
    > > of Jenkins would give all of us some peace of mind.
    > >
    > > I am certain that this, along with a couple other manual tests from
    other
    > > PRs, could form the basis of a really great regression acceptance-test
    > > suite and I'd like to propose that we do that, as a community.
    > >
    > > What I'd like to see from such a suite has the following
    characteristics:
    > >
    > > - Can be run on any Metron cluster, including but not limited to
    > > - Vagrant
    > > - AWS
    > > - An existing deployment
    > > - Can be *deployed* from ansible, but must be able to be deployed
    > > manually
    > > - With instructions in the readme
    > > - Tests should be idempotent and independent
    > > - Tear down what you set up
    > >
    > > I think between the Stellar REPL and the fundamental scriptability of
    the
    > > Hadoop services, we can accomplish these tests with a combination of
    > shell
    > > scripts and python.
    > >
    > > I propose we break this into the following parts:
    > >
    > > - Acceptance Testing Framework with a small smoketest
    > > - Baseline Metron Test
    > > - Send squid data through the squid topology
    > > - Add an threat triage alert
    > > - Ensure it gets through to the other side with alerts preserved
    > > - + Enrichment
    > > - Add an enrichment in the enrichment pipeline to the above
    > > - + Profiler
    > > - Add a profile with a tick of 1 minute to count per destination
    > > address
    > > - Base PCap test
    > > - Something like the manual test for METRON-743 (
    > > https://github.com/apache/incubator-metron/pull/467#
    > issue-210285324
    > > <https://github.com/apache/incubator-metron/pull/467#
    > issue-210285324>
    > > )
    > >
    > > Thoughts?
    > >
    > >
    > > Best,
    > >
    > > Casey
    > >
    >
    



Mime
View raw message