metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: [DISCUSS][PROPOSAL] Acceptance Tests
Date Mon, 06 Mar 2017 18:32:56 GMT
No, I'm saying that they shouldn't be restricted to real-world use-cases.
The E2E tests I laid out weren't real-world, but they did exercise the
components similar to real-world use-cases.  They should also be able to be
able to tread outside of the happy-path for those use-cases.

On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> "I don't think acceptance tests should loosely associate with real uses,
> but they should
> be free to delve into weird non-happy-pathways."
>
> Not following - are you saying they should *tightly* associate with real
> uses and additonally include non-happy-path?
>
> On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <cestella@gmail.com> wrote:
>
> > It is absolutely not a naive question, Matt.  We don't have a lot (or
> any)
> > docs about our integration tests; it's more of a "follow the lead" type
> of
> > thing at the moment, but that should be rectified.
> >
> > The integration tests spin up and down infrastructure in-process, some of
> > which are real and some of which are mock versions of the services.
> These
> > are good for catching some types of bugs, but often things sneak through,
> > like:
> >
> >    - Hbase and storm can't exist in the same JVM, so HBase is mocked in
> >    those cases.
> >    - The FileSystem that we get for Hadoop is the LocalRawFileSystem, not
> >    truly HDFS.  There are differences and we've run into
> them..hilariously
> > at
> >    times. ;)
> >    - Things done statically in a bolt are shared across all bolts because
> >    they all are threads in the same process
> >
> > It's good, it catches bugs, it lets us debug things easily, it runs with
> > every single build automatically via travis.
> > It's bad because it's awkward to get the dependencies isolated
> sufficiently
> > for all of these components to get them to play nice in the same JVM.
> >
> > Acceptance tests would be run against a real cluster, so they would:
> >
> >    - run against real components, not testing or mock components
> >    - run against multiple nodes
> >
> > I can imagine a world where we can unify the two to a certain degree in
> > many cases if we could spin up a docker version of Metron to run as part
> of
> > the build, but I think in the meantime, we should focus on providing
> both.
> >
> > I suspect the reference application is possibly inspiring my suggestions
> > here, but I think the main difference here is that the reference
> > application is intended to be informational from a end-user perspective:
> > it's detailing a use-case that users will understand.  I don't think
> > acceptance tests should loosely associate with real uses, but they should
> > be free to delve into weird non-happy-pathways.
> >
> > On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley <mattf@apache.org> wrote:
> >
> > > Automating stuff that now has to be done manually gets a big +1.
> > >
> > > But, Casey, could you please clarify the relationship between what you
> > > plan to do and the current “integration test” framework?  Will this be
> in
> > > the form of additional integration tests? Or a different test
> framework?
> > > Can it be done in the integration test framework, rather than creating
> > new
> > > mechanism?
> > >
> > > BTW, if that’s a naïve question, forgive me, but I could find zero
> > > documentation for the existing integration test capability, neither
> wiki
> > > pages nor READMEs nor Jiras.  If there are any docs, please point me at
> > > them.  Or even archived email threads.
> > >
> > > There is also something called the “Reference Application”
> > > https://cwiki.apache.org/confluence/display/METRON/
> > > Metron+Reference+Application which sounds remarkably like what you
> > > propose to automate.  Is there / can there / should there be a
> > relationship?
> > >
> > > Thanks,
> > > --Matt
> > >
> > > On 3/3/17, 7:40 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:
> > >
> > >     +1
> > >
> > >     I agree with Justin’s points.
> > >
> > >
> > >     On March 3, 2017 at 08:41:37, Justin Leet (justinjleet@gmail.com)
> > > wrote:
> > >
> > >     +1 to both. Having this would especially ease a lot of testing that
> > > hits
> > >     multiple areas (which there is a fair amount of, given that we're
> > > building
> > >     pretty quickly).
> > >
> > >     I do want to point out that adding this type of thing makes the
> speed
> > > of
> > >     our builds and tests more important, because they already take up a
> > > good
> > >     amount of time. There are obviously tickets to optimize these
> things,
> > > but
> > >     I would like to make sure we don't pile too much on to every
> testing
> > > cycle
> > >     before a PR. Having said that, I think the testing proposed is
> > > absolutely
> > >     valuable enough to go forward with.
> > >
> > >     Justin
> > >
> > >     On Fri, Mar 3, 2017 at 8:33 AM, Casey Stella <cestella@gmail.com>
> > > wrote:
> > >
> > >     > I also propose, once this is done, that we modify the developer
> > > bylaws
> > >     and
> > >     > the github PR script to ensure that PR authors:
> > >     >
> > >     > - Update the acceptance tests where appropriate
> > >     > - Run the tests as a smoketest
> > >     >
> > >     >
> > >     >
> > >     > On Fri, Mar 3, 2017 at 8:21 AM, Casey Stella <cestella@gmail.com
> >
> > > wrote:
> > >     >
> > >     > > Hi All,
> > >     > >
> > >     > > After doing METRON-744, where I had to walk through a manual
> test
> > > of
> > >     > every
> > >     > > place that Stellar touched, it occurred to me that we should
> > script
> > >     this.
> > >     > > It also occurred to me that some scripts that are run by the
PR
> > > author
> > >     to
> > >     > > ensure no regressions and, eventually maybe, even run on an
> INFRA
> > >     > instance
> > >     > > of Jenkins would give all of us some peace of mind.
> > >     > >
> > >     > > I am certain that this, along with a couple other manual tests
> > from
> > >     other
> > >     > > PRs, could form the basis of a really great regression
> > > acceptance-test
> > >     > > suite and I'd like to propose that we do that, as a community.
> > >     > >
> > >     > > What I'd like to see from such a suite has the following
> > >     characteristics:
> > >     > >
> > >     > > - Can be run on any Metron cluster, including but not limited
> to
> > >     > > - Vagrant
> > >     > > - AWS
> > >     > > - An existing deployment
> > >     > > - Can be *deployed* from ansible, but must be able to be
> deployed
> > >     > > manually
> > >     > > - With instructions in the readme
> > >     > > - Tests should be idempotent and independent
> > >     > > - Tear down what you set up
> > >     > >
> > >     > > I think between the Stellar REPL and the fundamental
> > scriptability
> > > of
> > >     the
> > >     > > Hadoop services, we can accomplish these tests with a
> combination
> > > of
> > >     > shell
> > >     > > scripts and python.
> > >     > >
> > >     > > I propose we break this into the following parts:
> > >     > >
> > >     > > - Acceptance Testing Framework with a small smoketest
> > >     > > - Baseline Metron Test
> > >     > > - Send squid data through the squid topology
> > >     > > - Add an threat triage alert
> > >     > > - Ensure it gets through to the other side with alerts
> preserved
> > >     > > - + Enrichment
> > >     > > - Add an enrichment in the enrichment pipeline to the above
> > >     > > - + Profiler
> > >     > > - Add a profile with a tick of 1 minute to count per
> destination
> > >     > > address
> > >     > > - Base PCap test
> > >     > > - Something like the manual test for METRON-743 (
> > >     > > https://github.com/apache/incubator-metron/pull/467#
> > >     > issue-210285324
> > >     > > <https://github.com/apache/incubator-metron/pull/467#
> > >     > issue-210285324>
> > >     > > )
> > >     > >
> > >     > > Thoughts?
> > >     > >
> > >     > >
> > >     > > Best,
> > >     > >
> > >     > > Casey
> > >     > >
> > >     >
> > >
> > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message