metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <merrim...@gmail.com>
Subject Re: [DISCUSS][PROPOSAL] Acceptance Tests
Date Wed, 22 Mar 2017 19:05:47 GMT
I think we'll have non manual rest/web deployment soon regardless of this
discussion.

On Wed, Mar 22, 2017 at 2:00 PM, Ryan Merriman <merrimanr@gmail.com> wrote:

> I don't think a cluster installed by ansible is a prerequisite to using
> ansible to integration test.  They would be completely separate modules
> except maybe sharing some property or inventory files.  Just need to run
> scripts and hit rest endpoints right?  Just an idea, maybe it's overkill.
> I'm cool with rolling our own.
>
> On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella <cestella@gmail.com> wrote:
>
>> Maybe, but I'd argue that we would want this to be run against a
>> non-ansible installed cluster.  For a first pass, I'd recommend just a set
>> of shell scripts utilizing the REPL and the REST API along with shell
>> commands.  Most of our capabilities are quite scriptable.
>>
>> On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman <merrimanr@gmail.com>
>> wrote:
>>
>> > Bumping this thread.  Looks like we have several +1s so I propose we
>> move
>> > to the next step.  I'm anxious to get this done because these tests
>> would
>> > have saved me time over the last couple weeks.  The management UI in
>> > https://github.com/apache/incubator-metron/pull/484 has a set of e2e
>> tests
>> > being maintained in another branch so those could also be included in
>> this
>> > test suite when the UI makes it into master.
>> >
>> > Ideas for an "Acceptance Testing Framework"?  Could Ansible be good fit
>> for
>> > this since we already have it in our stack?
>> >
>> > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
>> > michael.miklavcic@gmail.com> wrote:
>> >
>> > > Ok, yes I agree. In my experience with e2e/acceptance tests, they're
>> best
>> > > kept general with an emphasis on verifying that all the plumbing works
>> > > together. So yes, there are definite edge cases I think we'll want to
>> > test
>> > > here, but I say that with the caveat that I think we should ideally
>> cover
>> > > as many non-happy-path cases in unit and integration tests as
>> possible.
>> > As
>> > > an example, I don't think it makes sense to cover most of the profiler
>> > > windowing DSL language edge cases in acceptance tests instead of or in
>> > > addition to unit/integration tests unless there is something specific
>> to
>> > > the integration with a given an environment that we think could be
>> > > problematic.
>> > >
>> > > M
>> > >
>> > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella <cestella@gmail.com>
>> > wrote:
>> > >
>> > > > No, I'm saying that they shouldn't be restricted to real-world
>> > use-cases.
>> > > > The E2E tests I laid out weren't real-world, but they did exercise
>> the
>> > > > components similar to real-world use-cases.  They should also be
>> able
>> > to
>> > > be
>> > > > able to tread outside of the happy-path for those use-cases.
>> > > >
>> > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
>> > > > michael.miklavcic@gmail.com> wrote:
>> > > >
>> > > > > "I don't think acceptance tests should loosely associate with
real
>> > > uses,
>> > > > > but they should
>> > > > > be free to delve into weird non-happy-pathways."
>> > > > >
>> > > > > Not following - are you saying they should *tightly* associate
>> with
>> > > real
>> > > > > uses and additonally include non-happy-path?
>> > > > >
>> > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <cestella@gmail.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > It is absolutely not a naive question, Matt.  We don't have
a
>> lot
>> > (or
>> > > > > any)
>> > > > > > docs about our integration tests; it's more of a "follow
the
>> lead"
>> > > type
>> > > > > of
>> > > > > > thing at the moment, but that should be rectified.
>> > > > > >
>> > > > > > The integration tests spin up and down infrastructure
>> in-process,
>> > > some
>> > > > of
>> > > > > > which are real and some of which are mock versions of the
>> services.
>> > > > > These
>> > > > > > are good for catching some types of bugs, but often things
sneak
>> > > > through,
>> > > > > > like:
>> > > > > >
>> > > > > >    - Hbase and storm can't exist in the same JVM, so HBase
is
>> > mocked
>> > > in
>> > > > > >    those cases.
>> > > > > >    - The FileSystem that we get for Hadoop is the
>> > LocalRawFileSystem,
>> > > > not
>> > > > > >    truly HDFS.  There are differences and we've run into
>> > > > > them..hilariously
>> > > > > > at
>> > > > > >    times. ;)
>> > > > > >    - Things done statically in a bolt are shared across
all
>> bolts
>> > > > because
>> > > > > >    they all are threads in the same process
>> > > > > >
>> > > > > > It's good, it catches bugs, it lets us debug things easily,
it
>> runs
>> > > > with
>> > > > > > every single build automatically via travis.
>> > > > > > It's bad because it's awkward to get the dependencies isolated
>> > > > > sufficiently
>> > > > > > for all of these components to get them to play nice in
the same
>> > JVM.
>> > > > > >
>> > > > > > Acceptance tests would be run against a real cluster, so
they
>> > would:
>> > > > > >
>> > > > > >    - run against real components, not testing or mock components
>> > > > > >    - run against multiple nodes
>> > > > > >
>> > > > > > I can imagine a world where we can unify the two to a certain
>> > degree
>> > > in
>> > > > > > many cases if we could spin up a docker version of Metron
to
>> run as
>> > > > part
>> > > > > of
>> > > > > > the build, but I think in the meantime, we should focus
on
>> > providing
>> > > > > both.
>> > > > > >
>> > > > > > I suspect the reference application is possibly inspiring
my
>> > > > suggestions
>> > > > > > here, but I think the main difference here is that the reference
>> > > > > > application is intended to be informational from a end-user
>> > > > perspective:
>> > > > > > it's detailing a use-case that users will understand.  I
don't
>> > think
>> > > > > > acceptance tests should loosely associate with real uses,
but
>> they
>> > > > should
>> > > > > > be free to delve into weird non-happy-pathways.
>> > > > > >
>> > > > > > On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley <mattf@apache.org>
>> > wrote:
>> > > > > >
>> > > > > > > Automating stuff that now has to be done manually gets
a big
>> +1.
>> > > > > > >
>> > > > > > > But, Casey, could you please clarify the relationship
between
>> > what
>> > > > you
>> > > > > > > plan to do and the current “integration test” framework?
 Will
>> > this
>> > > > be
>> > > > > in
>> > > > > > > the form of additional integration tests? Or a different
test
>> > > > > framework?
>> > > > > > > Can it be done in the integration test framework, rather
than
>> > > > creating
>> > > > > > new
>> > > > > > > mechanism?
>> > > > > > >
>> > > > > > > BTW, if that’s a naïve question, forgive me, but
I could find
>> > zero
>> > > > > > > documentation for the existing integration test capability,
>> > neither
>> > > > > wiki
>> > > > > > > pages nor READMEs nor Jiras.  If there are any docs,
please
>> point
>> > > me
>> > > > at
>> > > > > > > them.  Or even archived email threads.
>> > > > > > >
>> > > > > > > There is also something called the “Reference Application”
>> > > > > > > https://cwiki.apache.org/confluence/display/METRON/
>> > > > > > > Metron+Reference+Application which sounds remarkably
like what
>> > you
>> > > > > > > propose to automate.  Is there / can there / should
there be a
>> > > > > > relationship?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > --Matt
>> > > > > > >
>> > > > > > > On 3/3/17, 7:40 AM, "Otto Fowler" <ottobackwards@gmail.com>
>> > wrote:
>> > > > > > >
>> > > > > > >     +1
>> > > > > > >
>> > > > > > >     I agree with Justin’s points.
>> > > > > > >
>> > > > > > >
>> > > > > > >     On March 3, 2017 at 08:41:37, Justin Leet (
>> > > justinjleet@gmail.com
>> > > > )
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >     +1 to both. Having this would especially ease a
lot of
>> > testing
>> > > > that
>> > > > > > > hits
>> > > > > > >     multiple areas (which there is a fair amount of,
given
>> that
>> > > we're
>> > > > > > > building
>> > > > > > >     pretty quickly).
>> > > > > > >
>> > > > > > >     I do want to point out that adding this type of
thing
>> makes
>> > the
>> > > > > speed
>> > > > > > > of
>> > > > > > >     our builds and tests more important, because they
already
>> > take
>> > > > up a
>> > > > > > > good
>> > > > > > >     amount of time. There are obviously tickets to
optimize
>> these
>> > > > > things,
>> > > > > > > but
>> > > > > > >     I would like to make sure we don't pile too much
on to
>> every
>> > > > > testing
>> > > > > > > cycle
>> > > > > > >     before a PR. Having said that, I think the testing
>> proposed
>> > is
>> > > > > > > absolutely
>> > > > > > >     valuable enough to go forward with.
>> > > > > > >
>> > > > > > >     Justin
>> > > > > > >
>> > > > > > >     On Fri, Mar 3, 2017 at 8:33 AM, Casey Stella <
>> > > cestella@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >     > I also propose, once this is done, that we
modify the
>> > > developer
>> > > > > > > bylaws
>> > > > > > >     and
>> > > > > > >     > the github PR script to ensure that PR authors:
>> > > > > > >     >
>> > > > > > >     > - Update the acceptance tests where appropriate
>> > > > > > >     > - Run the tests as a smoketest
>> > > > > > >     >
>> > > > > > >     >
>> > > > > > >     >
>> > > > > > >     > On Fri, Mar 3, 2017 at 8:21 AM, Casey Stella
<
>> > > > cestella@gmail.com
>> > > > > >
>> > > > > > > wrote:
>> > > > > > >     >
>> > > > > > >     > > Hi All,
>> > > > > > >     > >
>> > > > > > >     > > After doing METRON-744, where I had to
walk through a
>> > > manual
>> > > > > test
>> > > > > > > of
>> > > > > > >     > every
>> > > > > > >     > > place that Stellar touched, it occurred
to me that we
>> > > should
>> > > > > > script
>> > > > > > >     this.
>> > > > > > >     > > It also occurred to me that some scripts
that are run
>> by
>> > > the
>> > > > PR
>> > > > > > > author
>> > > > > > >     to
>> > > > > > >     > > ensure no regressions and, eventually
maybe, even run
>> on
>> > an
>> > > > > INFRA
>> > > > > > >     > instance
>> > > > > > >     > > of Jenkins would give all of us some
peace of mind.
>> > > > > > >     > >
>> > > > > > >     > > I am certain that this, along with a
couple other
>> manual
>> > > > tests
>> > > > > > from
>> > > > > > >     other
>> > > > > > >     > > PRs, could form the basis of a really
great regression
>> > > > > > > acceptance-test
>> > > > > > >     > > suite and I'd like to propose that we
do that, as a
>> > > > community.
>> > > > > > >     > >
>> > > > > > >     > > What I'd like to see from such a suite
has the
>> following
>> > > > > > >     characteristics:
>> > > > > > >     > >
>> > > > > > >     > > - Can be run on any Metron cluster, including
but not
>> > > limited
>> > > > > to
>> > > > > > >     > > - Vagrant
>> > > > > > >     > > - AWS
>> > > > > > >     > > - An existing deployment
>> > > > > > >     > > - Can be *deployed* from ansible, but
must be able to
>> be
>> > > > > deployed
>> > > > > > >     > > manually
>> > > > > > >     > > - With instructions in the readme
>> > > > > > >     > > - Tests should be idempotent and independent
>> > > > > > >     > > - Tear down what you set up
>> > > > > > >     > >
>> > > > > > >     > > I think between the Stellar REPL and
the fundamental
>> > > > > > scriptability
>> > > > > > > of
>> > > > > > >     the
>> > > > > > >     > > Hadoop services, we can accomplish these
tests with a
>> > > > > combination
>> > > > > > > of
>> > > > > > >     > shell
>> > > > > > >     > > scripts and python.
>> > > > > > >     > >
>> > > > > > >     > > I propose we break this into the following
parts:
>> > > > > > >     > >
>> > > > > > >     > > - Acceptance Testing Framework with a
small smoketest
>> > > > > > >     > > - Baseline Metron Test
>> > > > > > >     > > - Send squid data through the squid topology
>> > > > > > >     > > - Add an threat triage alert
>> > > > > > >     > > - Ensure it gets through to the other
side with alerts
>> > > > > preserved
>> > > > > > >     > > - + Enrichment
>> > > > > > >     > > - Add an enrichment in the enrichment
pipeline to the
>> > above
>> > > > > > >     > > - + Profiler
>> > > > > > >     > > - Add a profile with a tick of 1 minute
to count per
>> > > > > destination
>> > > > > > >     > > address
>> > > > > > >     > > - Base PCap test
>> > > > > > >     > > - Something like the manual test for
METRON-743 (
>> > > > > > >     > > https://github.com/apache/incubator-metron/pull/467#
>> > > > > > >     > issue-210285324
>> > > > > > >     > > <https://github.com/apache/incubator-metron/pull/467#
>> > > > > > >     > issue-210285324>
>> > > > > > >     > > )
>> > > > > > >     > >
>> > > > > > >     > > Thoughts?
>> > > > > > >     > >
>> > > > > > >     > >
>> > > > > > >     > > Best,
>> > > > > > >     > >
>> > > > > > >     > > Casey
>> > > > > > >     > >
>> > > > > > >     >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message