ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artem Shutak <ashu...@gridgain.com>
Subject Re: Full API coverage enhancement
Date Mon, 08 Feb 2016 14:07:58 GMT
Hi all,

I have an update.

I've started from *CacheConfiguration* permutations. I wrote out a list
with all CacheConfiguration setters and filtered it with Alexey G.

Finally we have:

   1. CacheMode - 3 variants
   2. CacheAtomicityMode - 2 variants
   3. CacheMemoryMode - 3 variants
   4. setLoadPreviousValue - 2 variants
   5. setReadFromBackup - 2 variants
   6. setStoreKeepBinary - 2 variants
   7. setRebalanceMode - SYNC and ASYNC (2 variants)
   8. setSwapEnabled - 2 variants
   9. setCopyOnRead - 2 variants
   10. NearConfiguration disabled / default NearConfiguration / custom
   NearConfiguration - 3 variants
   11. With and without a complex parameter. The complex parameter defines
   not-default Eviction policy and filter, cache store configuration
   (storeFactory and storeSessionListenerFactory), rebalancing configuration,
   affinity function, offHeapMaxMemory, interceptor, topology validator and
   CacheEntryListener.

I've run 123 Cache Full Api test cases for all permutations of parameters
1-9 and got 256896 test cases (1152 configuration variants * 123 test
cases). All these tests take 4 hours 40 minutes. Not all tests pass, so MAY
BE when all tests will pass it will take less time (3,5 hours for example).

As we can see the tests take a lot of time.

The following permutation should be supported too:

   1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes and 1
   backups, 4 nodes and 2 backups - 3 variants
   2. Client and Server nodes - 2 variants
   3. Indexing enabled and disabled for cache - 2 variants
   4. IgniteConfiguration permutations - how many variants? I see at least
   4 (2 Marshallers, P2P).

Plus we need add new test cases to test different key and value types and
etc.

So, we need multiply more then 3,5-4,5 hours on ~250. If we will split all
tests on 250 suites and run on all 30 TC agents it will take about 30-40
hours. Ok, we can do it during weekends.

I think it will take too much time.

As an option we can start a cache for each configuration and run tests
concurrently. But we need to implement this opportunity in our test
framework.

Any other thoughts how we can decrease time for tests?

Thanks,
-- Artem --

On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <sboikov@gridgain.com> wrote:

> Artem,
>
> One more thing for new tests: I think test should start both server and
> client nodes and use Ignite API from all nodes.
>
> On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <ashutak@gridgain.com> wrote:
>
> > Dmitriy,
> >
> > Actually, I don't have a list with all the permutations.
> >
> > At first, we need to split in our discussion test cases and Ignite
> > configuration which should be covered.
> >
> > For example, new Full Api test cases for cache are based on old Full Api
> > test cases. So, it need to think what the test cases was not covered
> > before.
> >
> > About Ignite configurations, I'm going to add permutation for each
> > IgniteConfiguration and CacheConfiguration property.
> >
> > By the way, the jira contains the following list of permutation (feel
> free
> > to add something):
> >
> > The following tests should be added (for functional blocks):
> >
> >    1. Interceptor
> >    2. Queries: continuous, scan, SQL, fields and text queries.
> >    3. cache events
> >    4. We should also test with Serializable, Externalizable, and plain
> >    Pojos for keys and values.
> >    5. The Pojo in the above test should contain an enum value
> >    6. We should also test Enums as keys and Enums as values
> >    7. All operations should have single-key and multi-key operations
> >
> > New tests should cover all combinations for following properties:
> >
> >    1. cache modes
> >    2. operation from client nodes and server nodes
> >    3. store enabled/disabled
> >    4. evicts sycn/non-sync
> >    5. eviction policies
> >    6. near on/off
> >    7. marshallers (+ Binary marshaller with different mappers)
> >    8. keys and values - externalizable, serializable, binaryzable, "none
> of
> >    previous"
> >    9. classes available on servers: true/false
> >    10. Peer loading on/off
> >    11. Affinity functions
> >    12. expiry policies
> >
> >
> > Thanks,
> > -- Artem --
> >
> > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> > wrote:
> >
> > > Artem, I think it is best to specify all the permutations here, so
> others
> > > can make additional suggestions. Otherwise, we cannot get a full
> picture.
> > >
> > > Thanks,
> > > D.
> > >
> > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <ashutak@gridgain.com>
> > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I thought a little bit more and think we need to add a support for
> the
> > > > following permutations too (I've added these to the jira
> description):
> > > > - We should also test with Serializable, Externalizable, and plain
> > Pojos
> > > > for keys and values.
> > > > - The Pojo in the above test should contain an enum value
> > > > - We should also test Enums as keys and Enums as values
> > > > - All operations should have single-key and multi-key operations
> > > >
> > > > Maybe someone see any other permutation to be supported?
> > > >
> > > > -- Artem --
> > > >
> > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak <ashutak@gridgain.com>
> > > > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > There is a branch at my fork and a pull request at Ignite. See
> > comment
> > > > > about pull request at the ticket (PR-446).
> > > > >
> > > > > But I have to notice that the branch under hard development and you
> > it
> > > > can
> > > > > not work (have compilation or test errors) at some moments.
> > > > >
> > > > > Good luck!
> > > > >
> > > > > -- Artem --
> > > > >
> > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy Setrakyan <
> > > dsetrakyan@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Artem,
> > > > >>
> > > > >> This is great. I have noticed from the ticket that you have
> created
> > > some
> > > > >> initial suite already. Is there a branch I can look at it?
> > > > >>
> > > > >> D.
> > > > >>
> > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem Shutak <
> ashutak@gridgain.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > Igniters,
> > > > >> >
> > > > >> > I'm working on an enhancement of Full API coverage [1] [2].
> > > > >> >
> > > > >> > Ignite already has Full API test, but currently it's hard
to
> test
> > > all
> > > > >> > configuration combinations.
> > > > >> >
> > > > >> > Feel free to add comments in the jira if you have any thought.
> > > > >> >
> > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > >> > [2]
> > > > >>
> > https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > >> >
> > > > >> > Thanks,
> > > > >> > -- Artem --
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message