ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artem Shutak <ashu...@gridgain.com>
Subject Re: Full API coverage enhancement
Date Mon, 08 Feb 2016 17:52:05 GMT
Alexey,

First of all, it's strange to have disk-dependent tests for in-memory data
fabric ;) I'm sure we can solve this issue without bottleneck. I see many
technical issues with that approach.

Another bad thing about the approach is we don't know how to find test
result by configuration.

I think, if we will split tests it should be done manually: by marshaller,
enable/disable indexing, enabled/disabled near configuration and etc. It
will be really simple when I will finish with the test framework.

In my view, it's not really hard to run tests concurrently. My question is
in approach. May be we can another options or we cannot do it at all by any
reason.

-- Artem --

On Mon, Feb 8, 2016 at 6:38 PM, Alexey Kuznetsov <akuznetsov@gridgain.com>
wrote:

> How about such approach:
>
> Generate all permutation descriptors and save them to database.
> On each agent start some process that will grab from DB some descriptors
> (lets say 100) mark them as "in progress" and execute them.
> Other agents will grab remaining "not in progress" descriptors and do the
> same.
> So we could run in parallel all permutations even if we have 1M of them.
>
> What do you think?
>
>
> On Mon, Feb 8, 2016 at 9:52 PM, Sergey Kozlov <skozlov@gridgain.com>
> wrote:
>
> > 1000 caches x 50MB = 50GB heap. Do we really have >50GB RAM on each
> agents?
> >
> > On Mon, Feb 8, 2016 at 5:45 PM, Artem Shutak <ashutak@gridgain.com>
> wrote:
> >
> > > Sergey,
> > >
> > > I think we should start more caches, like 1000 in one time. But we have
> > to
> > > have enough memory on our TC agents. As I know, empty cache is require
> > > about 50 mb (without indexing), am I right?
> > >
> > > You are right, I keep in mind that *backups* and *REPLICATED* mode make
> > no
> > > sense together, but we still have to test it in one node and multi node
> > > cases—é
> > >
> > > Any other *no sense* combinations?
> > >
> > > I forgot about custom BinaryConfiguration at IgniteConfiguration for
> > > BinaryMarshaller. So, at least 6 IgniteConfigurations.
> > >
> > > -- Artem --
> > >
> > > On Mon, Feb 8, 2016 at 5:17 PM, Sergey Kozlov <skozlov@gridgain.com>
> > > wrote:
> > >
> > > > Hi Artem
> > > >
> > > > It's good idea to create 20-30 cache configurations at once and then
> to
> > > > iterate tests over those caches in parallel (but make sure that cache
> > > names
> > > > are unique).
> > > > Another point that some combinations make no sense like *backups *and
> > > > *REPLICATED
> > > > *cache
> > > >
> > > >
> > > > On Mon, Feb 8, 2016 at 5:07 PM, Artem Shutak <ashutak@gridgain.com>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have an update.
> > > > >
> > > > > I've started from *CacheConfiguration* permutations. I wrote out
a
> > list
> > > > > with all CacheConfiguration setters and filtered it with Alexey G.
> > > > >
> > > > > Finally we have:
> > > > >
> > > > >    1. CacheMode - 3 variants
> > > > >    2. CacheAtomicityMode - 2 variants
> > > > >    3. CacheMemoryMode - 3 variants
> > > > >    4. setLoadPreviousValue - 2 variants
> > > > >    5. setReadFromBackup - 2 variants
> > > > >    6. setStoreKeepBinary - 2 variants
> > > > >    7. setRebalanceMode - SYNC and ASYNC (2 variants)
> > > > >    8. setSwapEnabled - 2 variants
> > > > >    9. setCopyOnRead - 2 variants
> > > > >    10. NearConfiguration disabled / default NearConfiguration /
> > custom
> > > > >    NearConfiguration - 3 variants
> > > > >    11. With and without a complex parameter. The complex parameter
> > > > defines
> > > > >    not-default Eviction policy and filter, cache store
> configuration
> > > > >    (storeFactory and storeSessionListenerFactory), rebalancing
> > > > > configuration,
> > > > >    affinity function, offHeapMaxMemory, interceptor, topology
> > validator
> > > > and
> > > > >    CacheEntryListener.
> > > > >
> > > > > I've run 123 Cache Full Api test cases for all permutations of
> > > parameters
> > > > > 1-9 and got 256896 test cases (1152 configuration variants * 123
> test
> > > > > cases). All these tests take 4 hours 40 minutes. Not all tests
> pass,
> > so
> > > > MAY
> > > > > BE when all tests will pass it will take less time (3,5 hours for
> > > > example).
> > > > >
> > > > > As we can see the tests take a lot of time.
> > > > >
> > > > > The following permutation should be supported too:
> > > > >
> > > > >    1. Nodes count and Bakups count - 1 node and 0 backup, 3 nodes
> > and 1
> > > > >    backups, 4 nodes and 2 backups - 3 variants
> > > > >    2. Client and Server nodes - 2 variants
> > > > >    3. Indexing enabled and disabled for cache - 2 variants
> > > > >    4. IgniteConfiguration permutations - how many variants? I see
> at
> > > > least
> > > > >    4 (2 Marshallers, P2P).
> > > > >
> > > > > Plus we need add new test cases to test different key and value
> types
> > > and
> > > > > etc.
> > > > >
> > > > > So, we need multiply more then 3,5-4,5 hours on ~250. If we will
> > split
> > > > all
> > > > > tests on 250 suites and run on all 30 TC agents it will take about
> > > 30-40
> > > > > hours. Ok, we can do it during weekends.
> > > > >
> > > > > I think it will take too much time.
> > > > >
> > > > > As an option we can start a cache for each configuration and run
> > tests
> > > > > concurrently. But we need to implement this opportunity in our test
> > > > > framework.
> > > > >
> > > > > Any other thoughts how we can decrease time for tests?
> > > > >
> > > > > Thanks,
> > > > > -- Artem --
> > > > >
> > > > > On Thu, Feb 4, 2016 at 8:43 AM, Semyon Boikov <
> sboikov@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > Artem,
> > > > > >
> > > > > > One more thing for new tests: I think test should start both
> server
> > > and
> > > > > > client nodes and use Ignite API from all nodes.
> > > > > >
> > > > > > On Wed, Feb 3, 2016 at 6:40 PM, Artem Shutak <
> ashutak@gridgain.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Dmitriy,
> > > > > > >
> > > > > > > Actually, I don't have a list with all the permutations.
> > > > > > >
> > > > > > > At first, we need to split in our discussion test cases
and
> > Ignite
> > > > > > > configuration which should be covered.
> > > > > > >
> > > > > > > For example, new Full Api test cases for cache are based
on old
> > > Full
> > > > > Api
> > > > > > > test cases. So, it need to think what the test cases was
not
> > > covered
> > > > > > > before.
> > > > > > >
> > > > > > > About Ignite configurations, I'm going to add permutation
for
> > each
> > > > > > > IgniteConfiguration and CacheConfiguration property.
> > > > > > >
> > > > > > > By the way, the jira contains the following list of permutation
> > > (feel
> > > > > > free
> > > > > > > to add something):
> > > > > > >
> > > > > > > The following tests should be added (for functional blocks):
> > > > > > >
> > > > > > >    1. Interceptor
> > > > > > >    2. Queries: continuous, scan, SQL, fields and text queries.
> > > > > > >    3. cache events
> > > > > > >    4. We should also test with Serializable, Externalizable,
> and
> > > > plain
> > > > > > >    Pojos for keys and values.
> > > > > > >    5. The Pojo in the above test should contain an enum
value
> > > > > > >    6. We should also test Enums as keys and Enums as values
> > > > > > >    7. All operations should have single-key and multi-key
> > > operations
> > > > > > >
> > > > > > > New tests should cover all combinations for following
> properties:
> > > > > > >
> > > > > > >    1. cache modes
> > > > > > >    2. operation from client nodes and server nodes
> > > > > > >    3. store enabled/disabled
> > > > > > >    4. evicts sycn/non-sync
> > > > > > >    5. eviction policies
> > > > > > >    6. near on/off
> > > > > > >    7. marshallers (+ Binary marshaller with different mappers)
> > > > > > >    8. keys and values - externalizable, serializable,
> > binaryzable,
> > > > > "none
> > > > > > of
> > > > > > >    previous"
> > > > > > >    9. classes available on servers: true/false
> > > > > > >    10. Peer loading on/off
> > > > > > >    11. Affinity functions
> > > > > > >    12. expiry policies
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -- Artem --
> > > > > > >
> > > > > > > On Wed, Feb 3, 2016 at 6:14 PM, Dmitriy Setrakyan <
> > > > > dsetrakyan@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Artem, I think it is best to specify all the permutations
> here,
> > > so
> > > > > > others
> > > > > > > > can make additional suggestions. Otherwise, we cannot
get a
> > full
> > > > > > picture.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > D.
> > > > > > > >
> > > > > > > > On Wed, Feb 3, 2016 at 2:02 AM, Artem Shutak <
> > > ashutak@gridgain.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Igniters,
> > > > > > > > >
> > > > > > > > > I thought a little bit more and think we need
to add a
> > support
> > > > for
> > > > > > the
> > > > > > > > > following permutations too (I've added these
to the jira
> > > > > > description):
> > > > > > > > > - We should also test with Serializable, Externalizable,
> and
> > > > plain
> > > > > > > Pojos
> > > > > > > > > for keys and values.
> > > > > > > > > - The Pojo in the above test should contain an
enum value
> > > > > > > > > - We should also test Enums as keys and Enums
as values
> > > > > > > > > - All operations should have single-key and multi-key
> > > operations
> > > > > > > > >
> > > > > > > > > Maybe someone see any other permutation to be
supported?
> > > > > > > > >
> > > > > > > > > -- Artem --
> > > > > > > > >
> > > > > > > > > On Tue, Feb 2, 2016 at 10:05 PM, Artem Shutak
<
> > > > > ashutak@gridgain.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Dmitriy,
> > > > > > > > > >
> > > > > > > > > > There is a branch at my fork and a pull
request at
> Ignite.
> > > See
> > > > > > > comment
> > > > > > > > > > about pull request at the ticket (PR-446).
> > > > > > > > > >
> > > > > > > > > > But I have to notice that the branch under
hard
> development
> > > and
> > > > > you
> > > > > > > it
> > > > > > > > > can
> > > > > > > > > > not work (have compilation or test errors)
at some
> moments.
> > > > > > > > > >
> > > > > > > > > > Good luck!
> > > > > > > > > >
> > > > > > > > > > -- Artem --
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 2, 2016 at 9:45 PM, Dmitriy
Setrakyan <
> > > > > > > > dsetrakyan@apache.org
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Artem,
> > > > > > > > > >>
> > > > > > > > > >> This is great. I have noticed from the
ticket that you
> > have
> > > > > > created
> > > > > > > > some
> > > > > > > > > >> initial suite already. Is there a branch
I can look at
> it?
> > > > > > > > > >>
> > > > > > > > > >> D.
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Feb 2, 2016 at 10:02 AM, Artem
Shutak <
> > > > > > ashutak@gridgain.com
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Igniters,
> > > > > > > > > >> >
> > > > > > > > > >> > I'm working on an enhancement of
Full API coverage [1]
> > > [2].
> > > > > > > > > >> >
> > > > > > > > > >> > Ignite already has Full API test,
but currently it's
> > hard
> > > to
> > > > > > test
> > > > > > > > all
> > > > > > > > > >> > configuration combinations.
> > > > > > > > > >> >
> > > > > > > > > >> > Feel free to add comments in the
jira if you have any
> > > > thought.
> > > > > > > > > >> >
> > > > > > > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-2521
> > > > > > > > > >> > [2]
> > > > > > > > > >>
> > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Implementing+Tests
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > -- Artem --
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sergey Kozlov
> > > > GridGain Systems
> > > > www.gridgain.com
> > > >
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message