ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: IgniteSet implementation: changes required
Date Tue, 27 Feb 2018 08:20:40 GMT
I think the root issue is that we are trying to mix different cases in a
single solution. What is the common usage patterns of sets?
1) Small mostly-read sets - current implementation is ideal for them -
everything is available locally, on-heap and in deserialized form
2) Big data sets - IgniteCache API is the best candidate here; we can
simply add a method "Set<K> IgniteCache.asSet()" which will return thin
wrapper for Set interface around cache. An you will have recovery and
heap-resistance with almost no additional efforts.
There is no need to choose between one or another solution. We can support
both.

As far as current igniteSet implementation - all issues described above
will go away if you use approach suggested by Dmitry. I do not see a need
for any major changes.

Vladimir.

On Mon, Feb 19, 2018 at 10:44 AM, Pavel Pereslegin <xxtern@gmail.com> wrote:

> Hello Vladimir,
>
> > What we can do is to optionally disable on-heap caching for specific set
> at the cost of lower performance if user wants so.
>
> I want to make sure that we are speaking about the same thing. By
> "on-heap caching" I mean custom datastructure and not standard on-heap
> cache, we can't disable it now, because it is the part of current
> IgniteSet implementation. Is it acceptable to use GridOffHeapMap
> optionally instead of ConcurrentHashMap for "off-heap mode"?
>
>
> 2018-02-15 23:20 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org>:
> > On Thu, Feb 15, 2018 at 6:08 AM, Vladimir Ozerov <vozerov@gridgain.com>
> > wrote:
> >
> >> I do not think indexes is the right approach - set do not have indexes,
> and
> >> you will have to maintain additional counter for it in order to know
> when
> >> to stop.
> >>
> >> From what I see there are two distinct problems:
> >> 1) Broken recovery - this is just a bug which needs to be fixed. As
> soon as
> >> data is stored in real persistent cache, recovery of data structure
> state
> >> should be trivial task.
> >> 2) Heap items - this should not be a problem in common case when set
> >> contains moderate number of elements. If set is excessively large, then
> >> this is not the right structure for your use case and you should use
> >> standard IgniteCache API instead. What we can do is to optionally
> disable
> >> on-heap caching for specific set at the cost of lower performance if
> user
> >> wants so.
> >>
> >
> > Vladimir, I am not sure I agree. In my view, set should be similar to
> > cache, just a different API. I am not sure why we should make an
> > assumptions that set data should be smaller than cache, especially given
> > that it is a trivial task to implement a set based on Ignite cache API
> (we
> > could just store key-key mappings in cache instead of key-value mappings
> > internally).
> >
> > Can you clarify why you believe that IgniteSet should need to have
> on-heap
> > entries?
> >
> > D.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message