ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: Best Effort Affinity for thin clients
Date Mon, 04 Feb 2019 13:26:45 GMT
Igor,

My idea is simply to add the list of caches with the same distribution to
the end of partition response. Client can use this information to populate
partition info for more caches in a single request.

On Mon, Feb 4, 2019 at 3:06 PM Igor Sapego <isapego@apache.org> wrote:

> Vladimir,
>
> So correct me if I'm wrong, what you propose is to avoid mentioning
> of cache groups, and use instead of "cache group" term something like
> "distribution"? Or do you propose some changes in protocol? If so, can
> you briefly explain, what kind of changes they are?
>
> Best Regards,
> Igor
>
>
> On Mon, Feb 4, 2019 at 1:13 PM Vladimir Ozerov <vozerov@gridgain.com>
> wrote:
>
> > Igor,
> >
> > Yes, cache groups are public API. However, we try to avoid new APIs
> > depending on them.
> > The main point from my side is that “similar cache group” can be easily
> > generalized to “similar distribution”. This way we avoid cache groups on
> > protocol level at virtually no cost.
> >
> > Vladimir.
> >
> > пн, 4 февр. 2019 г. в 12:48, Igor Sapego <isapego@apache.org>:
> >
> > > Guys,
> > >
> > > Can you explain why do we want to avoid Cache groups in protocol?
> > >
> > > If it's about simplicity of the protocol, then removing cache groups
> will
> > > not help much with it - we will still need to include "knownCacheIds"
> > > field in request and "cachesWithTheSamePartitioning" field in response.
> > > And also, since when do Ignite prefers simplicity over performance?
> > >
> > > If it's about not wanting to show internals of Ignite then it sounds
> like
> > > a very weak argument to me, since Cache Groups is a public thing [1].
> > >
> > > [1] - https://apacheignite.readme.io/docs/cache-groups
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Mon, Feb 4, 2019 at 11:47 AM Vladimir Ozerov <vozerov@gridgain.com>
> > > wrote:
> > >
> > > > Pavel, Igor,
> > > >
> > > > This is not very accurate to say that this will not save memory. In
> > > > practice we observed a number of OOME issues on the server-side due
> to
> > > many
> > > > caches and it was one of motivations for cache groups (another one
> disk
> > > > access optimizations). On the other hand, I agree that we'd better to
> > > avoid
> > > > cache groups in the protocol because this is internal implementation
> > > detail
> > > > which is likely (I hope so) to be changed in future.
> > > >
> > > > So I have another proposal - let's track caches with the same
> affinity
> > > > distribution instead. That is, normally most of PARTITIONED caches
> will
> > > > have very few variants of configuration: it will be Rendezvous
> affinity
> > > > function, most likely with default partition number and with 1-2
> > backups
> > > at
> > > > most. So when affinity distribution for specific cache is requested,
> we
> > > can
> > > > append to the response *list of caches with the same distribution*.
> > I.e.:
> > > >
> > > > class AffinityResponse {
> > > >     Object distribution;    // Actual distribution
> > > >     List<Integer> cacheIds; // Caches with similar distribution
> > > > }
> > > >
> > > > Makes sense?
> > > >
> > > > On Sun, Feb 3, 2019 at 8:31 PM Pavel Tupitsyn <ptupitsyn@apache.org>
> > > > wrote:
> > > >
> > > > > Igor, I have a feeling that we should omit Cache Group stuff from
> the
> > > > > protocol.
> > > > > It is a rare use case and even then dealing with them on client
> > barely
> > > > > saves some memory.
> > > > >
> > > > > We can keep it simple and have partition map per cacheId. Thoughts?
> > > > >
> > > > > On Fri, Feb 1, 2019 at 6:49 PM Igor Sapego <isapego@apache.org>
> > wrote:
> > > > >
> > > > > > Guys, I've updated the proposal once again [1], so please,
> > > > > > take a look and let me know what you think.
> > > > > >
> > > > > > [1] -
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > > > > >
> > > > > > Best Regards,
> > > > > > Igor
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 17, 2019 at 1:05 PM Igor Sapego <isapego@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > Yeah, I'll add it.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Igor
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jan 16, 2019 at 11:08 PM Pavel Tupitsyn <
> > > > ptupitsyn@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> >  to every server
> > > > > > >> I did not think of this issue. Now I agree with your
approach.
> > > > > > >> Can you please add an explanation of this to the IEP?
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >> On Wed, Jan 16, 2019 at 2:53 PM Igor Sapego <
> isapego@apache.org
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >> > Pavel,
> > > > > > >> >
> > > > > > >> > Yeah, it makes sense, but to me it seems that
this approach
> > can
> > > > lead
> > > > > > >> > to more complicated client logic, as it will require
to make
> > > > > > additional
> > > > > > >> > call
> > > > > > >> > to every server, that reports affinity topology
change.
> > > > > > >> >
> > > > > > >> > Guys, WDYT?
> > > > > > >> >
> > > > > > >> > Best Regards,
> > > > > > >> > Igor
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, Jan 15, 2019 at 10:59 PM Pavel Tupitsyn
<
> > > > > ptupitsyn@apache.org
> > > > > > >
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Igor,
> > > > > > >> > >
> > > > > > >> > > >  It is proposed to add flag to every
response, that
> shows
> > > > > whether
> > > > > > >> the
> > > > > > >> > > Affinity Topology Version of the cluster
has changed since
> > the
> > > > > last
> > > > > > >> > request
> > > > > > >> > > from the client.
> > > > > > >> > > I propose to keep this flag. So no need for
periodic
> checks.
> > > > Makes
> > > > > > >> sense?
> > > > > > >> > >
> > > > > > >> > > On Tue, Jan 15, 2019 at 4:45 PM Igor Sapego
<
> > > isapego@apache.org
> > > > >
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > > Pavel,
> > > > > > >> > > >
> > > > > > >> > > > This will require from client to send
this new request
> > > > > > periodically,
> > > > > > >> > I'm
> > > > > > >> > > > not
> > > > > > >> > > > sure this will make clients simpler.
Anyway, let's
> discuss
> > > it.
> > > > > > >> > > >
> > > > > > >> > > > Vladimir,
> > > > > > >> > > >
> > > > > > >> > > > With current proposal, we will have
affinity info in
> > message
> > > > > > header.
> > > > > > >> > > >
> > > > > > >> > > > Best Regards,
> > > > > > >> > > > Igor
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Jan 15, 2019 at 11:01 AM Vladimir
Ozerov <
> > > > > > >> vozerov@gridgain.com
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Igor,
> > > > > > >> > > > >
> > > > > > >> > > > > I think that "Cache Partitions
Request" should contain
> > > > > affinity
> > > > > > >> > > topology
> > > > > > >> > > > > version. Otherwise we do not know
what distribution is
> > > > > returned
> > > > > > -
> > > > > > >> the
> > > > > > >> > > one
> > > > > > >> > > > > we expected, or some newer one.
The latter may happen
> in
> > > > case
> > > > > > >> > topology
> > > > > > >> > > > > changed or late affinity assignment
happened between
> > > server
> > > > > > >> response
> > > > > > >> > > and
> > > > > > >> > > > > subsequent client partitions request.
> > > > > > >> > > > >
> > > > > > >> > > > > Vladimir.
> > > > > > >> > > > >
> > > > > > >> > > > > On Mon, Jan 14, 2019 at 6:08 PM
Igor Sapego <
> > > > > isapego@apache.org
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hello guys,
> > > > > > >> > > > > >
> > > > > > >> > > > > > I've updated IEP page [1]
describing proposed
> solution
> > > in
> > > > > more
> > > > > > >> > > details
> > > > > > >> > > > > and
> > > > > > >> > > > > > proposing some changes for
a protocol.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Please, take a look and let
me know what you think.
> > > > > > >> > > > > >
> > > > > > >> > > > > > [1] -
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> > > > > > >> > > > > >
> > > > > > >> > > > > > Best Regards,
> > > > > > >> > > > > > Igor
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Tue, Jun 19, 2018 at 11:54
AM Vladimir Ozerov <
> > > > > > >> > > vozerov@gridgain.com
> > > > > > >> > > > >
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Denis,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Yes, in principle we
can extend it. We are going
> to
> > > > > > implement
> > > > > > >> it
> > > > > > >> > in
> > > > > > >> > > > > > > subsequent phases of
this IEP.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Tue, Jun 19, 2018
at 4:30 AM, Dmitriy
> Setrakyan <
> > > > > > >> > > > > > dsetrakyan@apache.org>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > On Mon, Jun 18,
2018 at 11:07 AM, Denis Magda <
> > > > > > >> > dmagda@apache.org
> > > > > > >> > > >
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Folks,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Feel that this
functionality can be extended
> to
> > > the
> > > > > > >> automatic
> > > > > > >> > > > > > > reconnect,
> > > > > > >> > > > > > > > > can't it? Presently
we require to provide a
> > static
> > > > > list
> > > > > > of
> > > > > > >> > IPs
> > > > > > >> > > to
> > > > > > >> > > > > be
> > > > > > >> > > > > > > used
> > > > > > >> > > > > > > > > at a reconnect
time. By having a partition map
> > of
> > > > all
> > > > > > the
> > > > > > >> > > nodes,
> > > > > > >> > > > > the
> > > > > > >> > > > > > > thin
> > > > > > >> > > > > > > > > client should
be able to automate this piece.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Not sure if static
IP list can be avoided. What
> > Igor
> > > > is
> > > > > > >> > > suggesting
> > > > > > >> > > > is
> > > > > > >> > > > > > > that
> > > > > > >> > > > > > > > we try to pick the
best node out of the static
> IP
> > > > list.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > D.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message