ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Custom Affinity Functions proposed for removal?
Date Wed, 04 Nov 2020 10:43:40 GMT
Could you give an example of such mapping?

If that’s possible, it might also be very helpful to see the implementation
of your mapper. Looking at the code is often the best way to understand a
use case :)

-Val

On Wed, Nov 4, 2020 at 12:29 PM Raymond Wilson <raymond_wilson@trimble.com>
wrote:

> Actually, it's worse than that...
>
> We have more than one key -> partition mapping for the same key (part of a
> CQRS pattern we use).
>
> Aren't key affinity functions essentially an API in any event?
>
> Cheers,
> Raymond.
>
>
> On Wed, Nov 4, 2020 at 9:54 PM Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Thanks, Raymond. So the reason why you couldn't use the
> @AffinityKeyMapped
> > annotation or the CacheKeyConfiguration is that collocation is based on
> > *two fields*, not just one field. Is my understanding correct?
> >
> > If that's the case, I believe it can be easily improved by providing ways
> > to specify multiple fields for collocation. The fact that this scenario
> > requires a custom affinity mapper is a usability issue in my view.
> >
> > -Val
> >
> > On Tue, Nov 3, 2020 at 10:26 AM Raymond Wilson <
> raymond_wilson@trimble.com
> > >
> > wrote:
> >
> > > If I have a primary key vector like this <x><y><t>, where
<x> & <y> can
> > > have varying values for <t>, and I want all keys with the same <x>
and
> > <y>
> > > to reside in the same partition for processing colocation requirements
> > then
> > > I can't use the standard Ignite mapping to do this, I need to use a
> > custom
> > > mapper than uses just the <x> & <y> components.
> > >
> > >
> > > On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Raymond,
> > > >
> > > > Thanks for the details. So it sounds like you have a custom affinity
> > > > mapper, not affinity function. This makes things simpler, but I'm
> still
> > > > failing to understand why standard mechanisms for collocation [1]
> > didn't
> > > > work for you. Could you please clarify?
> > > >
> > > > [1]
> > > >
> > https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
> > > >
> > > > -Val
> > > >
> > > > On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <
> > > raymond_wilson@trimble.com
> > > > >
> > > > wrote:
> > > >
> > > > > In terms of Key -> Partition -> Node mapping, we provide customer
> > > > affinity
> > > > > mappers for the Key -> Partition stage an allow Ignite to map
> > > partitions
> > > > to
> > > > > nodes.
> > > > >
> > > > > Our keys are structs with multiple fields forming a composite
> primary
> > > > key,
> > > > > parts of which are spatially identifying and parts contain other
> > > > > characteristics. We want to ensure all related spatial elements
> > resolve
> > > > to
> > > > > the same partition and so will be placed on the same node for query
> > > > > colocation purposes.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
> > > > > valentin.kulichenko@gmail.com> wrote:
> > > > >
> > > > > > Moti, Raymond,
> > > > > >
> > > > > > Could you please describe your use cases in more detail? What
are
> > the
> > > > > types
> > > > > > used as cache keys? What is the custom logic that you use for
> > > affinity
> > > > > > mapping? What was the exact reason to customize versus using
> > built-in
> > > > > > collocation mechanisms?
> > > > > >
> > > > > > Ultimately, I'm sure that custom affinity functions will go
away
> > > cause
> > > > > they
> > > > > > introduce multiple potential issues. However, I would definitely
> > like
> > > > to
> > > > > > make sure that your use cases are still supported in Ignite
3.0
> via
> > > > some
> > > > > > other means.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> > > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > > >
> > > > > > > Thanks for the clarification.
> > > > > > >
> > > > > > > There was no intention to remove the customizable key to
> > partition
> > > > > > mapping.
> > > > > > >
> > > > > > > Difficulties arise when mapping partitions to nodes, so
it's
> > > > desirable
> > > > > to
> > > > > > > have internally tested implementation with a way to customize
> > it's
> > > > > > behavior
> > > > > > > without additional coding on the user side.
> > > > > > >
> > > > > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
> > > > > raymond_wilson@trimble.com
> > > > > > >:
> > > > > > >
> > > > > > > > Just to be clear, the affinity functions we are using
convert
> > > keys
> > > > to
> > > > > > > > partitions, we do not map partitions to nodes and
leave that
> to
> > > > > Ignite.
> > > > > > > >
> > > > > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > > > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hello.
> > > > > > > > >
> > > > > > > > > Custom affinity functions can cause weird bugs
and data
> loss
> > if
> > > > > > > > implemented
> > > > > > > > > wrongly.
> > > > > > > > > There is an intention  to keep a backup filter
based on
> user
> > > > > > attributes
> > > > > > > > > (with additional validation logic to ensure correctness)
> for
> > > > > > > controllable
> > > > > > > > > data placement.
> > > > > > > > >
> > > > > > > > > Can you describe more precisely why you had to
implement
> > custom
> > > > > > > affinity
> > > > > > > > > functions and not resort to default rendezvous
affinity +
> > > backup
> > > > > > > filter ?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond
Wilson <
> > > > > > > raymond_wilson@trimble.com
> > > > > > > > >:
> > > > > > > > >
> > > > > > > > > > We also use custom affinity functions (vis
the C#
> client).
> > > > > > > > > >
> > > > > > > > > > The wish list mentions use of a particular
annotation
> > > > > > > > > > (@CentralizedAffinityFunction):
> > > > > > > > > > Is the wish to remove just this annotation,
or the
> ability
> > to
> > > > > > define
> > > > > > > > > custom
> > > > > > > > > > affinity functions at all?
> > > > > > > > > >
> > > > > > > > > > In our case we use affinity functions to
ensure
> particular
> > > > > > > distribution
> > > > > > > > > of
> > > > > > > > > > spatial data across a processing cluster
to ensure even
> > load
> > > > > > > > management.
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson
<
> > > > > > > moti.nisenson@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I saw at
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > > > > > > that custom affinity functions are
on the potential
> > > wishlist
> > > > > for
> > > > > > > > > removal.
> > > > > > > > > > > The way we're using it's very critical
that we be able
> to
> > > > > control
> > > > > > > the
> > > > > > > > > > > placement of data quite precisely -
as part of that we
> > > > specify
> > > > > > > > > explicitly
> > > > > > > > > > > the partition we want in the key, and
then our affinity
> > > > > function
> > > > > > > uses
> > > > > > > > > > that
> > > > > > > > > > > (else delegating to default rendezvous).
We don't need
> > all
> > > > the
> > > > > > > > > > > abilities there, although I think that
often others do.
> > > > > > > > > > >
> > > > > > > > > > > This seems to me to be a case that
the benefit of
> > removing
> > > > this
> > > > > > is
> > > > > > > > > > minimal
> > > > > > > > > > > and could cause quite a lot of disruption
to users.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > <http://www.trimble.com/>
> > > > > > > > > > Raymond Wilson
> > > > > > > > > > Solution Architect, Civil Construction Software
Systems
> > > (CCSS)
> > > > > > > > > > 11 Birmingham Drive | Christchurch, New
Zealand
> > > > > > > > > > +64-21-2013317 Mobile
> > > > > > > > > > raymond_wilson@trimble.com
> > > > > > > > > >
> > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Alexei Scherbakov
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > <http://www.trimble.com/>
> > > > > > > > Raymond Wilson
> > > > > > > > Solution Architect, Civil Construction Software Systems
> (CCSS)
> > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > > > +64-21-2013317 Mobile
> > > > > > > > raymond_wilson@trimble.com
> > > > > > > >
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Alexei Scherbakov
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > <http://www.trimble.com/>
> > > > > Raymond Wilson
> > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > +64-21-2013317 Mobile
> > > > > raymond_wilson@trimble.com
> > > > >
> > > > > <
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > <http://www.trimble.com/>
> > > Raymond Wilson
> > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > 11 Birmingham Drive | Christchurch, New Zealand
> > > +64-21-2013317 Mobile
> > > raymond_wilson@trimble.com
> > >
> > > <
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > >
> > >
> >
>
>
> --
> <http://www.trimble.com/>
> Raymond Wilson
> Solution Architect, Civil Construction Software Systems (CCSS)
> 11 Birmingham Drive | Christchurch, New Zealand
> +64-21-2013317 Mobile
> raymond_wilson@trimble.com
>
> <
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message