ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Wilson <raymond_wil...@trimble.com>
Subject Re: Custom Affinity Functions proposed for removal?
Date Tue, 03 Nov 2020 18:26:07 GMT
If I have a primary key vector like this <x><y><t>, where <x> &
<y> can
have varying values for <t>, and I want all keys with the same <x> and <y>
to reside in the same partition for processing colocation requirements then
I can't use the standard Ignite mapping to do this, I need to use a custom
mapper than uses just the <x> & <y> components.


On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:

> Raymond,
>
> Thanks for the details. So it sounds like you have a custom affinity
> mapper, not affinity function. This makes things simpler, but I'm still
> failing to understand why standard mechanisms for collocation [1] didn't
> work for you. Could you please clarify?
>
> [1]
> https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
>
> -Val
>
> On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <raymond_wilson@trimble.com
> >
> wrote:
>
> > In terms of Key -> Partition -> Node mapping, we provide customer
> affinity
> > mappers for the Key -> Partition stage an allow Ignite to map partitions
> to
> > nodes.
> >
> > Our keys are structs with multiple fields forming a composite primary
> key,
> > parts of which are spatially identifying and parts contain other
> > characteristics. We want to ensure all related spatial elements resolve
> to
> > the same partition and so will be placed on the same node for query
> > colocation purposes.
> >
> >
> >
> > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Moti, Raymond,
> > >
> > > Could you please describe your use cases in more detail? What are the
> > types
> > > used as cache keys? What is the custom logic that you use for affinity
> > > mapping? What was the exact reason to customize versus using built-in
> > > collocation mechanisms?
> > >
> > > Ultimately, I'm sure that custom affinity functions will go away cause
> > they
> > > introduce multiple potential issues. However, I would definitely like
> to
> > > make sure that your use cases are still supported in Ignite 3.0 via
> some
> > > other means.
> > >
> > > -Val
> > >
> > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> > > alexey.scherbakoff@gmail.com> wrote:
> > >
> > > > Thanks for the clarification.
> > > >
> > > > There was no intention to remove the customizable key to partition
> > > mapping.
> > > >
> > > > Difficulties arise when mapping partitions to nodes, so it's
> desirable
> > to
> > > > have internally tested implementation with a way to customize it's
> > > behavior
> > > > without additional coding on the user side.
> > > >
> > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
> > raymond_wilson@trimble.com
> > > >:
> > > >
> > > > > Just to be clear, the affinity functions we are using convert keys
> to
> > > > > partitions, we do not map partitions to nodes and leave that to
> > Ignite.
> > > > >
> > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > > > alexey.scherbakoff@gmail.com> wrote:
> > > > >
> > > > > > Hello.
> > > > > >
> > > > > > Custom affinity functions can cause weird bugs and data loss
if
> > > > > implemented
> > > > > > wrongly.
> > > > > > There is an intention  to keep a backup filter based on user
> > > attributes
> > > > > > (with additional validation logic to ensure correctness) for
> > > > controllable
> > > > > > data placement.
> > > > > >
> > > > > > Can you describe more precisely why you had to implement custom
> > > > affinity
> > > > > > functions and not resort to default rendezvous affinity + backup
> > > > filter ?
> > > > > >
> > > > > >
> > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> > > > raymond_wilson@trimble.com
> > > > > >:
> > > > > >
> > > > > > > We also use custom affinity functions (vis the C# client).
> > > > > > >
> > > > > > > The wish list mentions use of a particular annotation
> > > > > > > (@CentralizedAffinityFunction):
> > > > > > > Is the wish to remove just this annotation, or the ability
to
> > > define
> > > > > > custom
> > > > > > > affinity functions at all?
> > > > > > >
> > > > > > > In our case we use affinity functions to ensure particular
> > > > distribution
> > > > > > of
> > > > > > > spatial data across a processing cluster to ensure even
load
> > > > > management.
> > > > > > >
> > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> > > > moti.nisenson@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I saw at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > > > that custom affinity functions are on the potential
wishlist
> > for
> > > > > > removal.
> > > > > > > > The way we're using it's very critical that we be
able to
> > control
> > > > the
> > > > > > > > placement of data quite precisely - as part of that
we
> specify
> > > > > > explicitly
> > > > > > > > the partition we want in the key, and then our affinity
> > function
> > > > uses
> > > > > > > that
> > > > > > > > (else delegating to default rendezvous). We don't
need all
> the
> > > > > > > > abilities there, although I think that often others
do.
> > > > > > > >
> > > > > > > > This seems to me to be a case that the benefit of
removing
> this
> > > is
> > > > > > > minimal
> > > > > > > > and could cause quite a lot of disruption to users.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > <http://www.trimble.com/>
> > > > > > > Raymond Wilson
> > > > > > > Solution Architect, Civil Construction Software Systems
(CCSS)
> > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > > > +64-21-2013317 Mobile
> > > > > > > raymond_wilson@trimble.com
> > > > > > >
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best regards,
> > > > > > Alexei Scherbakov
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > <http://www.trimble.com/>
> > > > > Raymond Wilson
> > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > +64-21-2013317 Mobile
> > > > > raymond_wilson@trimble.com
> > > > >
> > > > > <
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Alexei Scherbakov
> > > >
> > >
> >
> >
> > --
> > <http://www.trimble.com/>
> > Raymond Wilson
> > Solution Architect, Civil Construction Software Systems (CCSS)
> > 11 Birmingham Drive | Christchurch, New Zealand
> > +64-21-2013317 Mobile
> > raymond_wilson@trimble.com
> >
> > <
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > >
> >
>


-- 
<http://www.trimble.com/>
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
+64-21-2013317 Mobile
raymond_wilson@trimble.com

<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message