ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Wilson <raymond_wil...@trimble.com>
Subject Re: Custom Affinity Functions proposed for removal?
Date Tue, 03 Nov 2020 08:25:27 GMT
In terms of Key -> Partition -> Node mapping, we provide customer affinity
mappers for the Key -> Partition stage an allow Ignite to map partitions to
nodes.

Our keys are structs with multiple fields forming a composite primary key,
parts of which are spatially identifying and parts contain other
characteristics. We want to ensure all related spatial elements resolve to
the same partition and so will be placed on the same node for query
colocation purposes.



On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:

> Moti, Raymond,
>
> Could you please describe your use cases in more detail? What are the types
> used as cache keys? What is the custom logic that you use for affinity
> mapping? What was the exact reason to customize versus using built-in
> collocation mechanisms?
>
> Ultimately, I'm sure that custom affinity functions will go away cause they
> introduce multiple potential issues. However, I would definitely like to
> make sure that your use cases are still supported in Ignite 3.0 via some
> other means.
>
> -Val
>
> On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
> > Thanks for the clarification.
> >
> > There was no intention to remove the customizable key to partition
> mapping.
> >
> > Difficulties arise when mapping partitions to nodes, so it's desirable to
> > have internally tested implementation with a way to customize it's
> behavior
> > without additional coding on the user side.
> >
> > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <raymond_wilson@trimble.com
> >:
> >
> > > Just to be clear, the affinity functions we are using convert keys to
> > > partitions, we do not map partitions to nodes and leave that to Ignite.
> > >
> > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
> > > alexey.scherbakoff@gmail.com> wrote:
> > >
> > > > Hello.
> > > >
> > > > Custom affinity functions can cause weird bugs and data loss if
> > > implemented
> > > > wrongly.
> > > > There is an intention  to keep a backup filter based on user
> attributes
> > > > (with additional validation logic to ensure correctness) for
> > controllable
> > > > data placement.
> > > >
> > > > Can you describe more precisely why you had to implement custom
> > affinity
> > > > functions and not resort to default rendezvous affinity + backup
> > filter ?
> > > >
> > > >
> > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson <
> > raymond_wilson@trimble.com
> > > >:
> > > >
> > > > > We also use custom affinity functions (vis the C# client).
> > > > >
> > > > > The wish list mentions use of a particular annotation
> > > > > (@CentralizedAffinityFunction):
> > > > > Is the wish to remove just this annotation, or the ability to
> define
> > > > custom
> > > > > affinity functions at all?
> > > > >
> > > > > In our case we use affinity functions to ensure particular
> > distribution
> > > > of
> > > > > spatial data across a processing cluster to ensure even load
> > > management.
> > > > >
> > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
> > moti.nisenson@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I saw at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
> > > > > > that custom affinity functions are on the potential wishlist
for
> > > > removal.
> > > > > > The way we're using it's very critical that we be able to control
> > the
> > > > > > placement of data quite precisely - as part of that we specify
> > > > explicitly
> > > > > > the partition we want in the key, and then our affinity function
> > uses
> > > > > that
> > > > > > (else delegating to default rendezvous). We don't need all the
> > > > > > abilities there, although I think that often others do.
> > > > > >
> > > > > > This seems to me to be a case that the benefit of removing this
> is
> > > > > minimal
> > > > > > and could cause quite a lot of disruption to users.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > <http://www.trimble.com/>
> > > > > Raymond Wilson
> > > > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > > > 11 Birmingham Drive | Christchurch, New Zealand
> > > > > +64-21-2013317 Mobile
> > > > > raymond_wilson@trimble.com
> > > > >
> > > > > <
> > > > >
> > > >
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Alexei Scherbakov
> > > >
> > >
> > >
> > > --
> > > <http://www.trimble.com/>
> > > Raymond Wilson
> > > Solution Architect, Civil Construction Software Systems (CCSS)
> > > 11 Birmingham Drive | Christchurch, New Zealand
> > > +64-21-2013317 Mobile
> > > raymond_wilson@trimble.com
> > >
> > > <
> > >
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>


-- 
<http://www.trimble.com/>
Raymond Wilson
Solution Architect, Civil Construction Software Systems (CCSS)
11 Birmingham Drive | Christchurch, New Zealand
+64-21-2013317 Mobile
raymond_wilson@trimble.com

<https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message