ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Custom Affinity Functions proposed for removal?
Date Wed, 04 Nov 2020 09:04:34 GMT
I've created a ticket for this:
https://issues.apache.org/jira/browse/IGNITE-13671

-Val

On Wed, Nov 4, 2020 at 12:53 AM Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:

> Thanks, Raymond. So the reason why you couldn't use the @AffinityKeyMapped
> annotation or the CacheKeyConfiguration is that collocation is based on
> *two fields*, not just one field. Is my understanding correct?
>
> If that's the case, I believe it can be easily improved by providing ways
> to specify multiple fields for collocation. The fact that this scenario
> requires a custom affinity mapper is a usability issue in my view.
>
> -Val
>
> On Tue, Nov 3, 2020 at 10:26 AM Raymond Wilson <raymond_wilson@trimble.com>
> wrote:
>
>> If I have a primary key vector like this <x><y><t>, where <x>
& <y> can
>> have varying values for <t>, and I want all keys with the same <x> and
<y>
>> to reside in the same partition for processing colocation requirements
>> then
>> I can't use the standard Ignite mapping to do this, I need to use a custom
>> mapper than uses just the <x> & <y> components.
>>
>>
>> On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko <
>> valentin.kulichenko@gmail.com> wrote:
>>
>> > Raymond,
>> >
>> > Thanks for the details. So it sounds like you have a custom affinity
>> > mapper, not affinity function. This makes things simpler, but I'm still
>> > failing to understand why standard mechanisms for collocation [1] didn't
>> > work for you. Could you please clarify?
>> >
>> > [1]
>> >
>> https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation
>> >
>> > -Val
>> >
>> > On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson <
>> raymond_wilson@trimble.com
>> > >
>> > wrote:
>> >
>> > > In terms of Key -> Partition -> Node mapping, we provide customer
>> > affinity
>> > > mappers for the Key -> Partition stage an allow Ignite to map
>> partitions
>> > to
>> > > nodes.
>> > >
>> > > Our keys are structs with multiple fields forming a composite primary
>> > key,
>> > > parts of which are spatially identifying and parts contain other
>> > > characteristics. We want to ensure all related spatial elements
>> resolve
>> > to
>> > > the same partition and so will be placed on the same node for query
>> > > colocation purposes.
>> > >
>> > >
>> > >
>> > > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko <
>> > > valentin.kulichenko@gmail.com> wrote:
>> > >
>> > > > Moti, Raymond,
>> > > >
>> > > > Could you please describe your use cases in more detail? What are
>> the
>> > > types
>> > > > used as cache keys? What is the custom logic that you use for
>> affinity
>> > > > mapping? What was the exact reason to customize versus using
>> built-in
>> > > > collocation mechanisms?
>> > > >
>> > > > Ultimately, I'm sure that custom affinity functions will go away
>> cause
>> > > they
>> > > > introduce multiple potential issues. However, I would definitely
>> like
>> > to
>> > > > make sure that your use cases are still supported in Ignite 3.0 via
>> > some
>> > > > other means.
>> > > >
>> > > > -Val
>> > > >
>> > > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov <
>> > > > alexey.scherbakoff@gmail.com> wrote:
>> > > >
>> > > > > Thanks for the clarification.
>> > > > >
>> > > > > There was no intention to remove the customizable key to partition
>> > > > mapping.
>> > > > >
>> > > > > Difficulties arise when mapping partitions to nodes, so it's
>> > desirable
>> > > to
>> > > > > have internally tested implementation with a way to customize
it's
>> > > > behavior
>> > > > > without additional coding on the user side.
>> > > > >
>> > > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson <
>> > > raymond_wilson@trimble.com
>> > > > >:
>> > > > >
>> > > > > > Just to be clear, the affinity functions we are using convert
>> keys
>> > to
>> > > > > > partitions, we do not map partitions to nodes and leave
that to
>> > > Ignite.
>> > > > > >
>> > > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov <
>> > > > > > alexey.scherbakoff@gmail.com> wrote:
>> > > > > >
>> > > > > > > Hello.
>> > > > > > >
>> > > > > > > Custom affinity functions can cause weird bugs and
data loss
>> if
>> > > > > > implemented
>> > > > > > > wrongly.
>> > > > > > > There is an intention  to keep a backup filter based
on user
>> > > > attributes
>> > > > > > > (with additional validation logic to ensure correctness)
for
>> > > > > controllable
>> > > > > > > data placement.
>> > > > > > >
>> > > > > > > Can you describe more precisely why you had to implement
>> custom
>> > > > > affinity
>> > > > > > > functions and not resort to default rendezvous affinity
+
>> backup
>> > > > > filter ?
>> > > > > > >
>> > > > > > >
>> > > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson
<
>> > > > > raymond_wilson@trimble.com
>> > > > > > >:
>> > > > > > >
>> > > > > > > > We also use custom affinity functions (vis the
C# client).
>> > > > > > > >
>> > > > > > > > The wish list mentions use of a particular annotation
>> > > > > > > > (@CentralizedAffinityFunction):
>> > > > > > > > Is the wish to remove just this annotation, or
the ability
>> to
>> > > > define
>> > > > > > > custom
>> > > > > > > > affinity functions at all?
>> > > > > > > >
>> > > > > > > > In our case we use affinity functions to ensure
particular
>> > > > > distribution
>> > > > > > > of
>> > > > > > > > spatial data across a processing cluster to ensure
even load
>> > > > > > management.
>> > > > > > > >
>> > > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson <
>> > > > > moti.nisenson@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I saw at
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
>> > > > > > > > > that custom affinity functions are on the
potential
>> wishlist
>> > > for
>> > > > > > > removal.
>> > > > > > > > > The way we're using it's very critical that
we be able to
>> > > control
>> > > > > the
>> > > > > > > > > placement of data quite precisely - as part
of that we
>> > specify
>> > > > > > > explicitly
>> > > > > > > > > the partition we want in the key, and then
our affinity
>> > > function
>> > > > > uses
>> > > > > > > > that
>> > > > > > > > > (else delegating to default rendezvous).
We don't need all
>> > the
>> > > > > > > > > abilities there, although I think that often
others do.
>> > > > > > > > >
>> > > > > > > > > This seems to me to be a case that the benefit
of removing
>> > this
>> > > > is
>> > > > > > > > minimal
>> > > > > > > > > and could cause quite a lot of disruption
to users.
>> > > > > > > > >
>> > > > > > > > > Thanks!
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > <http://www.trimble.com/>
>> > > > > > > > Raymond Wilson
>> > > > > > > > Solution Architect, Civil Construction Software
Systems
>> (CCSS)
>> > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand
>> > > > > > > > +64-21-2013317 Mobile
>> > > > > > > > raymond_wilson@trimble.com
>> > > > > > > >
>> > > > > > > > <
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > >
>> > > > > > > Best regards,
>> > > > > > > Alexei Scherbakov
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > <http://www.trimble.com/>
>> > > > > > Raymond Wilson
>> > > > > > Solution Architect, Civil Construction Software Systems
(CCSS)
>> > > > > > 11 Birmingham Drive | Christchurch, New Zealand
>> > > > > > +64-21-2013317 Mobile
>> > > > > > raymond_wilson@trimble.com
>> > > > > >
>> > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Best regards,
>> > > > > Alexei Scherbakov
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > <http://www.trimble.com/>
>> > > Raymond Wilson
>> > > Solution Architect, Civil Construction Software Systems (CCSS)
>> > > 11 Birmingham Drive | Christchurch, New Zealand
>> > > +64-21-2013317 Mobile
>> > > raymond_wilson@trimble.com
>> > >
>> > > <
>> > >
>> >
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
>> > > >
>> > >
>> >
>>
>>
>> --
>> <http://www.trimble.com/>
>> Raymond Wilson
>> Solution Architect, Civil Construction Software Systems (CCSS)
>> 11 Birmingham Drive | Christchurch, New Zealand
>> +64-21-2013317 Mobile
>> raymond_wilson@trimble.com
>>
>> <
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message