ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Stable binary key representation
Date Mon, 10 Apr 2017 17:58:55 GMT
Vovan,

Before I fix the documentation, what’t the replacement for BinaryFieldIdentiyResolver we
used to define field for hash code calculation and equality comparison when DML statements
are used?
https://apacheignite.readme.io/docs/binary-marshaller#section-binary-field-identity-resolver
<https://apacheignite.readme.io/docs/binary-marshaller#section-binary-field-identity-resolver>

—
Denis

> On Apr 9, 2017, at 7:39 AM, Vladimir Ozerov <vozerov@gridgain.com> wrote:
> 
> Resolvers were essential for DML because we had broken comparison semantics
> of binary objects. This is not the case now.
> 
> Resolver as a whole is normal practice. E.g. it is implemented in .NET on
> core language level and widely used in many cases. Hazelcast has it as well
> AFAIK. So it is wrong to think that the whole idea is useless. Think of it
> as a comparator's brother.
> 
> The only reason why we need to remove it is missing hash index in new
> architecture. It makes sense, as it is better to have AI 2.0 without them,
> than no AI 2.0 :-)
> 
> 09 апр. 2017 г. 17:31 пользователь "Sergi Vladykin" <
> sergi.vladykin@gmail.com> написал:
> 
>> I guess Resolvers were added to DML just because they already existed since
>> 1.9 and we were forced to support them in all the parts of our product.
>> 
>> We have to stop this practice to add features without clear real life use
>> cases for them.
>> 
>> Sergi
>> 
>> 2017-04-09 17:00 GMT+03:00 Denis Magda <dmagda@gridgain.com>:
>> 
>>> Sergi, Vovan,
>>> 
>>> Sorry for being annoying but I still didn't get an answer on whether the
>>> resolvers are the must for DML. The main reason why we made them up some
>>> time ago is to support specific DML use cases. However I can't recall the
>>> use cases.
>>> 
>>> --
>>> Denis
>>> 
>>> On Sun, Apr 9, 2017 at 6:54 AM, Sergi Vladykin <sergi.vladykin@gmail.com
>>> 
>>> wrote:
>>> 
>>>> Ok, we need to do 2 things here:
>>>> 
>>>> 1. Drop the resolvers from the source code.
>>>> 2. Write a good page in docs on "What makes a correct cache key".
>>>> 
>>>> Who can do that?
>>>> 
>>>> Sergi
>>>> 
>>>> 2017-04-07 9:48 GMT+03:00 Sergi Vladykin <sergi.vladykin@gmail.com>:
>>>> 
>>>>> It is possible to try adding support of comparison to Resolvers, but
>>> the
>>>>> whole approach looks wrong and for now it is better to get rid of it
>>>> while
>>>>> we have a chance to break compatibility.
>>>>> 
>>>>> Sergi
>>>>> 
>>>>> 2017-04-07 9:19 GMT+03:00 Valentin Kulichenko <
>>>>> valentin.kulichenko@gmail.com>:
>>>>> 
>>>>>> The discussion should've been started with that :) If supporting
>>>> resolvers
>>>>>> in new architecture is not possible or means too big effort, then
>> it's
>>>>>> definitely not worth it.
>>>>>> 
>>>>>> -Val
>>>>>> 
>>>>>> On Thu, Apr 6, 2017 at 8:52 PM, Vladimir Ozerov <
>> vozerov@gridgain.com
>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Dima,
>>>>>>> 
>>>>>>> Yes, they may explode some internals of our indexes.
>>>>>>> 
>>>>>>> 06 апр. 2017 г. 23:32 пользователь "Dmitriy Setrakyan"
<
>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>> 
>>>>>>>> Guys,
>>>>>>>> 
>>>>>>>> Isn't the main issue here that we cannot use the Identity
>>> Resolvers
>>>> in
>>>>>>>> BTrees in the 2.0 version? If yes, then we have to remove
them
>> no
>>>>>> matter
>>>>>>>> what.
>>>>>>>> 
>>>>>>>> D.
>>>>>>>> 
>>>>>>>> On Thu, Apr 6, 2017 at 1:23 PM, Sergi Vladykin <
>>>>>> sergi.vladykin@gmail.com
>>>>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Binary key representation is stable when we always have
equal
>>>>>>> serialized
>>>>>>>>> bytes when the original keys are equal.
>>>>>>>>> 
>>>>>>>>> Resolver allows you to have some extra info in the Key
and
>> equal
>>>>>> Keys
>>>>>>>> will
>>>>>>>>> be serialized into different bytes, which is wrong.
>>>>>>>>> 
>>>>>>>>> Look at the example what you can do with resolvers:
>>>>>>>>> 
>>>>>>>>> We may have some data entry with fields a, b, c. Let's
say the
>>>>>> unique
>>>>>>>> part
>>>>>>>>> here is `a` and it the only fields used in Key equals()
and
>>>>>> hashCode().
>>>>>>>>> Still we may have the following layouts:
>>>>>>>>> 
>>>>>>>>> 1. Ka -> Vbc
>>>>>>>>> 2. Kab -> Vc
>>>>>>>>> 3. Kabc -> Boolean.TRUE
>>>>>>>>> 
>>>>>>>>> The only 1 is a correct layout, others are plain wrong
>> variants
>>>> (but
>>>>>>> they
>>>>>>>>> are still possible with Resolvers) because everything
that
>> does
>>>> not
>>>>>>> make
>>>>>>>>> Key unique must be in Value.
>>>>>>>>> 
>>>>>>>>> We want to clearly state that if you have something in
Key,
>> that
>>>> is
>>>>>> not
>>>>>>>>> part of equals(), then the Key is invalid and that stuff
must
>> be
>>>> in
>>>>>>>> Value.
>>>>>>>>> This allows us to rely on binary representation of a
Key to be
>>>>>> stable
>>>>>>> and
>>>>>>>>> have some more optimizations and code simplifications
with
>>> respect
>>>>>> to
>>>>>>>> these
>>>>>>>>> assumptions.
>>>>>>>>> 
>>>>>>>>> Sergi
>>>>>>>>> 
>>>>>>>>> 2017-04-06 14:24 GMT+03:00 Valentin Kulichenko <
>>>>>>>>> valentin.kulichenko@gmail.com>:
>>>>>>>>> 
>>>>>>>>>> Even with my vast expirience I would never claim
that I've
>>> seen
>>>>>>>>>> "everything" :)
>>>>>>>>>> 
>>>>>>>>>> What do you mean by stable binary key representation
and how
>>>>>>> resolvers
>>>>>>>>> make
>>>>>>>>>> it unstable?
>>>>>>>>>> 
>>>>>>>>>> -Val
>>>>>>>>>> 
>>>>>>>>>> On Thu, Apr 6, 2017 at 2:36 AM, Sergi Vladykin <
>>>>>>>> sergi.vladykin@gmail.com
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Val,
>>>>>>>>>>> 
>>>>>>>>>>> I know that you have really vast experience in
Ignite
>>>>>> deployments
>>>>>>> and
>>>>>>>>>>> probably saw everything that can happen. Did
you ever see
>>>>>> identity
>>>>>>>>>>> resolvers use in real life? I guess no.
>>>>>>>>>>> 
>>>>>>>>>>> Hibernate example is bad here, because if their
key is
>>>> unstable
>>>>>>>> across
>>>>>>>>>>> multiple JVMs, it means that it was not designed
for
>>>> distributed
>>>>>>>>> caches a
>>>>>>>>>>> priori.
>>>>>>>>>>> 
>>>>>>>>>>> Also knowing in advance about stable binary key
>>> representation
>>>>>>> allows
>>>>>>>>> us
>>>>>>>>>> to
>>>>>>>>>>> apply additional optimizations, like comparing
keys
>> without
>>>>>>> detaching
>>>>>>>>>> them
>>>>>>>>>>> from offheap memory.
>>>>>>>>>>> 
>>>>>>>>>>> We always will be able to add this stuff back
if we see
>>> users
>>>>>>> really
>>>>>>>>> need
>>>>>>>>>>> it. Let's remove it for 2.0.
>>>>>>>>>>> 
>>>>>>>>>>> Sergi
>>>>>>>>>>> 
>>>>>>>>>>> 2017-04-06 11:21 GMT+03:00 Valentin Kulichenko
<
>>>>>>>>>>> valentin.kulichenko@gmail.com>:
>>>>>>>>>>> 
>>>>>>>>>>>> Alex,
>>>>>>>>>>>> 
>>>>>>>>>>>> To be honest, I don't understand the reasoning
behind
>> the
>>>>>>> removal.
>>>>>>>> I
>>>>>>>>>>> think
>>>>>>>>>>>> resolvers provide good flexibility for different
corner
>>>> cases
>>>>>> and
>>>>>>>>> it's
>>>>>>>>>> a
>>>>>>>>>>>> good thing to have them. Note that they can
be applied
>> not
>>>>>> only
>>>>>>> to
>>>>>>>>>> cache
>>>>>>>>>>>> keys, but to any binary objects.
>>>>>>>>>>>> 
>>>>>>>>>>>> Hibernate issue is actually a good example
of such use
>>> case.
>>>>>> The
>>>>>>>> fact
>>>>>>>>>>> that
>>>>>>>>>>>> we found an alternative solution doesn't
actually mean
>>>>>> anything,
>>>>>>>>>> because
>>>>>>>>>>>> what if this happened not in our module,
but in user's
>>>>>>> application?
>>>>>>>>>>>> Unfortunately, we can't predict everything.
>>>>>>>>>>>> 
>>>>>>>>>>>> Error proneness is not a very strong argument
either,
>>>> because
>>>>>> in
>>>>>>> my
>>>>>>>>>> view
>>>>>>>>>>>> these resolvers are as much error prone as
>> BinaryIdMapper,
>>>> for
>>>>>>>>> example.
>>>>>>>>>>>> 
>>>>>>>>>>>> -Val
>>>>>>>>>>>> 
>>>>>>>>>>>> On Wed, Apr 5, 2017 at 11:44 PM, Alexey Goncharuk
<
>>>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Denis,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Can you suggest a use-case where identity
resolver is
>>>> needed
>>>>>>>> (given
>>>>>>>>>>> that
>>>>>>>>>>>> we
>>>>>>>>>>>>> agree that a key must contain only valuable
fields)?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2017-04-05 22:08 GMT+03:00 Denis Magda
<
>>> dmagda@apache.org
>>>>> :
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Where do you want to remove the identity
resolvers
>>> from?
>>>>>> If
>>>>>>>> it’s
>>>>>>>>>>>> related
>>>>>>>>>>>>>> to the internals of Hibernate module
then it’s fine
>>> but
>>>> if
>>>>>>> you
>>>>>>>>>>> suggest
>>>>>>>>>>>>>> removing identity resolvers public
interfaces then
>> it
>>>>>> might
>>>>>>> be
>>>>>>>> a
>>>>>>>>>>> haste
>>>>>>>>>>>>>> decision.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> —
>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Apr 5, 2017, at 7:42 AM, Alexey
Goncharuk <
>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> +1, I see no other reasons to
keep it.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2017-04-05 13:59 GMT+03:00 Sergi
Vladykin <
>>>>>>>>>>> sergi.vladykin@gmail.com
>>>>>>>>>>>>> :
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Lets drop them.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Sergi
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2017-04-05 13:50 GMT+03:00
Dmitriy Govorukhin <
>>>>>>>>>>>>>>>> dmitriy.govorukhin@gmail.com>
>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi guys, i implemented
proxy for IgniteCache in
>>>>>> hibernate
>>>>>>>>>>>>> integration,
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> proxy transformate cacheKey
to our key wrapper,
>>>> leaves
>>>>>>> only
>>>>>>>>>>>> required
>>>>>>>>>>>>>>>>> field. I think we can
remove identity resolve,
>> it
>>>>>> should
>>>>>>>> not
>>>>>>>>>>> broke
>>>>>>>>>>>>>>>>> integration with hibernate.
Any objections?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Mar 29, 2017
at 11:07 PM, Valentin
>>>> Kulichenko
>>>>>> <
>>>>>>>>>>>>>>>>> valentin.kulichenko@gmail.com>
wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I'm not saying there
is no alternative
>> solution.
>>>> But
>>>>>>> let's
>>>>>>>>>>>> implement
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> prove that it works
first, and remove resolvers
>>>> only
>>>>>>> after
>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> -Val
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Wed, Mar 29, 2017
at 12:18 PM, Sergi
>> Vladykin
>>> <
>>>>>>>>>>>>>>>>> sergi.vladykin@gmail.com
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Guys, nothing
is impossible if you know a bit
>>>> about
>>>>>>>>>> reflection
>>>>>>>>>>> in
>>>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> We had a look
at the CacheKey class and it is
>>>> easily
>>>>>>>>>>> replaceable.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Sergi
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 2017-03-29 21:49
GMT+03:00 Dmitriy Setrakyan <
>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Wed, Mar
29, 2017 at 11:43 AM, Valentin
>>>>>> Kulichenko
>>>>>>> <
>>>>>>>>>>>>>>>>>>>> valentin.kulichenko@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> "Hibernate
key" is the CacheKey class I was
>>>>>> referring
>>>>>>>> to.
>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>> provided
>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> Hibernate,
not by user and not by us. So I'm
>>> not
>>>>>> sure
>>>>>>>>> it's
>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> replace
it.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> If it is
impossible to replace or get rid of
>>> the
>>>>>>>> Hibernate
>>>>>>>>>>> key,
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> discussion
valid at all?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message