ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: Stable binary key representation
Date Mon, 10 Apr 2017 18:50:24 GMT
Considering this simple example

INSERT (id, orgId, name, age, address) into Person…

where id and orgId define Person’s affinity key - PersonKey(id, orgId)

How do we know which fields to use for hash code generation and equality comparison? QueryEntity?


No, it’s unclear how to document it properly.

—
Denis

> On Apr 10, 2017, at 11:14 AM, Vladimir Ozerov <vozerov@gridgain.com> wrote:
> 
> There is no more such resolver. It was removed.
> 
> On Mon, Apr 10, 2017 at 8:58 PM, Denis Magda <dmagda@apache.org> wrote:
> 
>> Vovan,
>> 
>> Before I fix the documentation, what’t the replacement for
>> BinaryFieldIdentiyResolver we used to define field for hash code
>> calculation and equality comparison when DML statements are used?
>> https://apacheignite.readme.io/docs/binary-marshaller#
>> section-binary-field-identity-resolver <https://apacheignite.readme.
>> io/docs/binary-marshaller#section-binary-field-identity-resolver>
>> 
>> —
>> Denis
>> 
>>> On Apr 9, 2017, at 7:39 AM, Vladimir Ozerov <vozerov@gridgain.com>
>> wrote:
>>> 
>>> Resolvers were essential for DML because we had broken comparison
>> semantics
>>> of binary objects. This is not the case now.
>>> 
>>> Resolver as a whole is normal practice. E.g. it is implemented in .NET on
>>> core language level and widely used in many cases. Hazelcast has it as
>> well
>>> AFAIK. So it is wrong to think that the whole idea is useless. Think of
>> it
>>> as a comparator's brother.
>>> 
>>> The only reason why we need to remove it is missing hash index in new
>>> architecture. It makes sense, as it is better to have AI 2.0 without
>> them,
>>> than no AI 2.0 :-)
>>> 
>>> 09 апр. 2017 г. 17:31 пользователь "Sergi Vladykin" <
>>> sergi.vladykin@gmail.com> написал:
>>> 
>>>> I guess Resolvers were added to DML just because they already existed
>> since
>>>> 1.9 and we were forced to support them in all the parts of our product.
>>>> 
>>>> We have to stop this practice to add features without clear real life
>> use
>>>> cases for them.
>>>> 
>>>> Sergi
>>>> 
>>>> 2017-04-09 17:00 GMT+03:00 Denis Magda <dmagda@gridgain.com>:
>>>> 
>>>>> Sergi, Vovan,
>>>>> 
>>>>> Sorry for being annoying but I still didn't get an answer on whether
>> the
>>>>> resolvers are the must for DML. The main reason why we made them up
>> some
>>>>> time ago is to support specific DML use cases. However I can't recall
>> the
>>>>> use cases.
>>>>> 
>>>>> --
>>>>> Denis
>>>>> 
>>>>> On Sun, Apr 9, 2017 at 6:54 AM, Sergi Vladykin <
>> sergi.vladykin@gmail.com
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Ok, we need to do 2 things here:
>>>>>> 
>>>>>> 1. Drop the resolvers from the source code.
>>>>>> 2. Write a good page in docs on "What makes a correct cache key".
>>>>>> 
>>>>>> Who can do that?
>>>>>> 
>>>>>> Sergi
>>>>>> 
>>>>>> 2017-04-07 9:48 GMT+03:00 Sergi Vladykin <sergi.vladykin@gmail.com>:
>>>>>> 
>>>>>>> It is possible to try adding support of comparison to Resolvers,
but
>>>>> the
>>>>>>> whole approach looks wrong and for now it is better to get rid
of it
>>>>>> while
>>>>>>> we have a chance to break compatibility.
>>>>>>> 
>>>>>>> Sergi
>>>>>>> 
>>>>>>> 2017-04-07 9:19 GMT+03:00 Valentin Kulichenko <
>>>>>>> valentin.kulichenko@gmail.com>:
>>>>>>> 
>>>>>>>> The discussion should've been started with that :) If supporting
>>>>>> resolvers
>>>>>>>> in new architecture is not possible or means too big effort,
then
>>>> it's
>>>>>>>> definitely not worth it.
>>>>>>>> 
>>>>>>>> -Val
>>>>>>>> 
>>>>>>>> On Thu, Apr 6, 2017 at 8:52 PM, Vladimir Ozerov <
>>>> vozerov@gridgain.com
>>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Dima,
>>>>>>>>> 
>>>>>>>>> Yes, they may explode some internals of our indexes.
>>>>>>>>> 
>>>>>>>>> 06 апр. 2017 г. 23:32 пользователь "Dmitriy
Setrakyan" <
>>>>>>>>> dsetrakyan@apache.org> написал:
>>>>>>>>> 
>>>>>>>>>> Guys,
>>>>>>>>>> 
>>>>>>>>>> Isn't the main issue here that we cannot use the
Identity
>>>>> Resolvers
>>>>>> in
>>>>>>>>>> BTrees in the 2.0 version? If yes, then we have to
remove them
>>>> no
>>>>>>>> matter
>>>>>>>>>> what.
>>>>>>>>>> 
>>>>>>>>>> D.
>>>>>>>>>> 
>>>>>>>>>> On Thu, Apr 6, 2017 at 1:23 PM, Sergi Vladykin <
>>>>>>>> sergi.vladykin@gmail.com
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Binary key representation is stable when we always
have equal
>>>>>>>>> serialized
>>>>>>>>>>> bytes when the original keys are equal.
>>>>>>>>>>> 
>>>>>>>>>>> Resolver allows you to have some extra info in
the Key and
>>>> equal
>>>>>>>> Keys
>>>>>>>>>> will
>>>>>>>>>>> be serialized into different bytes, which is
wrong.
>>>>>>>>>>> 
>>>>>>>>>>> Look at the example what you can do with resolvers:
>>>>>>>>>>> 
>>>>>>>>>>> We may have some data entry with fields a, b,
c. Let's say the
>>>>>>>> unique
>>>>>>>>>> part
>>>>>>>>>>> here is `a` and it the only fields used in Key
equals() and
>>>>>>>> hashCode().
>>>>>>>>>>> Still we may have the following layouts:
>>>>>>>>>>> 
>>>>>>>>>>> 1. Ka -> Vbc
>>>>>>>>>>> 2. Kab -> Vc
>>>>>>>>>>> 3. Kabc -> Boolean.TRUE
>>>>>>>>>>> 
>>>>>>>>>>> The only 1 is a correct layout, others are plain
wrong
>>>> variants
>>>>>> (but
>>>>>>>>> they
>>>>>>>>>>> are still possible with Resolvers) because everything
that
>>>> does
>>>>>> not
>>>>>>>>> make
>>>>>>>>>>> Key unique must be in Value.
>>>>>>>>>>> 
>>>>>>>>>>> We want to clearly state that if you have something
in Key,
>>>> that
>>>>>> is
>>>>>>>> not
>>>>>>>>>>> part of equals(), then the Key is invalid and
that stuff must
>>>> be
>>>>>> in
>>>>>>>>>> Value.
>>>>>>>>>>> This allows us to rely on binary representation
of a Key to be
>>>>>>>> stable
>>>>>>>>> and
>>>>>>>>>>> have some more optimizations and code simplifications
with
>>>>> respect
>>>>>>>> to
>>>>>>>>>> these
>>>>>>>>>>> assumptions.
>>>>>>>>>>> 
>>>>>>>>>>> Sergi
>>>>>>>>>>> 
>>>>>>>>>>> 2017-04-06 14:24 GMT+03:00 Valentin Kulichenko
<
>>>>>>>>>>> valentin.kulichenko@gmail.com>:
>>>>>>>>>>> 
>>>>>>>>>>>> Even with my vast expirience I would never
claim that I've
>>>>> seen
>>>>>>>>>>>> "everything" :)
>>>>>>>>>>>> 
>>>>>>>>>>>> What do you mean by stable binary key representation
and how
>>>>>>>>> resolvers
>>>>>>>>>>> make
>>>>>>>>>>>> it unstable?
>>>>>>>>>>>> 
>>>>>>>>>>>> -Val
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Apr 6, 2017 at 2:36 AM, Sergi Vladykin
<
>>>>>>>>>> sergi.vladykin@gmail.com
>>>>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Val,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I know that you have really vast experience
in Ignite
>>>>>>>> deployments
>>>>>>>>> and
>>>>>>>>>>>>> probably saw everything that can happen.
Did you ever see
>>>>>>>> identity
>>>>>>>>>>>>> resolvers use in real life? I guess no.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hibernate example is bad here, because
if their key is
>>>>>> unstable
>>>>>>>>>> across
>>>>>>>>>>>>> multiple JVMs, it means that it was not
designed for
>>>>>> distributed
>>>>>>>>>>> caches a
>>>>>>>>>>>>> priori.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Also knowing in advance about stable
binary key
>>>>> representation
>>>>>>>>> allows
>>>>>>>>>>> us
>>>>>>>>>>>> to
>>>>>>>>>>>>> apply additional optimizations, like
comparing keys
>>>> without
>>>>>>>>> detaching
>>>>>>>>>>>> them
>>>>>>>>>>>>> from offheap memory.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We always will be able to add this stuff
back if we see
>>>>> users
>>>>>>>>> really
>>>>>>>>>>> need
>>>>>>>>>>>>> it. Let's remove it for 2.0.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Sergi
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2017-04-06 11:21 GMT+03:00 Valentin Kulichenko
<
>>>>>>>>>>>>> valentin.kulichenko@gmail.com>:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alex,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> To be honest, I don't understand
the reasoning behind
>>>> the
>>>>>>>>> removal.
>>>>>>>>>> I
>>>>>>>>>>>>> think
>>>>>>>>>>>>>> resolvers provide good flexibility
for different corner
>>>>>> cases
>>>>>>>> and
>>>>>>>>>>> it's
>>>>>>>>>>>> a
>>>>>>>>>>>>>> good thing to have them. Note that
they can be applied
>>>> not
>>>>>>>> only
>>>>>>>>> to
>>>>>>>>>>>> cache
>>>>>>>>>>>>>> keys, but to any binary objects.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hibernate issue is actually a good
example of such use
>>>>> case.
>>>>>>>> The
>>>>>>>>>> fact
>>>>>>>>>>>>> that
>>>>>>>>>>>>>> we found an alternative solution
doesn't actually mean
>>>>>>>> anything,
>>>>>>>>>>>> because
>>>>>>>>>>>>>> what if this happened not in our
module, but in user's
>>>>>>>>> application?
>>>>>>>>>>>>>> Unfortunately, we can't predict everything.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Error proneness is not a very strong
argument either,
>>>>>> because
>>>>>>>> in
>>>>>>>>> my
>>>>>>>>>>>> view
>>>>>>>>>>>>>> these resolvers are as much error
prone as
>>>> BinaryIdMapper,
>>>>>> for
>>>>>>>>>>> example.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -Val
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Wed, Apr 5, 2017 at 11:44 PM,
Alexey Goncharuk <
>>>>>>>>>>>>>> alexey.goncharuk@gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Denis,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Can you suggest a use-case where
identity resolver is
>>>>>> needed
>>>>>>>>>> (given
>>>>>>>>>>>>> that
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> agree that a key must contain
only valuable fields)?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2017-04-05 22:08 GMT+03:00 Denis
Magda <
>>>>> dmagda@apache.org
>>>>>>> :
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Where do you want to remove
the identity resolvers
>>>>> from?
>>>>>>>> If
>>>>>>>>>> it’s
>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>> to the internals of Hibernate
module then it’s fine
>>>>> but
>>>>>> if
>>>>>>>>> you
>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>> removing identity resolvers
public interfaces then
>>>> it
>>>>>>>> might
>>>>>>>>> be
>>>>>>>>>> a
>>>>>>>>>>>>> haste
>>>>>>>>>>>>>>>> decision.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> —
>>>>>>>>>>>>>>>> Denis
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Apr 5, 2017, at 7:42
AM, Alexey Goncharuk <
>>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> +1, I see no other reasons
to keep it.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 2017-04-05 13:59 GMT+03:00
Sergi Vladykin <
>>>>>>>>>>>>> sergi.vladykin@gmail.com
>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Lets drop them.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Sergi
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 2017-04-05 13:50
GMT+03:00 Dmitriy Govorukhin <
>>>>>>>>>>>>>>>>>> dmitriy.govorukhin@gmail.com>
>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi guys, i implemented
proxy for IgniteCache in
>>>>>>>> hibernate
>>>>>>>>>>>>>>> integration,
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> proxy transformate
cacheKey to our key wrapper,
>>>>>> leaves
>>>>>>>>> only
>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>>>>>> field. I think
we can remove identity resolve,
>>>> it
>>>>>>>> should
>>>>>>>>>> not
>>>>>>>>>>>>> broke
>>>>>>>>>>>>>>>>>>> integration with
hibernate. Any objections?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, Mar 29,
2017 at 11:07 PM, Valentin
>>>>>> Kulichenko
>>>>>>>> <
>>>>>>>>>>>>>>>>>>> valentin.kulichenko@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I'm not saying
there is no alternative
>>>> solution.
>>>>>> But
>>>>>>>>> let's
>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> prove that
it works first, and remove resolvers
>>>>>> only
>>>>>>>>> after
>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> -Val
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Wed, Mar
29, 2017 at 12:18 PM, Sergi
>>>> Vladykin
>>>>> <
>>>>>>>>>>>>>>>>>>> sergi.vladykin@gmail.com
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Guys,
nothing is impossible if you know a bit
>>>>>> about
>>>>>>>>>>>> reflection
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> Java
>>>>>>>>>>>>>>>>>>> :)
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> We had
a look at the CacheKey class and it is
>>>>>> easily
>>>>>>>>>>>>> replaceable.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Sergi
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 2017-03-29
21:49 GMT+03:00 Dmitriy Setrakyan <
>>>>>>>>>>>>>>> dsetrakyan@apache.org
>>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On
Wed, Mar 29, 2017 at 11:43 AM, Valentin
>>>>>>>> Kulichenko
>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>> valentin.kulichenko@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>
"Hibernate key" is the CacheKey class I was
>>>>>>>> referring
>>>>>>>>>> to.
>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>> provided
>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>
Hibernate, not by user and not by us. So I'm
>>>>> not
>>>>>>>> sure
>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>
replace it.
>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> If
it is impossible to replace or get rid of
>>>>> the
>>>>>>>>>> Hibernate
>>>>>>>>>>>>> key,
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> discussion
valid at all?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> 


Mime
View raw message