ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Stable binary key representation
Date Tue, 11 Apr 2017 18:50:29 GMT
Denis, I think it is important that we know which specific field to use for
the affinity resolution, but I don't see any issue in using both, primary
and foreign keys, for hashcode and equality. Do you?

D.

On Tue, Apr 11, 2017 at 11:46 AM, Igor Sapego <isapego@apache.org> wrote:

> Denis,
>
> The whole binary representation of the object is used now
> for hash code generation and equality comparison. So the
> answer - all fields are used for this.
>
> Best Regards,
> Igor
>
> On Mon, Apr 10, 2017 at 9:50 PM, Denis Magda <dmagda@apache.org> wrote:
>
> > Considering this simple example
> >
> > INSERT (id, orgId, name, age, address) into Person…
> >
> > where id and orgId define Person’s affinity key - PersonKey(id, orgId)
> >
> > How do we know which fields to use for hash code generation and equality
> > comparison? QueryEntity?
> >
> > No, it’s unclear how to document it properly.
> >
> > —
> > Denis
> >
> > > On Apr 10, 2017, at 11:14 AM, Vladimir Ozerov <vozerov@gridgain.com>
> > wrote:
> > >
> > > There is no more such resolver. It was removed.
> > >
> > > On Mon, Apr 10, 2017 at 8:58 PM, Denis Magda <dmagda@apache.org>
> wrote:
> > >
> > >> Vovan,
> > >>
> > >> Before I fix the documentation, what’t the replacement for
> > >> BinaryFieldIdentiyResolver we used to define field for hash code
> > >> calculation and equality comparison when DML statements are used?
> > >> https://apacheignite.readme.io/docs/binary-marshaller#
> > >> section-binary-field-identity-resolver <https://apacheignite.readme.
> > >> io/docs/binary-marshaller#section-binary-field-identity-resolver>
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Apr 9, 2017, at 7:39 AM, Vladimir Ozerov <vozerov@gridgain.com>
> > >> wrote:
> > >>>
> > >>> Resolvers were essential for DML because we had broken comparison
> > >> semantics
> > >>> of binary objects. This is not the case now.
> > >>>
> > >>> Resolver as a whole is normal practice. E.g. it is implemented in
> .NET
> > on
> > >>> core language level and widely used in many cases. Hazelcast has it
> as
> > >> well
> > >>> AFAIK. So it is wrong to think that the whole idea is useless. Think
> of
> > >> it
> > >>> as a comparator's brother.
> > >>>
> > >>> The only reason why we need to remove it is missing hash index in new
> > >>> architecture. It makes sense, as it is better to have AI 2.0 without
> > >> them,
> > >>> than no AI 2.0 :-)
> > >>>
> > >>> 09 апр. 2017 г. 17:31 пользователь "Sergi Vladykin"
<
> > >>> sergi.vladykin@gmail.com> написал:
> > >>>
> > >>>> I guess Resolvers were added to DML just because they already
> existed
> > >> since
> > >>>> 1.9 and we were forced to support them in all the parts of our
> > product.
> > >>>>
> > >>>> We have to stop this practice to add features without clear real
> life
> > >> use
> > >>>> cases for them.
> > >>>>
> > >>>> Sergi
> > >>>>
> > >>>> 2017-04-09 17:00 GMT+03:00 Denis Magda <dmagda@gridgain.com>:
> > >>>>
> > >>>>> Sergi, Vovan,
> > >>>>>
> > >>>>> Sorry for being annoying but I still didn't get an answer on
> whether
> > >> the
> > >>>>> resolvers are the must for DML. The main reason why we made
them up
> > >> some
> > >>>>> time ago is to support specific DML use cases. However I can't
> recall
> > >> the
> > >>>>> use cases.
> > >>>>>
> > >>>>> --
> > >>>>> Denis
> > >>>>>
> > >>>>> On Sun, Apr 9, 2017 at 6:54 AM, Sergi Vladykin <
> > >> sergi.vladykin@gmail.com
> > >>>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Ok, we need to do 2 things here:
> > >>>>>>
> > >>>>>> 1. Drop the resolvers from the source code.
> > >>>>>> 2. Write a good page in docs on "What makes a correct cache
key".
> > >>>>>>
> > >>>>>> Who can do that?
> > >>>>>>
> > >>>>>> Sergi
> > >>>>>>
> > >>>>>> 2017-04-07 9:48 GMT+03:00 Sergi Vladykin <
> sergi.vladykin@gmail.com
> > >:
> > >>>>>>
> > >>>>>>> It is possible to try adding support of comparison
to Resolvers,
> > but
> > >>>>> the
> > >>>>>>> whole approach looks wrong and for now it is better
to get rid of
> > it
> > >>>>>> while
> > >>>>>>> we have a chance to break compatibility.
> > >>>>>>>
> > >>>>>>> Sergi
> > >>>>>>>
> > >>>>>>> 2017-04-07 9:19 GMT+03:00 Valentin Kulichenko <
> > >>>>>>> valentin.kulichenko@gmail.com>:
> > >>>>>>>
> > >>>>>>>> The discussion should've been started with that
:) If supporting
> > >>>>>> resolvers
> > >>>>>>>> in new architecture is not possible or means too
big effort,
> then
> > >>>> it's
> > >>>>>>>> definitely not worth it.
> > >>>>>>>>
> > >>>>>>>> -Val
> > >>>>>>>>
> > >>>>>>>> On Thu, Apr 6, 2017 at 8:52 PM, Vladimir Ozerov
<
> > >>>> vozerov@gridgain.com
> > >>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Dima,
> > >>>>>>>>>
> > >>>>>>>>> Yes, they may explode some internals of our
indexes.
> > >>>>>>>>>
> > >>>>>>>>> 06 апр. 2017 г. 23:32 пользователь
"Dmitriy Setrakyan" <
> > >>>>>>>>> dsetrakyan@apache.org> написал:
> > >>>>>>>>>
> > >>>>>>>>>> Guys,
> > >>>>>>>>>>
> > >>>>>>>>>> Isn't the main issue here that we cannot
use the Identity
> > >>>>> Resolvers
> > >>>>>> in
> > >>>>>>>>>> BTrees in the 2.0 version? If yes, then
we have to remove them
> > >>>> no
> > >>>>>>>> matter
> > >>>>>>>>>> what.
> > >>>>>>>>>>
> > >>>>>>>>>> D.
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Apr 6, 2017 at 1:23 PM, Sergi Vladykin
<
> > >>>>>>>> sergi.vladykin@gmail.com
> > >>>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Binary key representation is stable
when we always have equal
> > >>>>>>>>> serialized
> > >>>>>>>>>>> bytes when the original keys are equal.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Resolver allows you to have some extra
info in the Key and
> > >>>> equal
> > >>>>>>>> Keys
> > >>>>>>>>>> will
> > >>>>>>>>>>> be serialized into different bytes,
which is wrong.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Look at the example what you can do
with resolvers:
> > >>>>>>>>>>>
> > >>>>>>>>>>> We may have some data entry with fields
a, b, c. Let's say
> the
> > >>>>>>>> unique
> > >>>>>>>>>> part
> > >>>>>>>>>>> here is `a` and it the only fields
used in Key equals() and
> > >>>>>>>> hashCode().
> > >>>>>>>>>>> Still we may have the following layouts:
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. Ka -> Vbc
> > >>>>>>>>>>> 2. Kab -> Vc
> > >>>>>>>>>>> 3. Kabc -> Boolean.TRUE
> > >>>>>>>>>>>
> > >>>>>>>>>>> The only 1 is a correct layout, others
are plain wrong
> > >>>> variants
> > >>>>>> (but
> > >>>>>>>>> they
> > >>>>>>>>>>> are still possible with Resolvers)
because everything that
> > >>>> does
> > >>>>>> not
> > >>>>>>>>> make
> > >>>>>>>>>>> Key unique must be in Value.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We want to clearly state that if you
have something in Key,
> > >>>> that
> > >>>>>> is
> > >>>>>>>> not
> > >>>>>>>>>>> part of equals(), then the Key is invalid
and that stuff must
> > >>>> be
> > >>>>>> in
> > >>>>>>>>>> Value.
> > >>>>>>>>>>> This allows us to rely on binary representation
of a Key to
> be
> > >>>>>>>> stable
> > >>>>>>>>> and
> > >>>>>>>>>>> have some more optimizations and code
simplifications with
> > >>>>> respect
> > >>>>>>>> to
> > >>>>>>>>>> these
> > >>>>>>>>>>> assumptions.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Sergi
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2017-04-06 14:24 GMT+03:00 Valentin
Kulichenko <
> > >>>>>>>>>>> valentin.kulichenko@gmail.com>:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Even with my vast expirience I
would never claim that I've
> > >>>>> seen
> > >>>>>>>>>>>> "everything" :)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> What do you mean by stable binary
key representation and how
> > >>>>>>>>> resolvers
> > >>>>>>>>>>> make
> > >>>>>>>>>>>> it unstable?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -Val
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Apr 6, 2017 at 2:36 AM,
Sergi Vladykin <
> > >>>>>>>>>> sergi.vladykin@gmail.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Val,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I know that you have really
vast experience in Ignite
> > >>>>>>>> deployments
> > >>>>>>>>> and
> > >>>>>>>>>>>>> probably saw everything that
can happen. Did you ever see
> > >>>>>>>> identity
> > >>>>>>>>>>>>> resolvers use in real life?
I guess no.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hibernate example is bad here,
because if their key is
> > >>>>>> unstable
> > >>>>>>>>>> across
> > >>>>>>>>>>>>> multiple JVMs, it means that
it was not designed for
> > >>>>>> distributed
> > >>>>>>>>>>> caches a
> > >>>>>>>>>>>>> priori.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Also knowing in advance about
stable binary key
> > >>>>> representation
> > >>>>>>>>> allows
> > >>>>>>>>>>> us
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>> apply additional optimizations,
like comparing keys
> > >>>> without
> > >>>>>>>>> detaching
> > >>>>>>>>>>>> them
> > >>>>>>>>>>>>> from offheap memory.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We always will be able to add
this stuff back if we see
> > >>>>> users
> > >>>>>>>>> really
> > >>>>>>>>>>> need
> > >>>>>>>>>>>>> it. Let's remove it for 2.0.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Sergi
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 2017-04-06 11:21 GMT+03:00
Valentin Kulichenko <
> > >>>>>>>>>>>>> valentin.kulichenko@gmail.com>:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Alex,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> To be honest, I don't understand
the reasoning behind
> > >>>> the
> > >>>>>>>>> removal.
> > >>>>>>>>>> I
> > >>>>>>>>>>>>> think
> > >>>>>>>>>>>>>> resolvers provide good
flexibility for different corner
> > >>>>>> cases
> > >>>>>>>> and
> > >>>>>>>>>>> it's
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>> good thing to have them.
Note that they can be applied
> > >>>> not
> > >>>>>>>> only
> > >>>>>>>>> to
> > >>>>>>>>>>>> cache
> > >>>>>>>>>>>>>> keys, but to any binary
objects.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hibernate issue is actually
a good example of such use
> > >>>>> case.
> > >>>>>>>> The
> > >>>>>>>>>> fact
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>> we found an alternative
solution doesn't actually mean
> > >>>>>>>> anything,
> > >>>>>>>>>>>> because
> > >>>>>>>>>>>>>> what if this happened not
in our module, but in user's
> > >>>>>>>>> application?
> > >>>>>>>>>>>>>> Unfortunately, we can't
predict everything.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Error proneness is not
a very strong argument either,
> > >>>>>> because
> > >>>>>>>> in
> > >>>>>>>>> my
> > >>>>>>>>>>>> view
> > >>>>>>>>>>>>>> these resolvers are as
much error prone as
> > >>>> BinaryIdMapper,
> > >>>>>> for
> > >>>>>>>>>>> example.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> -Val
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Wed, Apr 5, 2017 at
11:44 PM, Alexey Goncharuk <
> > >>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Denis,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Can you suggest a use-case
where identity resolver is
> > >>>>>> needed
> > >>>>>>>>>> (given
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>> agree that a key must
contain only valuable fields)?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2017-04-05 22:08 GMT+03:00
Denis Magda <
> > >>>>> dmagda@apache.org
> > >>>>>>> :
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Where do you want
to remove the identity resolvers
> > >>>>> from?
> > >>>>>>>> If
> > >>>>>>>>>> it’s
> > >>>>>>>>>>>>>> related
> > >>>>>>>>>>>>>>>> to the internals
of Hibernate module then it’s fine
> > >>>>> but
> > >>>>>> if
> > >>>>>>>>> you
> > >>>>>>>>>>>>> suggest
> > >>>>>>>>>>>>>>>> removing identity
resolvers public interfaces then
> > >>>> it
> > >>>>>>>> might
> > >>>>>>>>> be
> > >>>>>>>>>> a
> > >>>>>>>>>>>>> haste
> > >>>>>>>>>>>>>>>> decision.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> —
> > >>>>>>>>>>>>>>>> Denis
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Apr 5, 2017,
at 7:42 AM, Alexey Goncharuk <
> > >>>>>>>>>>>>>>> alexey.goncharuk@gmail.com>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> +1, I see no
other reasons to keep it.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 2017-04-05
13:59 GMT+03:00 Sergi Vladykin <
> > >>>>>>>>>>>>> sergi.vladykin@gmail.com
> > >>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> +1
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Lets drop
them.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Sergi
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 2017-04-05
13:50 GMT+03:00 Dmitriy Govorukhin <
> > >>>>>>>>>>>>>>>>>> dmitriy.govorukhin@gmail.com>
> > >>>>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi
guys, i implemented proxy for IgniteCache in
> > >>>>>>>> hibernate
> > >>>>>>>>>>>>>>> integration,
> > >>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>> proxy
transformate cacheKey to our key wrapper,
> > >>>>>> leaves
> > >>>>>>>>> only
> > >>>>>>>>>>>>>> required
> > >>>>>>>>>>>>>>>>>>> field.
I think we can remove identity resolve,
> > >>>> it
> > >>>>>>>> should
> > >>>>>>>>>> not
> > >>>>>>>>>>>>> broke
> > >>>>>>>>>>>>>>>>>>> integration
with hibernate. Any objections?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On
Wed, Mar 29, 2017 at 11:07 PM, Valentin
> > >>>>>> Kulichenko
> > >>>>>>>> <
> > >>>>>>>>>>>>>>>>>>> valentin.kulichenko@gmail.com>
wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
I'm not saying there is no alternative
> > >>>> solution.
> > >>>>>> But
> > >>>>>>>>> let's
> > >>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>
prove that it works first, and remove resolvers
> > >>>>>> only
> > >>>>>>>>> after
> > >>>>>>>>>>>> that.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
-Val
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
On Wed, Mar 29, 2017 at 12:18 PM, Sergi
> > >>>> Vladykin
> > >>>>> <
> > >>>>>>>>>>>>>>>>>>> sergi.vladykin@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
Guys, nothing is impossible if you know a bit
> > >>>>>> about
> > >>>>>>>>>>>> reflection
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>> Java
> > >>>>>>>>>>>>>>>>>>> :)
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
We had a look at the CacheKey class and it is
> > >>>>>> easily
> > >>>>>>>>>>>>> replaceable.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
Sergi
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
2017-03-29 21:49 GMT+03:00 Dmitriy Setrakyan <
> > >>>>>>>>>>>>>>> dsetrakyan@apache.org
> > >>>>>>>>>>>>>>>>>>> :
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
On Wed, Mar 29, 2017 at 11:43 AM, Valentin
> > >>>>>>>> Kulichenko
> > >>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>
valentin.kulichenko@gmail.com> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
"Hibernate key" is the CacheKey class I was
> > >>>>>>>> referring
> > >>>>>>>>>> to.
> > >>>>>>>>>>>>> It's
> > >>>>>>>>>>>>>>>>>>>>
provided
> > >>>>>>>>>>>>>>>>>>>>>>
by
> > >>>>>>>>>>>>>>>>>>>>>>>
Hibernate, not by user and not by us. So I'm
> > >>>>> not
> > >>>>>>>> sure
> > >>>>>>>>>>> it's
> > >>>>>>>>>>>>>>>>>> possible
> > >>>>>>>>>>>>>>>>>>>>
to
> > >>>>>>>>>>>>>>>>>>>>>>>
replace it.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
If it is impossible to replace or get rid of
> > >>>>> the
> > >>>>>>>>>> Hibernate
> > >>>>>>>>>>>>> key,
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>
discussion valid at all?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message