ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Tupitsyn <ptupit...@apache.org>
Subject Re: IEP-76 Thin Client Protocol for Ignite 3.0
Date Tue, 06 Jul 2021 06:59:32 GMT
Ivan, Val, and others - are there any open objections or questions?
Can we accept the proposal?

On Mon, Jul 5, 2021 at 1:28 PM Pavel Tupitsyn <ptupitsyn@apache.org> wrote:

> Igniters,
>
> I've updated the IEP to support "Live Schema" [1] from IEP-54.
> Some operations now have schemaless variants, where tuples are serialized
> as maps (String -> val).
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)
>
> On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky <ivandasch@gmail.com>
> wrote:
>
>> Val, my understanding about it was exactly the same as yours, but, again,
>> I
>> heard a different opinion.
>>
>> But nevertheless, binary protocol should not be about objects, records aka
>> tuples are the best varii, simple and powerful.
>>
>> As for me, I want to take part in implementing python and golang thin
>> clients for ignite 3, so consider my remarks using this info. I am not
>> just
>> a rude critic,
>> I am just interested in consice and universal binary prorocol
>> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
>> valentin.kulichenko@gmail.com
>> >:
>>
>> > Ivan,
>> >
>> > KV view does work over the tuples. Nested objects and arbitrary
>> structures
>> > can be stored as blobs. So if you need a basic KV cache, you can always
>> > create a table with two blob fields - one for key and one for value -
>> and
>> > store anything there.
>> >
>> > -Val
>> >
>> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky <ivandasch@gmail.com>
>> > wrote:
>> >
>> > > Val, am I right, that kv view over the tuples is just simple mapping
>> from
>> > > POJO to tuple? No collections, no nested objects? I have discussed
>> this
>> > in
>> > > private with Igor and Pavel and they told me different info.
>> > >
>> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
>> > > valentin.kulichenko@gmail.com
>> > > >:
>> > >
>> > > > Ivan,
>> > > >
>> > > > I was answering your question about the KV API. The API I provided
>> has
>> > > been
>> > > > discussed and agreed upon. One of the goals of the protocol is to
>> > > implement
>> > > > this API, so it should give you a clear idea of what we're looking
>> for.
>> > > >
>> > > > Of course, I agree with you that the protocol should be simple and
>> > > flexible
>> > > > enough to allow other implementations for different languages and
>> > > > platforms.
>> > > >
>> > > > -Val
>> > > >
>> > > > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky <ivandasch@gmail.com
>> >
>> > > > wrote:
>> > > >
>> > > > > Andrey, yep, you are right. This was just a quick idea. As for
>> me, I
>> > > just
>> > > > > don't want to repeat the same problem with compactFooter in thin
>> > client
>> > > > api
>> > > > > of ignite 2.x.
>> > > > >
>> > > > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov <
>> > andrey.mashenkov@gmail.com
>> > > >:
>> > > > >
>> > > > > > >
>> > > > > > > I suppose that we should describe this more verbose
and
>> > explicit. I
>> > > > > > > nevertheless suggest to also consider writing values
this way:
>> > > > > > > - arr of fields names (if name is missed, corresponding
field
>> is
>> > > nil)
>> > > > > > > - arr of rows (row as array, length equal to fields
array)
>> > > > > >
>> > > > > >
>> > > > > > Ivan,
>> > > > > > I think GET and PUT operation parameters should be consistent.
>> > > > > > With PUT operation this way may be tricky.
>> > > > > >
>> > > > > > SQL INSERT operation (which is similar PUT operation) semantic
>> > allows
>> > > > > > skipping columns that have a default value.
>> > > > > > Assume we have smth like this:
>> > > > > >
>> > > > > > CREATE TABLE t1 (
>> > > > > >    'id' INT;
>> > > > > >    'colname' VARCHAR DEFAULT "abc";
>> > > > > > )
>> > > > > > INSERT INTO t1 VALUES(1)
>> > > > > >
>> > > > > > Actually, this will add a row (1, "abc")
>> > > > > >
>> > > > > > Your suggestion related to missed fields will not work this
way
>> as
>> > it
>> > > > is
>> > > > > > impossible to distinct
>> > > > > > case with 'null' value from the case with a default value.
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky <
>> > ivandasch@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > > Here is the description of TUPLE_GET_ALL:
>> > > > > > > - UUID: table ID
>> > > > > > > - int: schema ID
>> > > > > > > - arr of arr: array of rows with values for all columns
in
>> given
>> > > > schema
>> > > > > > >
>> > > > > > > I suppose that we should describe this more verbose
and
>> > explicit. I
>> > > > > > > nevertheless suggest to also consider writing values
this way:
>> > > > > > > - arr of fields names (if name is missed, corresponding
field
>> is
>> > > nil)
>> > > > > > > - arr of rows (row as array, length equal to fields
array)
>> > > > > > >
>> > > > > > > It is quite simple and if we use str8 (it is more than
enough
>> for
>> > > any
>> > > > > > utf-8
>> > > > > > > reasonable field name), overhead will be negligible,
but
>> > > realization
>> > > > > of a
>> > > > > > > client will be way simpler
>> > > > > > >
>> > > > > > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn <
>> ptupitsyn@apache.org>:
>> > > > > > >
>> > > > > > > > > No it isn't, I have carefully read code and
IEP, in your
>> code
>> > > you
>> > > > > > write
>> > > > > > > > > schema id in each tuple.
>> > > > > > > >
>> > > > > > > > There is no code for batch operations yet.
>> > > > > > > >
>> > > > > > > > Here is the description of TUPLE_GET_ALL:
>> > > > > > > > - UUID: table ID
>> > > > > > > > - int: schema ID
>> > > > > > > > - arr of arr: array of rows with values for all
columns in
>> > given
>> > > > > schema
>> > > > > > > > (nil when value is missing for a column)
>> > > > > > > >
>> > > > > > > > As you can see, schema ID is written once for
all rows.
>> > > > > > > > A row is just a set of values according to the
schema.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > > Also, my biggest concern -- extra serde step.
I suppose we
>> > > should
>> > > > > > pass
>> > > > > > > > > bytearray to internal api, and use msgpack
throughout all
>> > wire
>> > > > > > > protocols,
>> > > > > > > > > as tarantool does.
>> > > > > > > >
>> > > > > > > > I agree. But this was decided before in IEP-54,
and is out
>> of
>> > > scope
>> > > > > for
>> > > > > > > > current IEP.
>> > > > > > > > Would you like to start a separate thread to discuss
this?
>> Or I
>> > > can
>> > > > > do
>> > > > > > > this
>> > > > > > > > a bit later.
>> > > > > > > >
>> > > > > > > > On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky
<
>> > > > ivandasch@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > > This is described in all operations
that include
>> multiple
>> > > > tuples.
>> > > > > > > > > No it isn't, I have carefully read code and
IEP, in your
>> code
>> > > you
>> > > > > > write
>> > > > > > > > > schema id in each tuple.
>> > > > > > > > >
>> > > > > > > > > Also, my biggest concern -- extra serde step.
I suppose we
>> > > should
>> > > > > > pass
>> > > > > > > > > bytearray to internal api, and use msgpack
throughout all
>> > wire
>> > > > > > > protocols,
>> > > > > > > > > as tarantool does.
>> > > > > > > > >
>> > > > > > > > > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn
<
>> > ptupitsyn@apache.org
>> > > >:
>> > > > > > > > >
>> > > > > > > > > > Ivan,
>> > > > > > > > > >
>> > > > > > > > > > >  that there is not neccesary to
write schema versions
>> in
>> > > each
>> > > > > row
>> > > > > > > > > > > in collectionof tuples
>> > > > > > > > > >
>> > > > > > > > > > This is described in all operations
that include
>> multiple
>> > > > tuples.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > it is not clear from your code
(probably
>> > > > > > > > > > > mistake?) how differ key tuples
and value tuples from
>> > each
>> > > > > other
>> > > > > > > > > >
>> > > > > > > > > > Key tuples include only key columns.
Key columns come
>> first
>> > > in
>> > > > > the
>> > > > > > > > > schema.
>> > > > > > > > > > Value tuples include all columns, key
and value. Added
>> "Key
>> > > > > tuples"
>> > > > > > > > > > section.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > > As for me, these excercises with
schema's doesn't
>> worth a
>> > > lot
>> > > > > > > > > >
>> > > > > > > > > > I'll add a benchmark and we'll see.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Jul 1, 2021 at 3:17 PM Ivan
Daschinsky <
>> > > > > > ivandasch@gmail.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > I suppose, that there is not neccesary
to write schema
>> > > > versions
>> > > > > > in
>> > > > > > > > each
>> > > > > > > > > > row
>> > > > > > > > > > > in collectionof tuples. Also it
is not clear from your
>> > code
>> > > > > > > (probably
>> > > > > > > > > > > mistake?) how differ key tuples
and value tuples from
>> > each
>> > > > > other.
>> > > > > > > In
>> > > > > > > > > > > readTuple you always read full
schema and check for
>> full
>> > > > > length.
>> > > > > > As
>> > > > > > > > for
>> > > > > > > > > > me,
>> > > > > > > > > > > these excercises with schema's
doesn't worth a lot.
>> I.e.
>> > > > > postgres
>> > > > > > > > just
>> > > > > > > > > > > writes field names and then simpy
rows with data.
>> Saving
>> > > few
>> > > > > > bytes
>> > > > > > > > > > doesn't
>> > > > > > > > > > > make much deal. Btw, msgpack has
special types for
>> short
>> > > > > strings
>> > > > > > > > (i.e.
>> > > > > > > > > > > str8). It is much easier use it
and write field name
>> as
>> > is.
>> > > > > > > > > > >
>> > > > > > > > > > > чт, 1 июл. 2021 г., 14:56
Pavel Tupitsyn <
>> > > > ptupitsyn@apache.org
>> > > > > >:
>> > > > > > > > > > >
>> > > > > > > > > > > > Ivan, tuple serialization
section added to the IEP,
>> let
>> > > me
>> > > > > know
>> > > > > > > if
>> > > > > > > > it
>> > > > > > > > > > is
>> > > > > > > > > > > > clear enough.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks!
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Thu, Jul 1, 2021 at 2:06
PM Ivan Daschinsky <
>> > > > > > > > ivandasch@gmail.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > I can't find any description
of tuple
>> serialization
>> > in
>> > > > IEP,
>> > > > > > > only
>> > > > > > > > in
>> > > > > > > > > > > code
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > чт, 1 июл. 2021
г., 13:59 Pavel Tupitsyn <
>> > > > > > ptupitsyn@apache.org
>> > > > > > > >:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Ivan,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 0. The IEP is not
in progress, it is ready for
>> > review
>> > > > and
>> > > > > > > > > > discussion.
>> > > > > > > > > > > > > > 1. Tuple serialization
is described in the IEP
>> and
>> > > > > > > demonstrated
>> > > > > > > > > in
>> > > > > > > > > > > the
>> > > > > > > > > > > > > PoC
>> > > > > > > > > > > > > > (see ClientMessageHandler#readTuple),
let me
>> know
>> > if
>> > > > more
>> > > > > > > > details
>> > > > > > > > > > are
>> > > > > > > > > > > > > > required
>> > > > > > > > > > > > > > 2. Tuple schema
serialization is described in
>> > > > SCHEMAS_GET
>> > > > > > > > > section.
>> > > > > > > > > > > > Table
>> > > > > > > > > > > > > > schema (configuration)
needs more details, you
>> are
>> > > > right
>> > > > > -
>> > > > > > > I'll
>> > > > > > > > > add
>> > > > > > > > > > > > them.
>> > > > > > > > > > > > > > 3. This IEP is about
tables (tuple-based) API
>> only,
>> > > > since
>> > > > > > it
>> > > > > > > is
>> > > > > > > > > the
>> > > > > > > > > > > > only
>> > > > > > > > > > > > > > API that we have
right now, as noted in Risks
>> and
>> > > > > > > Assumptions.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Thu, Jul 1, 2021
at 1:53 PM Ivan Daschinsky <
>> > > > > > > > > > ivandasch@gmail.com>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Also, is there
any clear information about KV
>> > api?
>> > > Is
>> > > > > > there
>> > > > > > > > any
>> > > > > > > > > > > plan
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > implement it?
Or is there any proposal about
>> it?
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > чт, 1 июл.
2021 г., 13:51 Ivan Daschinsky <
>> > > > > > > > ivandasch@gmail.com
>> > > > > > > > > >:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Pavel,
but IEP is in progress, isn't it?
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > 1. There
is not any information about tuple
>> > > > > > > serialization.
>> > > > > > > > > And
>> > > > > > > > > > > > there
>> > > > > > > > > > > > > > > isn't
>> > > > > > > > > > > > > > > > a clear
consensus about it.
>> > > > > > > > > > > > > > > > 2. There
is not any information about schrma
>> > > > > > > serialization
>> > > > > > > > > > > format.
>> > > > > > > > > > > > > And
>> > > > > > > > > > > > > > > > AFAIK,
there isn't a clear consensus also.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > чт,
1 июл. 2021 г., 13:26 Pavel Tupitsyn <
>> > > > > > > > > ptupitsyn@apache.org
>> > > > > > > > > > >:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> Igniters,
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> Please
review the IEP for thin client
>> protocol
>> > > in
>> > > > > 3.0
>> > > > > > > [1].
>> > > > > > > > > > > > > > > >> PoC
is in progress [2]
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> [1]
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> [2]
>> > https://github.com/apache/ignite-3/pull/191
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Best regards,
>> > > > > > Andrey V. Mashenkov
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message