ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: IEP-76 Thin Client Protocol for Ignite 3.0
Date Tue, 06 Jul 2021 19:30:31 GMT
The proposal looks good to me.

-Val

On Tue, Jul 6, 2021 at 2:24 AM Ivan Daschinsky <ivandasch@gmail.com> wrote:

> I suppose, that general idea is great. Some details are missing, but I
> suppose during implementation of clients we will add more details and
> redefine some parts.
>
> вт, 6 июл. 2021 г., 09:59 Pavel Tupitsyn <ptupitsyn@apache.org>:
>
> > Ivan, Val, and others - are there any open objections or questions?
> > Can we accept the proposal?
> >
> > On Mon, Jul 5, 2021 at 1:28 PM Pavel Tupitsyn <ptupitsyn@apache.org>
> > wrote:
> >
> > > Igniters,
> > >
> > > I've updated the IEP to support "Live Schema" [1] from IEP-54.
> > > Some operations now have schemaless variants, where tuples are
> serialized
> > > as maps (String -> val).
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)
> > >
> > > On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky <ivandasch@gmail.com>
> > > wrote:
> > >
> > >> Val, my understanding about it was exactly the same as yours, but,
> > again,
> > >> I
> > >> heard a different opinion.
> > >>
> > >> But nevertheless, binary protocol should not be about objects, records
> > aka
> > >> tuples are the best varii, simple and powerful.
> > >>
> > >> As for me, I want to take part in implementing python and golang thin
> > >> clients for ignite 3, so consider my remarks using this info. I am not
> > >> just
> > >> a rude critic,
> > >> I am just interested in consice and universal binary prorocol
> > >> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
> > >> valentin.kulichenko@gmail.com
> > >> >:
> > >>
> > >> > Ivan,
> > >> >
> > >> > KV view does work over the tuples. Nested objects and arbitrary
> > >> structures
> > >> > can be stored as blobs. So if you need a basic KV cache, you can
> > always
> > >> > create a table with two blob fields - one for key and one for value
> -
> > >> and
> > >> > store anything there.
> > >> >
> > >> > -Val
> > >> >
> > >> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky <ivandasch@gmail.com
> >
> > >> > wrote:
> > >> >
> > >> > > Val, am I right, that kv view over the tuples is just simple
> mapping
> > >> from
> > >> > > POJO to tuple? No collections, no nested objects? I have discussed
> > >> this
> > >> > in
> > >> > > private with Igor and Pavel and they told me different info.
> > >> > >
> > >> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
> > >> > > valentin.kulichenko@gmail.com
> > >> > > >:
> > >> > >
> > >> > > > Ivan,
> > >> > > >
> > >> > > > I was answering your question about the KV API. The API
I
> provided
> > >> has
> > >> > > been
> > >> > > > discussed and agreed upon. One of the goals of the protocol
is
> to
> > >> > > implement
> > >> > > > this API, so it should give you a clear idea of what we're
> looking
> > >> for.
> > >> > > >
> > >> > > > Of course, I agree with you that the protocol should be
simple
> and
> > >> > > flexible
> > >> > > > enough to allow other implementations for different languages
> and
> > >> > > > platforms.
> > >> > > >
> > >> > > > -Val
> > >> > > >
> > >> > > > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky <
> > ivandasch@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Andrey, yep, you are right. This was just a quick idea.
As for
> > >> me, I
> > >> > > just
> > >> > > > > don't want to repeat the same problem with compactFooter
in
> thin
> > >> > client
> > >> > > > api
> > >> > > > > of ignite 2.x.
> > >> > > > >
> > >> > > > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov <
> > >> > andrey.mashenkov@gmail.com
> > >> > > >:
> > >> > > > >
> > >> > > > > > >
> > >> > > > > > > I suppose that we should describe this more
verbose and
> > >> > explicit. I
> > >> > > > > > > nevertheless suggest to also consider writing
values this
> > way:
> > >> > > > > > > - arr of fields names (if name is missed,
corresponding
> > field
> > >> is
> > >> > > nil)
> > >> > > > > > > - arr of rows (row as array, length equal
to fields array)
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Ivan,
> > >> > > > > > I think GET and PUT operation parameters should
be
> consistent.
> > >> > > > > > With PUT operation this way may be tricky.
> > >> > > > > >
> > >> > > > > > SQL INSERT operation (which is similar PUT operation)
> semantic
> > >> > allows
> > >> > > > > > skipping columns that have a default value.
> > >> > > > > > Assume we have smth like this:
> > >> > > > > >
> > >> > > > > > CREATE TABLE t1 (
> > >> > > > > >    'id' INT;
> > >> > > > > >    'colname' VARCHAR DEFAULT "abc";
> > >> > > > > > )
> > >> > > > > > INSERT INTO t1 VALUES(1)
> > >> > > > > >
> > >> > > > > > Actually, this will add a row (1, "abc")
> > >> > > > > >
> > >> > > > > > Your suggestion related to missed fields will
not work this
> > way
> > >> as
> > >> > it
> > >> > > > is
> > >> > > > > > impossible to distinct
> > >> > > > > > case with 'null' value from the case with a default
value.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky
<
> > >> > ivandasch@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > > Here is the description of TUPLE_GET_ALL:
> > >> > > > > > > - UUID: table ID
> > >> > > > > > > - int: schema ID
> > >> > > > > > > - arr of arr: array of rows with values for
all columns in
> > >> given
> > >> > > > schema
> > >> > > > > > >
> > >> > > > > > > I suppose that we should describe this more
verbose and
> > >> > explicit. I
> > >> > > > > > > nevertheless suggest to also consider writing
values this
> > way:
> > >> > > > > > > - arr of fields names (if name is missed,
corresponding
> > field
> > >> is
> > >> > > nil)
> > >> > > > > > > - arr of rows (row as array, length equal
to fields array)
> > >> > > > > > >
> > >> > > > > > > It is quite simple and if we use str8 (it
is more than
> > enough
> > >> for
> > >> > > any
> > >> > > > > > utf-8
> > >> > > > > > > reasonable field name), overhead will be
negligible, but
> > >> > > realization
> > >> > > > > of a
> > >> > > > > > > client will be way simpler
> > >> > > > > > >
> > >> > > > > > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn
<
> > >> ptupitsyn@apache.org>:
> > >> > > > > > >
> > >> > > > > > > > > No it isn't, I have carefully read
code and IEP, in
> your
> > >> code
> > >> > > you
> > >> > > > > > write
> > >> > > > > > > > > schema id in each tuple.
> > >> > > > > > > >
> > >> > > > > > > > There is no code for batch operations
yet.
> > >> > > > > > > >
> > >> > > > > > > > Here is the description of TUPLE_GET_ALL:
> > >> > > > > > > > - UUID: table ID
> > >> > > > > > > > - int: schema ID
> > >> > > > > > > > - arr of arr: array of rows with values
for all columns
> in
> > >> > given
> > >> > > > > schema
> > >> > > > > > > > (nil when value is missing for a column)
> > >> > > > > > > >
> > >> > > > > > > > As you can see, schema ID is written
once for all rows.
> > >> > > > > > > > A row is just a set of values according
to the schema.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > > Also, my biggest concern -- extra
serde step. I
> suppose
> > we
> > >> > > should
> > >> > > > > > pass
> > >> > > > > > > > > bytearray to internal api, and
use msgpack throughout
> > all
> > >> > wire
> > >> > > > > > > protocols,
> > >> > > > > > > > > as tarantool does.
> > >> > > > > > > >
> > >> > > > > > > > I agree. But this was decided before
in IEP-54, and is
> out
> > >> of
> > >> > > scope
> > >> > > > > for
> > >> > > > > > > > current IEP.
> > >> > > > > > > > Would you like to start a separate thread
to discuss
> this?
> > >> Or I
> > >> > > can
> > >> > > > > do
> > >> > > > > > > this
> > >> > > > > > > > a bit later.
> > >> > > > > > > >
> > >> > > > > > > > On Thu, Jul 1, 2021 at 4:41 PM Ivan
Daschinsky <
> > >> > > > ivandasch@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > > This is described in all operations
that include
> > >> multiple
> > >> > > > tuples.
> > >> > > > > > > > > No it isn't, I have carefully read
code and IEP, in
> your
> > >> code
> > >> > > you
> > >> > > > > > write
> > >> > > > > > > > > schema id in each tuple.
> > >> > > > > > > > >
> > >> > > > > > > > > Also, my biggest concern -- extra
serde step. I
> suppose
> > we
> > >> > > should
> > >> > > > > > pass
> > >> > > > > > > > > bytearray to internal api, and
use msgpack throughout
> > all
> > >> > wire
> > >> > > > > > > protocols,
> > >> > > > > > > > > as tarantool does.
> > >> > > > > > > > >
> > >> > > > > > > > > чт, 1 июл. 2021 г., 16:15
Pavel Tupitsyn <
> > >> > ptupitsyn@apache.org
> > >> > > >:
> > >> > > > > > > > >
> > >> > > > > > > > > > Ivan,
> > >> > > > > > > > > >
> > >> > > > > > > > > > >  that there is not neccesary
to write schema
> > versions
> > >> in
> > >> > > each
> > >> > > > > row
> > >> > > > > > > > > > > in collectionof tuples
> > >> > > > > > > > > >
> > >> > > > > > > > > > This is described in all operations
that include
> > >> multiple
> > >> > > > tuples.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > > it is not clear from
your code (probably
> > >> > > > > > > > > > > mistake?) how differ
key tuples and value tuples
> > from
> > >> > each
> > >> > > > > other
> > >> > > > > > > > > >
> > >> > > > > > > > > > Key tuples include only key
columns. Key columns
> come
> > >> first
> > >> > > in
> > >> > > > > the
> > >> > > > > > > > > schema.
> > >> > > > > > > > > > Value tuples include all columns,
key and value.
> Added
> > >> "Key
> > >> > > > > tuples"
> > >> > > > > > > > > > section.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > > As for me, these excercises
with schema's doesn't
> > >> worth a
> > >> > > lot
> > >> > > > > > > > > >
> > >> > > > > > > > > > I'll add a benchmark and we'll
see.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Thu, Jul 1, 2021 at 3:17
PM Ivan Daschinsky <
> > >> > > > > > ivandasch@gmail.com>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > I suppose, that there
is not neccesary to write
> > schema
> > >> > > > versions
> > >> > > > > > in
> > >> > > > > > > > each
> > >> > > > > > > > > > row
> > >> > > > > > > > > > > in collectionof tuples.
Also it is not clear from
> > your
> > >> > code
> > >> > > > > > > (probably
> > >> > > > > > > > > > > mistake?) how differ
key tuples and value tuples
> > from
> > >> > each
> > >> > > > > other.
> > >> > > > > > > In
> > >> > > > > > > > > > > readTuple you always
read full schema and check
> for
> > >> full
> > >> > > > > length.
> > >> > > > > > As
> > >> > > > > > > > for
> > >> > > > > > > > > > me,
> > >> > > > > > > > > > > these excercises with
schema's doesn't worth a
> lot.
> > >> I.e.
> > >> > > > > postgres
> > >> > > > > > > > just
> > >> > > > > > > > > > > writes field names and
then simpy rows with data.
> > >> Saving
> > >> > > few
> > >> > > > > > bytes
> > >> > > > > > > > > > doesn't
> > >> > > > > > > > > > > make much deal. Btw,
msgpack has special types for
> > >> short
> > >> > > > > strings
> > >> > > > > > > > (i.e.
> > >> > > > > > > > > > > str8). It is much easier
use it and write field
> name
> > >> as
> > >> > is.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > чт, 1 июл. 2021
г., 14:56 Pavel Tupitsyn <
> > >> > > > ptupitsyn@apache.org
> > >> > > > > >:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Ivan, tuple serialization
section added to the
> > IEP,
> > >> let
> > >> > > me
> > >> > > > > know
> > >> > > > > > > if
> > >> > > > > > > > it
> > >> > > > > > > > > > is
> > >> > > > > > > > > > > > clear enough.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks!
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Thu, Jul 1, 2021
at 2:06 PM Ivan Daschinsky <
> > >> > > > > > > > ivandasch@gmail.com>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > I can't find
any description of tuple
> > >> serialization
> > >> > in
> > >> > > > IEP,
> > >> > > > > > > only
> > >> > > > > > > > in
> > >> > > > > > > > > > > code
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > чт, 1 июл.
2021 г., 13:59 Pavel Tupitsyn <
> > >> > > > > > ptupitsyn@apache.org
> > >> > > > > > > >:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Ivan,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > 0. The
IEP is not in progress, it is ready
> for
> > >> > review
> > >> > > > and
> > >> > > > > > > > > > discussion.
> > >> > > > > > > > > > > > > > 1. Tuple
serialization is described in the
> IEP
> > >> and
> > >> > > > > > > demonstrated
> > >> > > > > > > > > in
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > PoC
> > >> > > > > > > > > > > > > > (see ClientMessageHandler#readTuple),
let me
> > >> know
> > >> > if
> > >> > > > more
> > >> > > > > > > > details
> > >> > > > > > > > > > are
> > >> > > > > > > > > > > > > > required
> > >> > > > > > > > > > > > > > 2. Tuple
schema serialization is described
> in
> > >> > > > SCHEMAS_GET
> > >> > > > > > > > > section.
> > >> > > > > > > > > > > > Table
> > >> > > > > > > > > > > > > > schema
(configuration) needs more details,
> you
> > >> are
> > >> > > > right
> > >> > > > > -
> > >> > > > > > > I'll
> > >> > > > > > > > > add
> > >> > > > > > > > > > > > them.
> > >> > > > > > > > > > > > > > 3. This
IEP is about tables (tuple-based)
> API
> > >> only,
> > >> > > > since
> > >> > > > > > it
> > >> > > > > > > is
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > only
> > >> > > > > > > > > > > > > > API that
we have right now, as noted in
> Risks
> > >> and
> > >> > > > > > > Assumptions.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Thu,
Jul 1, 2021 at 1:53 PM Ivan
> > Daschinsky <
> > >> > > > > > > > > > ivandasch@gmail.com>
> > >> > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Also,
is there any clear information about
> > KV
> > >> > api?
> > >> > > Is
> > >> > > > > > there
> > >> > > > > > > > any
> > >> > > > > > > > > > > plan
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > implement
it? Or is there any proposal
> about
> > >> it?
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > чт,
1 июл. 2021 г., 13:51 Ivan Daschinsky
> <
> > >> > > > > > > > ivandasch@gmail.com
> > >> > > > > > > > > >:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
Pavel, but IEP is in progress, isn't it?
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
1. There is not any information about
> > tuple
> > >> > > > > > > serialization.
> > >> > > > > > > > > And
> > >> > > > > > > > > > > > there
> > >> > > > > > > > > > > > > > > isn't
> > >> > > > > > > > > > > > > > > >
a clear consensus about it.
> > >> > > > > > > > > > > > > > > >
2. There is not any information about
> > schrma
> > >> > > > > > > serialization
> > >> > > > > > > > > > > format.
> > >> > > > > > > > > > > > > And
> > >> > > > > > > > > > > > > > > >
AFAIK, there isn't a clear consensus
> also.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
чт, 1 июл. 2021 г., 13:26 Pavel
> Tupitsyn <
> > >> > > > > > > > > ptupitsyn@apache.org
> > >> > > > > > > > > > >:
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >>
Igniters,
> > >> > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > >>
Please review the IEP for thin client
> > >> protocol
> > >> > > in
> > >> > > > > 3.0
> > >> > > > > > > [1].
> > >> > > > > > > > > > > > > > > >>
PoC is in progress [2]
> > >> > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > >>
[1]
> > >> > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > >> > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > >>
[2]
> > >> > https://github.com/apache/ignite-3/pull/191
> > >> > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > --
> > >> > > > > > Best regards,
> > >> > > > > > Andrey V. Mashenkov
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message