ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Daschinsky <ivanda...@gmail.com>
Subject Re: IEP-76 Thin Client Protocol for Ignite 3.0
Date Thu, 01 Jul 2021 16:21:01 GMT
Igor, as you can see from the code, we read data from msgpack and build the
tuple from scratch. Then we serialize this tuple again when send it to
backups. I presume this is absolutely unnecessary

чт, 1 июл. 2021 г., 18:42 Igor Sapego <isapego@apache.org>:

> Ivan, what are extra serde steps you are talking about?
>
> Best Regards,
> Igor
>
>
> On Thu, Jul 1, 2021 at 5:52 PM Ivan Daschinsky <ivandasch@gmail.com>
> wrote:
>
> > > I agree. But this was decided before in IEP-54, and is out of scope for
> > current IEP.
> > Would you like to start a separate thread to discuss this? Or I can do
> this
> > a bit later.
> >
> > Great idea, let's discuss it. I suppose this will simplify many aspects
> of
> > realization and improve performance a lot
> >
> > чт, 1 июл. 2021 г., 17:50 Ivan Daschinsky <ivandasch@gmail.com>:
> >
> > > > Here is the description of TUPLE_GET_ALL:
> > > - UUID: table ID
> > > - int: schema ID
> > > - arr of arr: array of rows with values for all columns in given schema
> > >
> > > I suppose that we should describe this more verbose and explicit. I
> > > nevertheless suggest to also consider writing values this way:
> > > - arr of fields names (if name is missed, corresponding field is nil)
> > > - arr of rows (row as array, length equal to fields array)
> > >
> > > It is quite simple and if we use str8 (it is more than enough for any
> > > utf-8 reasonable field name), overhead will be negligible, but
> > realization
> > > of a client will be way simpler
> > >
> > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn <ptupitsyn@apache.org>:
> > >
> > >> > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > >> > schema id in each tuple.
> > >>
> > >> There is no code for batch operations yet.
> > >>
> > >> Here is the description of TUPLE_GET_ALL:
> > >> - UUID: table ID
> > >> - int: schema ID
> > >> - arr of arr: array of rows with values for all columns in given
> schema
> > >> (nil when value is missing for a column)
> > >>
> > >> As you can see, schema ID is written once for all rows.
> > >> A row is just a set of values according to the schema.
> > >>
> > >>
> > >> > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > >> > bytearray to internal api, and use msgpack throughout all wire
> > >> protocols,
> > >> > as tarantool does.
> > >>
> > >> I agree. But this was decided before in IEP-54, and is out of scope
> for
> > >> current IEP.
> > >> Would you like to start a separate thread to discuss this? Or I can do
> > >> this
> > >> a bit later.
> > >>
> > >> On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky <ivandasch@gmail.com>
> > >> wrote:
> > >>
> > >> > > This is described in all operations that include multiple tuples.
> > >> > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > >> > schema id in each tuple.
> > >> >
> > >> > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > >> > bytearray to internal api, and use msgpack throughout all wire
> > >> protocols,
> > >> > as tarantool does.
> > >> >
> > >> > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn <ptupitsyn@apache.org>:
> > >> >
> > >> > > Ivan,
> > >> > >
> > >> > > >  that there is not neccesary to write schema versions in
each
> row
> > >> > > > in collectionof tuples
> > >> > >
> > >> > > This is described in all operations that include multiple tuples.
> > >> > >
> > >> > >
> > >> > > > it is not clear from your code (probably
> > >> > > > mistake?) how differ key tuples and value tuples from each
other
> > >> > >
> > >> > > Key tuples include only key columns. Key columns come first in
the
> > >> > schema.
> > >> > > Value tuples include all columns, key and value. Added "Key
> tuples"
> > >> > > section.
> > >> > >
> > >> > >
> > >> > > > As for me, these excercises with schema's doesn't worth
a lot
> > >> > >
> > >> > > I'll add a benchmark and we'll see.
> > >> > >
> > >> > >
> > >> > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky <
> ivandasch@gmail.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > I suppose, that there is not neccesary to write schema versions
> in
> > >> each
> > >> > > row
> > >> > > > in collectionof tuples. Also it is not clear from your code
> > >> (probably
> > >> > > > mistake?) how differ key tuples and value tuples from each
> other.
> > In
> > >> > > > readTuple you always read full schema and check for full
length.
> > As
> > >> for
> > >> > > me,
> > >> > > > these excercises with schema's doesn't worth a lot. I.e.
> postgres
> > >> just
> > >> > > > writes field names and then simpy rows with data. Saving
few
> bytes
> > >> > > doesn't
> > >> > > > make much deal. Btw, msgpack has special types for short
strings
> > >> (i.e.
> > >> > > > str8). It is much easier use it and write field name as
is.
> > >> > > >
> > >> > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn <ptupitsyn@apache.org
> >:
> > >> > > >
> > >> > > > > Ivan, tuple serialization section added to the IEP,
let me
> know
> > >> if it
> > >> > > is
> > >> > > > > clear enough.
> > >> > > > >
> > >> > > > > Thanks!
> > >> > > > >
> > >> > > > > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky <
> > >> ivandasch@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > I can't find any description of tuple serialization
in IEP,
> > >> only in
> > >> > > > code
> > >> > > > > >
> > >> > > > > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn
<
> > ptupitsyn@apache.org
> > >> >:
> > >> > > > > >
> > >> > > > > > > Ivan,
> > >> > > > > > >
> > >> > > > > > > 0. The IEP is not in progress, it is ready
for review and
> > >> > > discussion.
> > >> > > > > > > 1. Tuple serialization is described in the
IEP and
> > >> demonstrated
> > >> > in
> > >> > > > the
> > >> > > > > > PoC
> > >> > > > > > > (see ClientMessageHandler#readTuple), let
me know if more
> > >> details
> > >> > > are
> > >> > > > > > > required
> > >> > > > > > > 2. Tuple schema serialization is described
in SCHEMAS_GET
> > >> > section.
> > >> > > > > Table
> > >> > > > > > > schema (configuration) needs more details,
you are right -
> > >> I'll
> > >> > add
> > >> > > > > them.
> > >> > > > > > > 3. This IEP is about tables (tuple-based)
API only, since
> it
> > >> is
> > >> > the
> > >> > > > > only
> > >> > > > > > > API that we have right now, as noted in Risks
and
> > Assumptions.
> > >> > > > > > >
> > >> > > > > > > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky
<
> > >> > > ivandasch@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Also, is there any clear information
about KV api? Is
> > there
> > >> any
> > >> > > > plan
> > >> > > > > to
> > >> > > > > > > > implement it? Or is there any proposal
about it?
> > >> > > > > > > >
> > >> > > > > > > > чт, 1 июл. 2021 г., 13:51 Ivan
Daschinsky <
> > >> ivandasch@gmail.com
> > >> > >:
> > >> > > > > > > >
> > >> > > > > > > > > Pavel, but IEP is in progress,
isn't it?
> > >> > > > > > > > >
> > >> > > > > > > > > 1. There is not any information
about tuple
> > serialization.
> > >> > And
> > >> > > > > there
> > >> > > > > > > > isn't
> > >> > > > > > > > > a clear consensus about it.
> > >> > > > > > > > > 2. There is not any information
about schrma
> > serialization
> > >> > > > format.
> > >> > > > > > And
> > >> > > > > > > > > AFAIK, there isn't a clear consensus
also.
> > >> > > > > > > > >
> > >> > > > > > > > > чт, 1 июл. 2021 г., 13:26
Pavel Tupitsyn <
> > >> > ptupitsyn@apache.org
> > >> > > >:
> > >> > > > > > > > >
> > >> > > > > > > > >> Igniters,
> > >> > > > > > > > >>
> > >> > > > > > > > >> Please review the IEP for thin
client protocol in 3.0
> > >> [1].
> > >> > > > > > > > >> PoC is in progress [2]
> > >> > > > > > > > >>
> > >> > > > > > > > >> [1]
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > >> > > > > > > > >>
> > >> > > > > > > > >> [2] https://github.com/apache/ignite-3/pull/191
> > >> > > > > > > > >>
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message