thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "GZ" ...@zygmund.net>
Subject Re: Serializing struct using c_glib into memory buffer Help!
Date Sun, 06 Dec 2015 16:21:49 GMT
Hi Nobuaki,

Since my last message, I have recompiled thrift with the latest fixes and identified an issue
in my code where I was reading the data back in to a new object. Something I was doing there
( maybe releasing values ) messed up the buffer. If I capture the buffer before this it works
fine. Thank-you for your help. For completeness I include my updated code.


    GError *error = NULL;
    gint32 len = 0;
    ThriftTransport *transport = NULL;
    ThriftProtocol *protocol = NULL;
    ThriftMemoryBuffer *tbuffer = NULL;
    char* pBuffer = NULL;
   	
    tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER, "buf_size", 1000, NULL);
    if(tbuffer){
        transport = THRIFT_TRANSPORT(tbuffer);
        thrift_transport_open (transport, &error);
        if(error){
            printf ("ERROR: Open Transport - %s\n", error->message);
            g_error_free (error);
            exit(1);
        }       
               
        protocol = THRIFT_PROTOCOL(g_object_new (THRIFT_TYPE_BINARY_PROTOCOL, "transport",
transport, NULL));
        if(protocol){
       
            rdata* pData;                    
            pData = g_object_new (TYPE_RDATA,"flag",STATUS_INCACHE,
                                    "id",300,
                                    "start",6537000,
                                    "done",6537458,
                                    "notif",6537150,
                                    "spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL);   

            if(pData){                 
                ThriftStructClass* cls = THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
                len = cls->write(THRIFT_STRUCT(pData),protocol,&error);
                if(error){
                    printf ("ERROR: %s\n", error->message);
                    g_error_free (error);
                }  
				
                /* Print out contents of Memory Buffer */          
                pBuffer = tbuffer->buf->data;
                if(pBuffer != NULL){
                    printHex(pBuffer,len);                    
                }
                              
                g_object_unref (pData);
            }          
            g_object_unref (protocol);
        }
        g_object_unref(tbuffer);
    }
    return (EXIT_SUCCESS);


Thanks Again!
George


Nobuaki Sukegawa <nsukeg@gmail.com> wrote ..
> Hi GZ,
> 
> (const void *)tbuffer->buf seems problematic: it's a GByteArray*.
> tbuffer->buf->data should point to the data.
> 
> On Sun, Dec 6, 2015 at 7:10 AM GZ <gz@zygmund.net> wrote:
> 
> > Hi Nobuaki,
> >
> > That was my understanding too. So when I write the object I would expect
> > to see the data serialised into the memory buffer... in tbuffer->buf But I
> > dont. It would be hard to miss the long "XXXXX.." string.
> >
> > The data must be correctly stored because I was able load the data into a
> > new object correctly.
> >
> > I did try your suggestion and try deserialise the buffer after it had been
> > moved but this did not work, the object was empty.
> >
> > Its very frustrating. I tried the same process using the cpp library and
> > it works!
> >
> > 013394F0 | 08000100 00000206 0002012C 0A000300 | ...........,....
> > 01339500 | 00000000 63BF280A 00040000 00000063 | ....c.(........c
> > 01339510 | C0F20A00 05000000 000063BF BE0B0006 | ..........c.....
> > 01339520 | 0000001D 58585858 58585858 58585858 | ....XXXXXXXXXXXX
> > 01339530 | 58585858 58585858 58585858 58585858 | XXXXXXXXXXXXXXXX
> > 01339540 | 5800                                | X.
> >
> > I think I must be getting the buffer incorrectly because the output above
> > is completely different for the buffer I get in the c_glib version
> > (identical object).
> >
> > Any Ideas ?!
> >
> > Thanks again
> > George
> >
> >
> >
> > Nobuaki Sukegawa <nsukeg@gmail.com> wrote ..
> > > Hi GZ,
> > >
> > > >  string laid out as if it was going out "over-the-wire"
> > > That's the whole point of memory buffer and it is supposed to work for
> > your
> > > use case.
> > > Even with the correct length you don't see the string ?
> > > You may want to just try sending the data of length reported by *read*
> > > method and deserializing on the other side of  the wire.
> > >
> > > On Tue, Dec 1, 2015 at 4:00 AM GZ <gz@zygmund.net> wrote:
> > >
> > > > Hi Nobuaki,
> > > >
> > > > Thanks... I will keep an eye out for the fix. As you pointed out, using
> > > > read returns the correct length (as far as I can tell) and I am able to
> > > > deserialise all the data into a new object correctly. Thanks! However,
> > when
> > > > I look at the memory buffer, I still do not see the string. I suspect
> > that
> > > > the buffer contains a pointer to the string, not the string laid out
> > as if
> > > > it was going out "over-the-wire".
> > > >
> > > > Mayber I am accessing the buffer incorrectly or using the wrong
> > transport?
> > > > Ultimately I want to be able to take the whole buffer and send it to
> > Kafka
> > > > as a message for desrialisation by other components.
> > > >
> > > > Your help is much appreciated!
> > > >
> > > > George
> > > >
> > > >
> > > >
> > > >
> > > > Nobuaki Sukegawa <nsukeg@gmail.com> wrote ..
> > > > > Hi GZ,
> > > > >
> > > > > You're absolutely right about both "&" and the length/contents.
> > > > > It turned out that it is a bug: as you pointed out the write method
> > > > ignores
> > > > > some bytes written.
> > > > > So the data is written and ready to be deserialized but the reported
> > > > length
> > > > > is wrong.
> > > > > If you happen to be building Thrift from the source, the fix will
be
> > > > > available on master branch soon.
> > > > >
> > > > > A very dirty and non-performant workaround is to deserialize the
> > buffer
> > > > > only to know the byte length before sending:
> > > > >    wrong_len = cls->write(THRIFT_STRUCT(pData2),  ...);
> > > > >    correct_len = cls->read(THRIFT_STRUCT(pData2),  ...);
> > > > > you'll see correct byte length since read is not affected by this
> > bug.
> > > > >
> > > > > On Sun, Nov 29, 2015 at 4:22 AM GZ <gz@zygmund.net> wrote:
> > > > >
> > > > > > Hi Nobuaki,
> > > > > >
> > > > > > I tried this originally but write is expecting a ThriftStruct*
> > rather
> > > > than
> > > > > > a ThriftStruct. I have tried with both &pData->parent
and pData..
> > the
> > > > > > results are pretty much the same. Below is a Hex printout of
the
> > memory
> > > > > > buffer (19 bytes).
> > > > > >
> > > > > > &pData->parent
> > > > > > 01FB5040 | 1081FB01 00000000 52000000 80000000 | ........R.......
> > > > > > 01FB5050 | 010000
> > > > > >
> > > > > > pData
> > > > > > 01EA4040 | 1071EA01 00000000 52000000 80000000 | .q......R.......
> > > > > > 01EA4050 | 010000
> > > > > >
> > > > > >
> > > > > > I would expect to see the "XXXXXXX" string serialised into this
> > buffer
> > > > but
> > > > > > it is not.
> > > > > >
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > > Nobuaki Sukegawa <nsukeg@gmail.com> wrote ..
> > > > > > > Hi GZ,
> > > > > > >
> > > > > > > Can you try with "pData->parent" part applied too ?
> > > > > > > Sorry I should have mentioned this too.
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Nov 28, 2015 at 8:25 AM GZ <gz@zygmund.net>
wrote:
> > > > > > >
> > > > > > > > Hi Nobuaki,
> > > > > > > >
> > > > > > > > Yes - the rdata_write was in the generated c file
not the
> > header.
> > > > > > > >
> > > > > > > > I made the changes you suggested, it was just what
I was
> > looking
> > > > for. I
> > > > > > > > have repeated the corrected code to help others but
> > unfortunately
> > > > the
> > > > > > > > original problem persists..
> > > > > > > >
> > > > > > > >
> > > > > > > >     tbuffer = g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> > "buf_size",
> > > > > > 1000,
> > > > > > > > NULL);
> > > > > > > >     if(tbuffer){
> > > > > > > >         transport = THRIFT_TRANSPORT(tbuffer);
> > > > > > > >         thrift_transport_open (transport, &error);
> > > > > > > >
> > > > > > > >         protocol = THRIFT_PROTOCOL(g_object_new
> > > > > > > > (THRIFT_TYPE_BINARY_PROTOCOL, "transport", transport,
NULL));
> > > > > > > >         if(protocol){
> > > > > > > >             rdata* pData;
> > > > > > > >             pData = g_object_new (TYPE_RDATA,"flag",1,
> > > > > > > >                                     "id",27,
> > > > > > > >                                     "start",12000,
> > > > > > > >                                     "done",345435,
> > > > > > > >                                     "notif",34455,
> > > > > > > >
> > > > > > > > "spid","XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",NULL);
> > > > > > > >             if(pData){
> > > > > > > >                 ThriftStructClass* cls =
> > > > > > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > >                 len = cls->write(pData,protocol,&error);
> > > > > > > >
> > > > > > > >                 printf ("%d Bytes written to buffer\n",
len);
> > > > > > > >
> > > > > > > >                 if(tbuffer->buf != NULL){
> > > > > > > >                     printHex((const void *)tbuffer->buf,
100);
> > > > > > > >                 }
> > > > > > > >                 g_object_unref (pData);
> > > > > > > >             }
> > > > > > > >             g_object_unref (protocol);
> > > > > > > >         }
> > > > > > > >         g_object_unref(tbuffer);
> > > > > > > >     }
> > > > > > > >
> > > > > > > >
> > > > > > > > The returned buffer length still appears incorrect
and the
> > string
> > > > does
> > > > > > not
> > > > > > > > appear to be serialised into the buffer.
> > > > > > > >
> > > > > > > > I can track the writes through the writing function
and I
> > notice
> > > > that
> > > > > > for
> > > > > > > > the spid string value
> > > > > > > >
> > > > > > > > thrift_protocol_write_field_begin returns 3 bytes
> > > > > > > > thrift_protocol_write_string returns 50 bytes
> > > > > > > > thrift_protocol_write_field_end returns 0
> > > > > > > >
> > > > > > > > the 50 bytes never get added to the length because
it is
> > overriden
> > > > by
> > > > > > the
> > > > > > > > 0. This is a snippit of the code..
> > > > > > > >
> > > > > > > >   if ((ret = thrift_protocol_write_field_begin (protocol,
> > "spid",
> > > > > > > > T_STRING, 6, error)) < 0)
> > > > > > > >     return -1;
> > > > > > > >   xfer += ret;
> > > > > > > >   if ((ret = thrift_protocol_write_string (protocol,
> > > > this_object->spid,
> > > > > > > > error)) < 0)
> > > > > > > >     return -1;
> > > > > > > >   if ((ret = thrift_protocol_write_field_end (protocol,
> > error)) <
> > > > 0)
> > > > > > > >     return -1;
> > > > > > > >   xfer += ret;
> > > > > > > >
> > > > > > > > The result is that the write function returns 19 bytes
and
> > when I
> > > > view
> > > > > > the
> > > > > > > > buffer I see no string. I guess I must still be missing
> > > > something...
> > > > > > The
> > > > > > > > spid string is 46 characters long. I would expect
to see this
> > > > > > reflected in
> > > > > > > > the length of the buffer.
> > > > > > > >
> > > > > > > > The idl record is
> > > > > > > >
> > > > > > > > enum Status {
> > > > > > > >   ONE = 1,
> > > > > > > >   TWO = 2,
> > > > > > > >   THREE = 4,
> > > > > > > > }
> > > > > > > >
> > > > > > > > struct rdata {
> > > > > > > >   1: Status flag,
> > > > > > > >   2: i16 id,
> > > > > > > >   3: i64 start,
> > > > > > > >   4: i64 done,
> > > > > > > >   5: i64 notif,
> > > > > > > >   6: string spid
> > > > > > > > }
> > > > > > > >
> > > > > > > > Any idea what might be happenning?
> > > > > > > >
> > > > > > > > Thanks again...
> > > > > > > >
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Nobuaki Sukegawa <nsukeg@gmail.com> wrote ..
> > > > > > > > > It is strange, the generated header has rdata_write
?
> > > > > > > > > Seeing generated code, it should be something
like this:
> > > > > > > > >
> > > > > > > > > ThriftStructClass* cls =
> > > > THRIFT_STRUCT_CLASS(RDATA_GET_CLASS(pData));
> > > > > > > > > cls->write(pData->parent, protocol, &error);
> > > > > > > > >
> > > > > > > > > On Thu, Nov 26, 2015 at 8:03 AM GZ <gz@zygmund.net>
wrote:
> > > > > > > > >
> > > > > > > > > > I am new to thrift and have been trying
to serialize a
> > thrift
> > > > > > structure
> > > > > > > > > > using c but have been unable to get it to
work. I want to
> > > > serialize
> > > > > > > > using
> > > > > > > > > > the binary protocol into a memory buffer.
I do not need
> > RPC, I
> > > > > > will be
> > > > > > > > > > passing the message into Kafka. Does anyone
know the
> > correct
> > > > way
> > > > > > to do
> > > > > > > > this?
> > > > > > > > > >
> > > > > > > > > > I have looked through all the examples and
test code and
> > have
> > > > come
> > > > > > up
> > > > > > > > with
> > > > > > > > > > the following. It compiles, but is not working
correctly.
> > I am
> > > > > > clearly
> > > > > > > > not
> > > > > > > > > > doing it correctly. rdata_write is a method
generated by
> > > > thrift.
> > > > > > > > > >
> > > > > > > > > >     GError *error = NULL;
> > > > > > > > > >     ThriftTransport *transport = NULL;
> > > > > > > > > >     ThriftProtocol *protocol = NULL;
> > > > > > > > > >
> > > > > > > > > >     transport=g_object_new (THRIFT_TYPE_MEMORY_BUFFER,
> > > > "buf_size",
> > > > > > 100,
> > > > > > > > > > NULL);
> > > > > > > > > >     if(transport){
> > > > > > > > > >         thrift_transport_open (transport,
&error);
> > > > > > > > > >         protocol = g_object_new
> > (THRIFT_TYPE_BINARY_PROTOCOL,
> > > > > > > > "transport",
> > > > > > > > > > transport,NULL);
> > > > > > > > > >         if(protocol){
> > > > > > > > > >             rdata* pData = g_object_new
> > (TYPE_RDATA,"flag",1,
> > > > > > > > > >                                     "id",27,
> > > > > > > > > >                                     "start",12000,
> > > > > > > > > >                                     "done",345435,
> > > > > > > > > >                                     "notif",34455,
> > > > > > > > > >                                     "spid","This
is the
> > > > > > SPID",NULL);
> > > > > > > > > >             if(pData){
> > > > > > > > > >
> > > > > > > > > >                 /* wanted something like
> > > > > > obj->write(protocol,&error)
> > > > > > > > > >                    But how to get object?
pData does not
> > have
> > > > write
> > > > > > > > method
> > > > > > > > > > */
> > > > > > > > > >
> > > > > > > > > >                 gint32 len =
> > > > > > > > > > rdata_write(pData,(ThriftProtocol*)protocol,&error);
> > > > > > > > > >
> > > > > > > > > >                 // Print Content of Buffer
> > > > > > > > > >                 ThriftMemoryBuffer* pMem
=
> > > > > > > > (ThriftMemoryBuffer*)transport;
> > > > > > > > > >                 if(pMem->buf != NULL){
> > > > > > > > > >                     printHex((const void
*)pMem->buf, len);
> > > > > > > > > >                 }
> > > > > > > > > >                 g_object_unref (pData);
> > > > > > > > > >             }
> > > > > > > > > >             g_object_unref (protocol);
> > > > > > > > > >         }
> > > > > > > > > >         g_object_unref(transport);
> > > > > > > > > >     }
> > > > > > > > > >
> > > > > > > > > > Error handling and support functions skipped
for brevity.
> > > > > > > > > >
> > > > > > > > > > Any help appreciated
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message