thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Randy Abernethy <randy.aberne...@gmail.com>
Subject Re: Error with binary data with node.js
Date Wed, 15 Oct 2014 21:47:19 GMT
The source at RandyAbernethy/thriftnode is a copy of
https://github.com/apache/thrift/tree/master/lib/nodejs. You can
easily make your own copy but you can not just fork the apache/thrift
repo (which is why I maintain a node only repo). The semver tags on
the apache repo will cause npm to use code from over a year ago (which
has lots of issues). This will be fixed when 0.9.2 is released in the
near future, causing npm install to pull a well tested node source
base.

On Wed, Oct 15, 2014 at 11:07 AM, Rodolfo Ochoa <rodolfo.ochoa@gmail.com> wrote:
> where is this generated?
> how can I generate my own?
>
> On Sat, Sep 20, 2014 at 6:03 PM, Randy Abernethy <randy.abernethy@gmail.com>
> wrote:
>
>> I think the official answer to that is always "real soon now".
>>
>> In the mean time you can get the current master with npm as follows:
>>
>> $ npm install RandyAbernethy/thriftnode
>>
>> This is a clone of the thrift master branch setup to work with npm.
>>
>>
>> On Fri, Sep 19, 2014 at 4:41 PM, Rodolfo Ochoa <rodolfo.ochoa@gmail.com>
>> wrote:
>> > Oh, it is fixed on newest version...
>> >
>> > do you have a release date?
>> >
>> > thanks
>> >
>> > On Fri, Sep 19, 2014 at 6:24 PM, Rodolfo Ochoa <rodolfo.ochoa@gmail.com>
>> > wrote:
>> >
>> >> this error start happening here:
>> >> https://issues.apache.org/jira/browse/THRIFT-1679
>> >> which is a correct fix, but then you need to correctly implement binary.
>> >>
>> >> -Rod.O
>> >>
>> >> On Fri, Sep 19, 2014 at 5:33 PM, Rodolfo Ochoa <rodolfo.ochoa@gmail.com
>> >
>> >> wrote:
>> >>
>> >>> the problem is...
>> >>> the problem is that for the generated code,
>> >>> thrift maps binary data into strings, and uses the method readString
>> from
>> >>> the BinaryProtocol
>> >>> protocol.js:280 TBinaryProtocol.prototype.readString = function() {
>> >>> instead of using readBinary,
>> >>> protocol.js:275 TBinaryProtocol.prototype.readBinary = function() {
>> >>> which would return a buffer object.
>> >>> readString method in the transport will use
>> buffer(...).toString("utf8")
>> >>> which will sanitize utf and will replace invalid unicode sequences
>> with the
>> >>> Unicode Replacement Character.
>> >>>
>> >>> *my code:*
>> >>> my binary field was mapped on ondb_types.js as:
>> >>> TGetResult.prototype = {};
>> >>> TGetResult.prototype.read = function(input) {
>> >>>   input.readStructBegin();
>> >>>   while (true)
>> >>>   {
>> >>>     var ret = input.readFieldBegin();
>> >>>     var fname = ret.fname;
>> >>>     var ftype = ret.ftype;
>> >>>     var fid = ret.fid;
>> >>>     if (ftype == Thrift.Type.STOP) {
>> >>>       break;
>> >>>     }
>> >>>     switch (fid)
>> >>>     {
>> >>>       case 1:
>> >>>       if (ftype == Thrift.Type.STRUCT) {
>> >>>         this.currentRow = new ttypes.TRow();
>> >>>         this.currentRow.read(input);
>> >>>       } else {
>> >>>         input.skip(ftype);
>> >>>       }
>> >>>       break;
>> >>>       case 2:
>> >>> *      if (ftype == Thrift.Type.STRING) {*
>> >>> *        this.currentRowVersion = input.readString();*
>> >>>       } else {
>> >>>         input.skip(ftype);
>> >>>       }
>> >>>       break;
>> >>>       default:
>> >>>         input.skip(ftype);
>> >>>     }
>> >>>     input.readFieldEnd();
>> >>>   }
>> >>>   input.readStructEnd();
>> >>>   return;
>> >>> };
>> >>>
>> >>>
>> >>> and TGetResult is defined in ondb.thrift file as:
>> >>> typedef binary TVersion
>> >>> struct TGetResult
>> >>> {
>> >>>     1:TRow currentRow;
>> >>>     2:TVersion currentRowVersion;
>> >>> }
>> >>>
>> >>> I'm not sure if this is already been reported or if this is already
>> been
>> >>> fixed on the newest version.
>> >>>
>> >>> -Rod.O
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Sep 18, 2014 at 11:23 PM, Randy Abernethy <
>> >>> randy.abernethy@gmail.com> wrote:
>> >>>
>> >>>> Hey Rodolfo,
>> >>>>
>> >>>> While I can not say for sure not having seen your code, I would
be
>> highly
>> >>>> suspect of any Thift node lib prior to 0.9.2, which is not yet
>> released.
>> >>>> Try the current master, it has too many fixes to mention and will
>> likely
>> >>>> solve your problem. You don't need to change versions on any of
the
>> other
>> >>>> languages. Ping back if you still have trouble.
>> >>>>
>> >>>> Best,
>> >>>> Randy
>> >>>>
>> >>>> On Thursday, September 18, 2014, Rodolfo Ochoa <
>> rodolfo.ochoa@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> > on node:
>> >>>> > thrift@0.9.1
>> >>>> >
>> >>>> > on java:
>> >>>> > libthrift-0.9.1.jar
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Sep 18, 2014 at 9:09 PM, Randy Abernethy <
>> >>>> > randy.abernethy@gmail.com <javascript:;>>
>> >>>> > wrote:
>> >>>> >
>> >>>> > > What version of thrift are you using?
>> >>>> > > On Sep 18, 2014 7:46 PM, "Rodolfo Ochoa" <rodolfo.ochoa@gmail.com
>> >>>> > <javascript:;>> wrote:
>> >>>> > >
>> >>>> > > > I have a Java server sending info to C/Python/node.js
clients,
>> >>>> > > > I don't have issues with C or Python, only with node.js
>> >>>> > > > on Java side, I'm sending a byte array, and if I
print the
>> values
>> >>>> of
>> >>>> > this
>> >>>> > > > byte array I got:
>> >>>> > > > byte codes:  -84 -19 0 5 119 44 0 5 -50 108 0 44
75 73 78 -120
>> >>>> -103 -59
>> >>>> > > > -124 3 -92 -121 -98 -104 0 0 0 0 0 0 37 -128 1 3
0 0 0 1 0 0 0
>> 1 0
>> >>>> 0 0
>> >>>> > 0
>> >>>> > > 0
>> >>>> > > > -78 68 -89
>> >>>> > > >
>> >>>> > > > but on node.js side I receive:
>> >>>> > > > byte codes:  65533 65533 0 5 119 44 0 5 65533 108
0 44 75 73 78
>> >>>> 65533
>> >>>> > > 65533
>> >>>> > > > 324 3 65533 65533 65533 65533 0 0 0 0 0 0 37 65533
1 3 0 0 0 1
>> 0 0
>> >>>> 0 1
>> >>>> > 0
>> >>>> > > 0
>> >>>> > > > 0 0 0 65533 68 65533
>> >>>> > > >
>> >>>> > > > this means that some characters were replaced, why?
how?
>> >>>> > > > how can avoid this behavior?
>> >>>> > > >
>> >>>> > > >
>> >>>> > > >
>> >>>> > > > --
>> >>>> > > > *Rod.O*
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > *Rod.O*
>> >>>> >
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> *Rod.O*
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> *Rod.O*
>> >>
>> >
>> >
>> >
>> > --
>> > *Rod.O*
>>
>
>
>
> --
> *Rod.O*

Mime
View raw message