calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From F21 <f21.gro...@gmail.com>
Subject Re: How do I send binary data to avatica?
Date Thu, 14 Apr 2016 22:45:21 GMT
Hey Josh,

Here's what I am doing:

Create the table: CREATE TABLE test ( int INTEGER PRIMARY KEY, bin 
VARBINARY) TRANSACTIONAL=false
Download the binary file we want to insert from here: 
https://raw.githubusercontent.com/golang-samples/gopher-vector/master/gopher.png

Prepare the statement: UPSERT INTO test VALUES (?, ?)

Here's the code I am using to execute the statement:

// read the file into a byte array []byte
file, _ := ioutil.ReadFile(filePath)

// create array of typed values containing the parameters
parameters := []*message.TypedValue{
    &message.TypedValue{
         Type: message.Rep_LONG,
         NumberValue: 1,
     },
    &message.TypedValue{
         Type: message.Rep_BYTE_STRING,
         BytesValues: file,
     },
}

// create message:
msg := &message.ExecuteRequest{
         StatementHandle:    &s.handle, // this is just the statement handle
         ParameterValues:    parameters,
         MaxRowCount:        100,
         HasParameterValues: true,
}

// encode the message:
wrapped, _ := proto.Marshal(message)

// wrap it in the wire message:
wire := &message.WireMessage{
         Name:           " org.apache.calcite.avatica.proto.Requests$ 
ExecuteRequest",
         WrappedMessage: wrapped,
}

// encode wire message and send over http:
body, _ := proto.Marshal(wire)
response, _ := ctxhttp.Post(context.Background(), httpClient, 
avaticaServer, "application/x-google-protobuf", bytes.NewReader(body))

Let me know if there's more information I can provide or if something's 
unclear :)

On 15/04/2016 2:46 AM, Josh Elser wrote:
> Yeah, that sounds right to me too.
>
> I think we have a test for random bytes. Maybe there's something weird 
> happening under the hood in the Avatica JDBC driver that isn't obvious 
> to you in the Go driver.
>
> Any chance you can share some example code you're running? I can try 
> to convert it to a Java test case, maybe help track down your issue.
>
> F21 wrote:
>> I also tried casting the data to a string and setting it to StringValue
>> and the Rep type to STRING.
>>
>> This works when I store and retrieve strings from the binary column, but
>> doesn't work correctly if I try to store something like a small image.
>>
>> On 14/04/2016 5:03 PM, Julian Hyde wrote:
>>> BytesValue sounds right. I’m not sure why it isn’t working for you.
>>>
>>>> On Apr 14, 2016, at 6:34 AM, F21 <f21.groups@gmail.com> wrote:
>>>>
>>>> As mentioned on the list, I am currently working on a golang
>>>> client/driver for avatica using protobufs for serialization.
>>>>
>>>> I've got all datatypes working, except for BINARY and VARBINARY.
>>>>
>>>> For my test table looks like this:
>>>> CREATE TABLE test (int INTEGER PRIMARY KEY, bin BINARY(20), varbin
>>>> VARBINARY) TRANSACTIONAL=false
>>>>
>>>> In go, we have a datatype called a slice of bytes ([]byte) which is
>>>> essentially an array of bytes (8-bits each).
>>>>
>>>> When I generated the golang protobufs using the .proto files, this is
>>>> the definition of TypedValue:
>>>> type TypedValue struct {
>>>> Type Rep `protobuf:"varint,1,opt,name=type,enum=Rep"
>>>> json:"type,omitempty"`
>>>> BoolValue bool
>>>> `protobuf:"varint,2,opt,name=bool_value,json=boolValue"
>>>> json:"bool_value,omitempty"`
>>>> StringValue string
>>>> `protobuf:"bytes,3,opt,name=string_value,json=stringValue"
>>>> json:"string_value,omitempty"`
>>>> NumberValue int64
>>>> `protobuf:"zigzag64,4,opt,name=number_value,json=numberValue"
>>>> json:"number_value,omitempty"`
>>>> BytesValues []byte
>>>> `protobuf:"bytes,5,opt,name=bytes_values,json=bytesValues,proto3"
>>>> json:"bytes_values,omitempty"`
>>>> DoubleValue float64
>>>> `protobuf:"fixed64,6,opt,name=double_value,json=doubleValue"
>>>> json:"double_value,omitempty"`
>>>> Null bool `protobuf:"varint,7,opt,name=null" json:"null,omitempty"`
>>>> }
>>>>
>>>> I am currently creating a TypedValue that looks like this when
>>>> sending binary data:
>>>> {BYTE_STRING false 0 [116 101 115 116] 0 false}
>>>>
>>>> So, the Rep is set to BYTE_STRING and ByteValues is populated with
>>>> the string "test" in bytes (it's shown here as decimal because I
>>>> printed it).
>>>>
>>>> The problem is that even though the insert executes properly, if I
>>>> look at the row using SquirrelSQL, both the BINARY and VARBINARY
>>>> columns are <null>.
>>>>
>>>> Is BYTE_STRING the correct rep type for sending binary data? Do I
>>>> also have to encode my bytes in a special format?
>>>>
>>>> Thanks!
>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message