calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Van Besien <>
Subject Re: avatica streaming
Date Tue, 22 Sep 2015 07:42:02 GMT
"Streaming" might have been a bad choice of words, batches of rows is
actually what I meant.

I looked into it a bit more and noticed that it actually works with
1.4.0-incubating but not with 1.3.0-incubating. There is no real
reason why I was still on 1.3.0, so it is no longer an issue for me.

I am using JdbcMeta with the out-of-the-box


On Mon, Sep 21, 2015 at 8:39 PM, Julian Hyde <> wrote:
> A trusim is that if you look sufficiently closely at any stream it is revealed to be
batches. The Avatica API supports incremental fetches[1] as batches of rows (called “frames”)
and so both its JSON and Protobuf formats.
> There used to be a limitation for metadata result sets (e.g. getTables) that all of the
rows were squashed into the the first frame, however many there were. (See class MetaResultSet.)
I don’t recall whether that limitation still holds.
> For other result sets, you can implement the “fetch” method and start returning rows
as soon as you have some.
> If there is a particular implementation of Avatica server that buffers so many rows that
it runs out of memory then that is a bug and it would be helpful to see the stack trace.
> Julian
> [1]
>> On Sep 21, 2015, at 8:28 AM, Josh Elser <> wrote:
>> Hi Jan,
>> Cool! I hope you're having success using Avatica.
>> I would call it missing functionality in Avatica itself. IIRC, you'll get bundles
of 100 messages that come back in one HTTP response from the Avatica server.
>> It's been a while since I've looked at state of the art web-tech, but I remember
seeing some neat natives built into gRPC that support streaming on top of HTTP/2. I'm not
sure how best (in terms of client compatibility) to support streaming results back instead
of bundling and sending once a certain size is reached, but it's definitely an area that could
be improved! Would love to have a discussion on the matter.
>> - Josh
>> Jan Van Besien wrote:
>>> Hi,
>>> I was looking into Avatica to create a "thin" jdbc client for our
>>> existing "thick" jdbc client implemented with calcite. I got something
>>> working very quickly, very much similar to what Apache Phoenix has
>>> done.
>>> However, I immediately notice that there is no streaming between
>>> client and server for large result sets. In other words, if I execute
>>> a query which results in a large result set, the client has to wait a
>>> long time without any feedback and if the result set is large enough
>>> the server goes OOM.
>>> I am wondering if this functionality is simply missing from Avatica or
>>> whether there is some extra work required on my end to make it work.
>>> Thanks
>>> Jan

View raw message