Thanks Samarth and Jesse.So, in effect setting the batch size (say, stmt.setFetchSize()) ensures that only that much data is copied over the wire en-mass? And 'behind-the-scenes', Phoenix driver would fetch next batch as each fetch size is exhausted?Thanks,SumitTo add to what Jesse said, you can override the default scanner fetch size programmatically via Phoenix by calling statement.setFetchSize(int).
On Tuesday, October 6, 2015, Jesse Yates <firstname.lastname@example.org> wrote:So HBase (and by extension, Phoenix) does not do true "streaming" of rows - rows are copied into memory from the HFiles and then eventually copied en-mass onto the wire. On the client they are pulled off in chunks and paged through by the client scanner. You can control the batch size (amount of rows in each 'page') via the usual HBase client configurationsOn Tue, Oct 6, 2015 at 8:38 AM Sumit Nigam <email@example.com> wrote:Hi,Does Phoenix buffer the result set internally? I mean when I fire a huge skip scan IN clause, then the data being returned may be too huge to contain in memory. So, ideally I'd like to stream data through the resultset.next() method. So, my question is does Phoenix really stream results?And if so, is there a way to control how much is loads in one time in client side before its next() fetches next batch of data from region servers to client?Best regards,Sumit