thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jens Geyer" <jensge...@hotmail.com>
Subject Re: Return results list in batches
Date Fri, 28 Mar 2014 20:27:15 GMT
Hi Anand,

as I understand it, you want to

  a) change the service API, and
  b) apply a client-side wrapper which makes the service API changes 
transparent to the calling code

Sounds good.

The only caveat I see is to keep in mind that the iterator makes remote 
calls behind the scenes, thus may have a increased latency every N calls, 
which is not necessarily expected behaviour. If (and only if) that starts to 
become a problem, a possible idea could be to use a background thread who 
asynchronously fetches the next N results and caches them locally, when the 
iterator hits a certain threshold.

Have fun,
JensG



-----Urspr√ľngliche Nachricht----- 
From: Anand Nalya
Sent: Friday, March 28, 2014 10:59 AM
To: user@thrift.apache.org
Subject: Re: Return results list in batches

Hi Jens,

Thanks for the reply.

Another way of implementing this would be to write a wrapper over the public
List<Map<String,String>> getResults(String query) method with following
signature that will internally paginate the query.

public Iterator<List<Map<String,String>>> getResults(String query)

On next() call, this iterator will fetch next set of results from the
server.

Does that seems an appropriate approach or are there any caveats in that.

Regards,
Anand


On 22 March 2014 14:30, Jens Geyer <jensgeyer@hotmail.com> wrote:

> I'm afraid, no (or not yet). There are several things that need to be
> addressed when adding such a feature to Thrift.
>
> We had a similar issue here, but designed the service accordingly from the
> beginning to return only batches of the first/next N entries, because we
> were aware of that limitation.
>
> ________________________________
> Von: Anand Nalya
> Gesendet: 22.03.2014 06:12
> An: user@thrift.apache.org
> Betreff: Return results list in batches
>
> Hi,
>
> I've a thrift service that has the following signature:
>
> public List<Map<String,String>> getResults(String query)
>
> Since the result list can be a lot bigger than the RAM, I cannot create 
> the
> whole result list in memory before handing it off to thrift. Is there a 
> way
> in thrift to fill this resultant list in batches, say 100 items at a time
> without any changes on the client side.
>
> Regards,
> Anand
> 


Mime
View raw message