ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: How to reduce Scan Query execution time?
Date Thu, 30 Aug 2018 14:49:38 GMT
In this case I would not advise to measure how long does it take to get the
first byte. We may set page size to 1 and get that byte very quickly, but
it doesn't help you to iterate over all results quickly. Correct
measurement here would be to calculate total iteration time. Provided that
object size is pretty big, I think optimal page size might be pretty small.

At this moment thin client SCAN implementation is really straightforward -
just request-response for every page. To get peak throughput it is better
to rewrite it to streaming approach, when server constantly pushes data to
the client. Also compression may help in your case (trade CPU to network).
But this is only ideas which are yet to be implemented in the product.

Also you may want to look at "partition" and "filter" parameters of
ScanQuery. With partition it is possible to start scanning in several
threads. With filter it is possible to pass less data over wire.

Vladimir.

On Thu, Aug 30, 2018 at 5:40 PM dmitrievanthony <dmitrievanthony@gmail.com>
wrote:

> To be precise it's not only about first page, it's about getting next pages
> as well.
>
> Regarding use case, in my client application I need to iterate over the
> dataset stored in Apache Ignite as fast as it possible. It means I should
> provide maximal throughput for simple "read all" operation.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message