lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <>
Subject Re: paging vs streaming. spawn from (Processing a lot of results in Solr)
Date Sat, 27 Jul 2013 19:52:18 GMT

Let me briefly explain  the design

special RequestParser stores servlet output stream into the context

then special component injects special PostFilter/DelegatingCollector which
writes right into output

here is how it streams the doc, you see it's lazy enough

I mention that it disables later collectors
hence, no facets with streaming, yet as well as memory consumption.

This test shows how it works

all other code purposed for distributed search.

On Sat, Jul 27, 2013 at 4:44 PM, Roman Chyla <> wrote:

> Mikhail,
> If your solution gives lazy loading of solr docs /and thus streaming of
> huge result lists/ it should be big YES!
> Roman
> On 27 Jul 2013 07:55, "Mikhail Khludnev" <>
> wrote:
> > Otis,
> > You gave links to 'deep paging' when I asked about response streaming.
> > Let me understand. From my POV, deep paging is a special case for regular
> > search scenarios. We definitely need it in Solr. However, if we are
> talking
> > about data analytic like problems, when we need to select an "endless"
> > stream of responses (or store them in file as Roman did), 'deep paging'
> is
> > a suboptimal hack.
> > What's your vision on this?
> >

Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message