lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eloi Rocha <e...@melontech.com.br>
Subject Re: Process entire result set
Date Tue, 10 Aug 2010 12:28:20 GMT
Thanks Jonathan!

We decided to create offline results and store them in a Non-sql storage
(HBase). So we can answer the requests selecting one the the offline
generated results. This offline results are generated everyday.

Thanks!

Eloi

On Thu, Aug 5, 2010 at 8:59 PM, Jonathan Rochkind <rochkind@jhu.edu> wrote:

> Eloi Rocha wrote:
>
>> Hi everybody,
>>
>> I would like to know if does make sense to use Solr in the following
>> scenario:
>>  - search for large amount of data (like 1000, 10000, 100000 registers)
>>  - each register contains four or five fields (strings and integers)
>>  - every time will request for entire result set (I can paginate the
>> results). It would be much better to get all results at once [...]
>>
>>
>
> Depends on what kinds of searching you're doing. Are you doing searching
> that needs an indexer like Solr?  Then Solr is a good tool for your job.
>  Are you not, and you can do what you want just as easily in an rdbms or
> non-sql store like MongoDB? Then I wouldn't use Solr.
>
> Assuming you really do need Solr, I think this should work, but I would not
> store the actual stored fields in Solr, I'd store those fields in an
> external store (key-value store, rdbms, whatever).   You store only what you
> need to index in Solr, you do your search, you get ID's back.  You ask for
> the entire result set back, why not.  If you give Solr enough RAM, and set
> your cache settings appropriately (really big document and related caches),
> then I _think_ it should perform okay. One way to find out.
>
> What you'd get back is just ID's, then you'd look up that ID in your
> external store to get your actual fields you want to operate on. _May_ not
> be neccesary, maybe you could do it with solr stored fields, but making Solr
> do only exactly what you really need from it (an index) will maximize it's
> ability to do what you need in available RAM.
>
> If you don't need Solr/Lucene indexing/faceting behavior, and you can do
> just fine with an rdbms or non-sql store, use that.
>
> Jonathan
>



-- 
Eloi Rocha Neto
Melon Tech - http://melontech.com.br
+55 83 8868-7025

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message