lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charlie Hull <char...@flax.co.uk>
Subject Re: External file fields
Date Fri, 02 Feb 2018 09:39:24 GMT
On 01/02/2018 18:55, Brian Yee wrote:
> Hello,
> 
> I want to use external file field to store frequently changing inventory and price data.
I got a proof of concept working with a mock text file and this will suit my needs.
> 
> What is the best way to keep this file updated in a fast way. Ideally I would like to
read changes from a Kafka queue and write to the file. But it seems like I would have to open
the whole file, read the whole file, find the line I want to change, and write the whole file
for every change. Is there a better way to do that? That approach seems like it would be difficult/slow
if the file is several million lines long.
> 
> Also, once I come up with a way to update the file quickly, what is the best way to distribute
the file to all the different solrcloud nodes in the correct directory?
> 
Another approach would be the XJoin plugin we wrote - if you wait a few 
days we should have an updated patch for Solr v6.5 and possibly v7. 
XJoin lets you filter/join/rank Solr results using an external data source.

http://www.flax.co.uk/blog/2016/01/25/xjoin-solr-part-1-filtering-using-price-discount-data/
http://www.flax.co.uk/blog/2016/01/29/xjoin-solr-part-2-click-example/


Cheers

Charlie


-- 
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Mime
View raw message