lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bill Au" <bill.w...@gmail.com>
Subject Re: Snapshooting or replicating recently indexed data
Date Tue, 24 Apr 2007 01:50:27 GMT
Here's the Solr Wiki on collection distribution:

http://wiki.apache.org/solr/CollectionDistribution

It describes the "incremental" nature of the distribution:

A collection is a directory of many files. Collections are distributed
to the slaves as snapshots of these files. Each snapshot is made up of
hard links to the files so copying of the actual files is not
necessary when snapshots are created. Lucene only significantly
rewrites files following an optimization command. Generally, a file
once written, will change very little if at all. This makes the
underlying transport of rsync very useful. Files that have already
been transfered and have not changed do not need to be re-transferred
with the new edition of a collection.

Bill

On 4/21/07, Kevin Lewandowski <kevinsl@gmail.com> wrote:
> snapshooter does create incremental builds of the index. It doesn't
> appear so if you look at the contents because the existing files are
> hard links. But it is incremental.
>
> On 4/20/07, Doss <itsmedoss@gmail.com> wrote:
> > Hi Yonik,
> >
> > Thanks for your quick response, my question is this, can we take incremental
> > backup/replication in SOLR?
> >
> > Regards,
> > Doss.
> >
> >
> > M. MOHANDOSS Software Engineer Ext: 507 (A BharatMatrimony Enterprise)
> > ----- Original Message -----
> > From: "Yonik Seeley" <yonik@apache.org>
> > To: <solr-user@lucene.apache.org>
> > Sent: Thursday, April 19, 2007 7:42 PM
> > Subject: Re: Snapshooting or replicating recently indexed data
> >
> >
> > > On 4/19/07, Doss <itsmedoss@gmail.com> wrote:
> > >> It seems the snapshooter  takes the exact copy of the indexed data, that
> > >> is all the contents inside the index directory,  how can we take the
> > >> recently added once?
> > >> ...
> > >> cp -lr ${data_dir}/index ${temp}
> > >> mv ${temp} ${name} ...
> > >
> > >
> > > I don't quite understand your question, but since hard links are used,
> > > it's more like pointing to the index files instead of copying them.
> > > Rsync is used as a transport to only move the files that were changed
> > > from the master to slaves.
> > >
> > > -Yonik
> >
> >
>

Mime
View raw message