lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <>
Subject Re: Synonym-time Reindexing Issues
Date Thu, 07 Apr 2011 12:44:58 GMT
OK, see below.

On Wed, Apr 6, 2011 at 6:22 PM, Preston Marshall <>wrote:

> Reply Inline:
> On Apr 6, 2011, at 8:12 AM, Erick Erickson wrote:
> > Hmmm, this should work just fine. Here are my questions.
> >
> > 1> are you absolutely sure that the new synonym file
> >     is available when reindexing?
> Not sure what you mean here, solr is running as root, and the file is never
> moved around or anything crazy.

Just a sanity check that you're changing the indexing file you think you're
changing. I've sometimes managed to be in the wrong directory, on the wrong
machine, etc.

Hmmm, what happens if you just stop/start the server instead of delete the
index? I'm wondering if the old file is used (assuming *nix here). I have no
evidence this could be the  case but it's an idea.

> > 2> does the sunspot program do anything wonky with
> >     the ids? The documents
> >     will only be replaced if the IDs are identical.
> Is there a way I can add debugging to show what it's doing with the IDs or
> something to view the index?  I tried using Luke, but I can't get it to
> actually show me the actual data of the objects, only the name and some
> other basic info.

The issue is seeing whatever has been defined as the <uniqueKey> field. In
the default schema, it's defined as "id". I'm NOT talking about the internal
Lucene ID, it's entirely about what's defined in your schema. Set
stored="true" for fields to see them easily. The point here is that Solr
updates documents based on <uniqueKey>. If there is no such field,
reindexing your documents will simply add another copy, the original is
still searchable.

> > 3> are you sure that a commit is done at the end?
> It appears that it commits a few times during reindexing.
> > 4> What happens if you optimize? At that point, maxdocs
> >     and numdocs should be the same, and should be the count
> >     of documents. if they differ by a factor of 2, I'd suspect your
> >     id field isn't being used correctly.
> I'm unaware of what you mean by optimizing, or even viewing maxdocs and
> numdocs, but I will RTFM to find out.  I did notice something strange
> earlier though that may relate to this.  When I ran a search there were
> duplicate results.

OK, see the <uniqueKey> discussion above. It really sounds
like re-indexing the data is
merely adding documents again and again and again, not
replacing the first copy with the second. If this is true, your numDocs
and maxDocs should be nearly equal the first time and grow
by the number of documents you index every time you
reindex. If/when you <uniqueKey> is working, you should see
numDocs stay constant and maxDocs go up by the number
of documents you re-index.

Sending an optimize command to the indexer will reclaim all
unused resources and bring numDocs and maxDocs back
to the same value, but this is probably not your problem.

I do see that "id" is the <uniqueKey> in your schema. So I'm
guessing, especially because the comment says that this
field is used by sunspot, that the sunspot stuff is creating
a new id for each document when you re-index. If all this is
true, it's an issue with sunspot. So here's what I predict. If you
look at the id field you'll see some sunspot-generated id that's
unique for every added document even if it's a new copy
of an old document, so Solr sees two separate, entirely
unrelated documents. The old one has the old synonyms and
the new one the new list.

The maxDocs/numDocs are available on the admin page, click
the "statistics" link.


> >
> > If the hypothesis that you id field isn't working correctly, your number
> > of hits should be going up after re-indexing...
> >
> > If none of that is relevant, let us know what you find and we'll
> > try something else....
> >
> > Best
> > Erick
> >
> > On Tue, Apr 5, 2011 at 10:46 PM, Preston Marshall <
> >
> >> Hello all, I am having an issue with Solr and the SynonymFilterFactory.
>  I
> >> am using a library to interface with Solr called "sunspot."  I realize
> that
> >> is not what this list is for, but I believe this may be an issue with
> Solr,
> >> not the library (plus the lib author doesn't know the answer). I am
> using
> >> the SynonymFilterFactory in my index-time analyzer, and it works great.
>  My
> >> only problem is when it comes to changing the synonyms file.  I would
> expect
> >> to be able to edit the file, run a reindex (this is through the
> library),
> >> and have the new synonyms function when the reindex is complete.
> >> Unfortunately this is not the case, as changing the synonyms file
> doesn't
> >> actually affect the search results.  What DOES work is deleting the
> existing
> >> index, and starting from scratch.  This is unacceptable for my usage
> though,
> >> because I need the old index to remain online while the new one is being
> >> built, so there is no downtime.
> >>
> >> Here's my schema in case anyone needs it:
> >>
> >>
> >> Thanks,
> >> Preston
> >>
> >> P.S. Sorry if this dupes, first post and I didn't see it show up in the
> >> archives.
> >>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message