lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hamed Ghavamnia <ghavamni...@gmail.com>
Subject Log indexing with lucene performance issues
Date Thu, 21 Apr 2016 15:01:30 GMT
Hello,

We've created a log management system using lucene 4.3. Each log has about
10 fields and all of them are stored.
We store each hour of the logs in a separate folder so when someone runs a
query only the folders specified in the time frame are searched.
The indexes are loaded using the mmap directory and the search is only done
on the required folders.
The problem we have is that when we run a query on three weeks of data
which has around 250M documents it takes around 200 seconds. Which is too
much.
We're using servers running virtualization and the storage is on SAN.
Is there anyway we can increase our performance? Can I distribute the
indexes on different servers and distribute the queries on them and
aggregate the results? By the way we have sorting on the queries. Has the
splitting the indexes in different folders created a problem? Is this
performance bad in the first place? Can I be expecting it to increase?
Sorry for the different questions.
Thanks for your help.
Hamed

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message