lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Glen Newton <glen.new...@gmail.com>
Subject Re: Having problems with the java api in 1.4.0
Date Tue, 24 Aug 2010 16:53:26 GMT
Liz,

I've built terrabyte (1-2 TB) test Lucene indexes, but have not
reached to the petabyte level, so I am not sure. Certainly there is
overhead in using the http and xml marshaling/de-marshaling, which may
or may not be a critical factor for you.

Could you give more information with respect to your application, i.e.
the nature of your data loading (i.e. many PB at once or GB per
hour/day/week accumulating to PB or MB per second/minute/hour
eventually accumulating to PB...;) searching ( i.e. the number of
fields indexed & the query complexity; if you are using facets, etc),
number of queries per second expected...

Lucene has a limit on the number of documents (in a single index) that
might impact your application:
http://lucene.apache.org/java/3_0_2/api/core/org/apache/lucene/index/IndexWriter.html#numDocs%28%29
of a 32bit int, 2 147 483 648.

-glen

On 24 August 2010 12:29, Liz Sommers <lizsworks@gmail.com> wrote:
> I was worried that it wouldn't scale.  We are going to be indexing petabytes
> of data.  Does the httpserver solution scale?
>
> Thanks
>
> Liz Sommers
> lizsworks@gmail.com
>
> On Tue, Aug 24, 2010 at 12:23 PM, Thomas Joiner
> <thomas.b.joiner@gmail.com>wrote:
>
>> Is there any reason you aren't using http://wiki.apache.org/solr/Solrj to
>> interact with Solr?
>>
>> On Tue, Aug 24, 2010 at 11:12 AM, Liz Sommers <lizsworks@gmail.com> wrote:
>>
>> > I am very new to the solr/lucene world.  I am using solr 1.4.0 and cannot
>> > move to 1.4.1.
>> >
>> > I have to index about 50 fields for each document, these fields are
>> already
>> > in key/value pairs by the time I get to my index methods.  I was able to
>> > index them with lucene without any problem, but found that I could not
>> then
>> > read the indexes with solr/admin.  So, I decided to use Solr for my
>> > indexing.
>> >
>> > The error I am currently getting is
>> > java.lang.RuntimeException: Can't find resource 'synonyms.txt' in
>> classpath
>> > or 'solr/conf'/'
>> >
>> > This exception is being thrown by SolrResourceLoader.openResource(line
>> > 260).
>> > which is called by IndexSchema<init> (line 102)
>> >
>> > My code that leads up to this follows:
>> >
>> > <code>
>> > String path = "c:/swdev/apache-solr-1.4.0/IDW"
>> > SolrConfig cfg new SolrConfig(path + "/solr/conf/solrconfig.xml");
>> > schema = new IndexSchema(cfg,path + "/solr/conf/schema.xml",null);
>> >
>> > </code>
>> >
>> > This also fails if I use
>> > schema = new IndexSchema(cfg,"schema.xml",null);
>> >
>> >
>> > Any help would be greatly appreciated.
>> >
>> > Thank you
>> >
>> > Liz Sommers
>> > lizsworks@gmail.com
>> >
>>
>



-- 

-

Mime
View raw message