lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: heap size problem when indexinf files with solrj
Date Mon, 03 Oct 2011 12:35:22 GMT
You're saying that the file you're indexing is 500M? Pretty big...

First, I'd ask if you really want to index it as a single file or whether
you can break it up into sub-files. It depends upon what it is I guess.

Second, you can certainly index something this big, you just need
enough memory. Sounds like a 64-bit machine is in order.

Third, make sure you're committing before you try to index this
thing, so you're sure you have as many resources available
as possible.

Fourth, where is your error coming from? The SolrJ program which
I presume is running on your local machine or the Solr server
which is running where?

Best
Erick

On Fri, Sep 30, 2011 at 7:55 AM, hadi <md.anbari@gmail.com> wrote:
> I write a simple program with solrj that index files but after a minute
> passed it crashed and the
> *java.lang.OutOfmemoryError : java heap space* appear
>
> I used Eclipse and my memory storage is abou 2GB and i set the
> -Xms1024M-Xmx2048M for both my VM arg of tomcat and my application in Debug
> Configuration and uncomment the maxBufferedDocs in solrconfig and set it to
> 100 then run again my application but it crashed soon when it reach the
> files greater than 500MB
>
> is there any config to index large files with solrj?
> the detail my solrj is as below:
>
> String urlString = "http://localhost:8983/solr/file";
> CommonsHttpSolrServer solr = new CommonsHttpSolrServer(urlString);
>
> ContentStreamUpdateRequest req = new
> ContentStreamUpdateRequest("/update/extract");
>
> eq.addFile(file);
> req.setParam("literal.id", file.getAbsolutePath());
> req.setParam("literal.name", file.getName());
> req.setAction(ACTION.COMMIT, true, true);
>
> solr.request(req);
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/heap-size-problem-when-indexinf-files-with-solrj-tp3382115p3382115.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Mime
View raw message