lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yonik Seeley <yo...@lucidimagination.com>
Subject Re: how to post(index) large file of 5 GB or greater than this
Date Thu, 08 Oct 2009 14:37:15 GMT
What is this huge file?  Solr XML? CSV?

Anyway, if it's a local file, you can get Solr to directly read/stream
it via stream.file
Examples in http://wiki.apache.org/solr/UpdateCSV
but it should work for any update format, not just CSV.

-Yonik
http://www.lucidimagination.com



On Thu, Oct 8, 2009 at 6:47 AM, Pravin Karne
<pravin_karne@persistent.co.in> wrote:
> Hi,
> I am new to solr. I am able to index, search and update with small size(around 500mb)
> But if I try to index file with 5 to 10 or more that (500mb) it gives memory heap exception.
> While investigation I found that post jar or post.sh load whole file in memory.
>
> I use one work around with dividing small file in small files..and it's working
>
> Is there any other way to post large file as above work around is not feasible for 1
TB file
>
> Thanks
> -Pravin
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property
of Persistent Systems Ltd. It is intended only for the use of the individual or entity to
which it is addressed. If you are not the intended recipient, you are not authorized to read,
retain, copy, print, distribute or use this message. If you have received this communication
in error, please notify the sender and delete all copies of this message. Persistent Systems
Ltd. does not accept any liability for virus infected mails.
>

Mime
View raw message