lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sangraal aiken" <sangr...@gmail.com>
Subject Re: Doc add limit
Date Thu, 27 Jul 2006 22:52:36 GMT
I think you're right... I will probably work on splitting the batches up
into smaller pieces at some point in the future. I think I will need the
capability to do large batches at some point though, so I want to make sure
the system can handle it. I also want to make sure this problem doesn't pop
up and bite me later.

-Sangraal

On 7/27/06, Mike Klaas <mike.klaas@gmail.com> wrote:
>
> Hi Sangraal:
>
> Sorry--I tried not to imply that this might affect your issue.  You
> may have to crank up the solr logging to determine where it is
> freezing (and what might be happening).
>
> It is certainly worth investigating why this occurs, but I wonder
> about the advantages of using such huge batches.  Assuming a few
> hundred bytes per document, 6100 docs produces a POST over 1MB in
> size.
>
> -Mike
>
> On 7/27/06, sangraal aiken <sangraal@gmail.com> wrote:
> > Mike,
> >  I've been posting with the content type set like this:
> >       conn.setRequestProperty("Content-Type",
> "application/octet-stream");
> >
> > I tried your suggestion though, and unfortunately there was no change.
> >       conn.setRequestProperty("Content-Type", "text/xml;
> charset=utf-8");
> >
> > -Sangraal
> >
> >
> > On 7/27/06, Mike Klaas <mike.klaas@gmail.com> wrote:
> > >
> > > On 7/27/06, Yonik Seeley <yonik@apache.org> wrote:
> > >
> > > > class SolrConnection:
> > > >   def __init__(self, host='localhost:8983', solrBase='/solr'):
> > > >     self.host = host
> > > >     self.solrBase = solrBase
> > > >     #a connection to the server is not opened at this point.
> > > >     self.conn = httplib.HTTPConnection(self.host)
> > > >     #self.conn.set_debuglevel(1000000)
> > > >     self.postheaders = {"Connection":"close"}
> > > >
> > > >   def doUpdateXML(self, request):
> > > >     try:
> > > >       self.conn.request('POST', self.solrBase+'/update', request,
> > > > self.postheaders)
> > >
> > > Disgressive note: I'm not sure if it is necessary with tomcat, but in
> > > my experience driving solr with python using Jetty, it was necessary
> > > to specify the content-type when posting utf-8 data:
> > >
> > > self.postheaders.update({'Content-Type': 'text/xml; charset=utf-8'})
> > >
> > > -Mike
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message