lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jed Reynolds <li...@benrey.is-a-geek.net>
Subject Re: merely a suggestion: schema.xml validator or better schema validation logging
Date Sat, 03 Mar 2007 02:26:30 GMT
Yonik Seeley wrote:

> If the actual schema was null, then that was probably some problem
> parsing the schema.
> If that's the case, hopefully you saw an exception in the logs on 
> startup?


Using apache-solr-1.1.0-incubating.


Actually not at first, but now I do. But I've gone back and re-created 
the (or a similar) error, and what the problem was happened to be the 
way I was watching my logs. When I first started, I was just doing a 
tail -F on catalina.out, but the exception was throwing to  the logfile 
localhost.2007-03-01.log. Ah, tomcat my best old buddy old pal. I've 
learned to just do a "tail -F *". I've obviously grown desinsitized by 
other java projects throwing exceptions to logs, and by so much logging 
duplication between catalina.out and the tomcat  contextual logs.

I almost didn't notice the exception fly by because there's soooo much 
log output, and I can see why I might not have noticed. Yay for 
scrollback! (Hrm, I might not have wanted to watch logging for 4 
instances of solr all at once. Might explain why so much logging.)

Another helpful modification would be returning 500 errors codes in the 
header. This would help a script detect error codes without needing to 
grep or dom process the result element. The output of my php script to 
load documents was showing me the snippet below. Possibly making the 
error code configurable might help (I can see cases where forcing a 200 
response is useful) .



Array
(
    [errno] => 0
    [errstr] =>
    [response] => HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=UTF-8
Content-Length: 1329
Date: Sat, 03 Mar 2007 02:04:12 GMT
Connection: close

<result status="1">java.lang.NullPointerException
        at org.apache.solr.core.SolrCore.update(SolrCore.java:763)
        at 
org.apache.solr.servlet.SolrUpdateServlet.doPost(SolrUpdateServlet.java:53)
--snip--
</result>
)





> Anyway, I agree that some config errors could be handled in a more
> user-friendly manner, and it would be nice if config failures could
> make it to the front-page admin screen or something.


That would groovy!

I was able to see instances where a field was not defined. Now that I'm 
looking at all the log files, I'm seeing the error I should have seen 
earlier.

Thanks guys!

Jed

PS Last night I was able to index about 180,000 documents in about 2.5 
hours. The resulting index is a bit over 800M. Compared to my 
self-crawling with Nutch, this is 1/4 the time to index and 1/30th the 
disk space used by indicies. I am really impressed. I threw four 
concurrent scripts making 50,000 distinct (select distinct tag from 
taglist;) requests at this solr instance and my solr server was serving 
50 requests per second per script and the solr server load average was 
about 3.2. That's 200 requests per second against a 4 core box. The 
tomcat instance was taking 606M ram, resident.



Mime
View raw message