lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Runo <mr...@zappos.com>
Subject Re: Replication broken.. no helpful errors?
Date Thu, 06 Sep 2007 18:54:25 GMT
The thing is that a new searcher is not opened if I look in the  
stats.jsp page. The index version never changes.

When I run..

sudo /opt/solr/bin/commit -V -u tomcat5

..I get a new searcher opened, but even though it (in theory)  
installed the new index, I see no docs in there. During the  
snapinstaller...

+ echo 2007/09/06 11:43:49 command: /opt/solr/bin/snapinstaller -M  
search1 -S /opt/solr/logs -d /opt/solr/data -V -u tomcat5
+ [[ -n '' ]]
++ ls /opt/solr/data
++ grep 'snapshot\.'
++ grep -v wip
++ sort -r
++ head -1
+ name=temp-snapshot.20070905150504
+ trap 'echo "caught INT/TERM, exiting now but partial installation  
may have already occured";/bin/rm -rf ${data_dir"/index.tmp$$;logExit  
aborted 13' INT TERM
+ [[ temp-snapshot.20070905150504 == '' ]]
+ name=/opt/solr/data/temp-snapshot.20070905150504
++ cat /opt/solr/logs/snapshot.current


...it would seem that snappuller might not be properly setting the  
directory name - or should it be temp-*?

I had replication working for a few weeks, and then it broke, and has  
been down since. We're going live with this project in about a week,  
and I really need to get this going before then =p

+--------------------------------------------------------+
  | Matthew Runo
  | Zappos Development
  | mruno@zappos.com
  | 702-943-7833
+--------------------------------------------------------+


On Sep 6, 2007, at 6:01 AM, Bill Au wrote:

> The snapinstaller script opens a new searcher by calling commit.   
> From the
> attached debug output it looks like that actually worked:
>
> + /opt/solr/bin/commit
> + [[ 0 != 0 ]]
> + logExit ended 0
>
> Try running the /opt/solr/bin/commit directly with the -V option.
>
> Bill
>
> On 9/5/07, Matthew Runo <mruno@zappos.com> wrote:
>>
>> If it helps anyone, this index is around a gig in size.
>>
>> +--------------------------------------------------------+
>>   | Matthew Runo
>>   | Zappos Development
>>   | mruno@zappos.com
>>   | 702-943-7833
>> +--------------------------------------------------------+
>>
>>
>> On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote:
>>
>>> It seems that the scripts cannot open new searchers at the end of
>>> the process, for some reason. Here's a message from cron, but I'm
>>> not sure what to make of it... It looks like the files properly
>>> copied over, but failed the install. I removed the temp* directory,
>>> but still SOLR could not launch a new searcher. I don't see any
>>> activity in catalina.out though...
>>>
>>>
>>> started by tomcat5
>>> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
>>> data -S /opt/solr/logs -d /opt/solr/data -v
>>> pulling snapshot temp-snapshot.20070905150504
>>> receiving file list ... done
>>> deleting segments_1ine
>>> deleting _164h_1.del
>>> deleting _164h.tis
>>> deleting _164h.tii
>>> deleting _164h.prx
>>> deleting _164h.nrm
>>> deleting _164h.frq
>>> deleting _164h.fnm
>>> deleting _164h.fdx
>>> deleting _164h.fdt
>>> deleting _164g_1.del
>>> deleting _164g.tis
>>> deleting _164g.tii
>>> deleting _164g.prx
>>> deleting _164g.nrm
>>> deleting _164g.frq
>>> deleting _164g.fnm
>>> deleting _164g.fdx
>>> deleting _164g.fdt
>>> deleting _164f_1.del
>>> deleting _164f.tis
>>> deleting _164f.tii
>>> deleting _164f.prx
>>> deleting _164f.nrm
>>> deleting _164f.frq
>>> deleting _164f.fnm
>>> deleting _164f.fdx
>>> deleting _164f.fdt
>>> deleting _164e_1.del
>>> deleting _164e.tis
>>> deleting _164e.tii
>>> deleting _164e.prx
>>> deleting _164e.nrm
>>> deleting _164e.frq
>>> deleting _164e.fnm
>>> deleting _164e.fdx
>>> deleting _164e.fdt
>>> deleting _164d_1.del
>>> deleting _164d.tis
>>> deleting _164d.tii
>>> deleting _164d.prx
>>> deleting _164d.nrm
>>> deleting _164d.frq
>>> deleting _164d.fnm
>>> deleting _164d.fdx
>>> deleting _164d.fdt
>>> deleting _164c_1.del
>>> deleting _164c.tis
>>> deleting _164c.tii
>>> deleting _164c.prx
>>> deleting _164c.nrm
>>> deleting _164c.frq
>>> deleting _164c.fnm
>>> deleting _164c.fdx
>>> deleting _164c.fdt
>>> deleting _164b_1.del
>>> deleting _164b.tis
>>> deleting _164b.tii
>>> deleting _164b.prx
>>> deleting _164b.nrm
>>> deleting _164b.frq
>>> deleting _164b.fnm
>>> deleting _164b.fdx
>>> deleting _164b.fdt
>>> deleting _164a_1.del
>>> deleting _164a.tis
>>> deleting _164a.tii
>>> deleting _164a.prx
>>> deleting _164a.nrm
>>> deleting _164a.frq
>>> deleting _164a.fnm
>>> deleting _164a.fdx
>>> deleting _164a.fdt
>>> deleting _163z_3.del
>>> deleting _163z.tis
>>> deleting _163z.tii
>>> deleting _163z.prx
>>> deleting _163z.nrm
>>> deleting _163z.frq
>>> deleting _163z.fnm
>>> deleting _163z.fdx
>>> deleting _163z.fdt
>>> deleting _163o_3.del
>>> deleting _163o.tis
>>> deleting _163o.tii
>>> deleting _163o.prx
>>> deleting _163o.nrm
>>> deleting _163o.frq
>>> deleting _163o.fnm
>>> deleting _163o.fdx
>>> deleting _163o.fdt
>>> deleting _163d_4.del
>>> deleting _163d.tis
>>> deleting _163d.tii
>>> deleting _163d.prx
>>> deleting _163d.nrm
>>> deleting _163d.frq
>>> deleting _163d.fnm
>>> deleting _163d.fdx
>>> deleting _163d.fdt
>>> deleting _1632_6.del
>>> deleting _1632.tis
>>> deleting _1632.tii
>>> deleting _1632.prx
>>> deleting _1632.nrm
>>> deleting _1632.frq
>>> deleting _1632.fnm
>>> deleting _1632.fdx
>>> deleting _1632.fdt
>>> deleting _162r_7.del
>>> deleting _162r.tis
>>> deleting _162r.tii
>>> deleting _162r.prx
>>> deleting _162r.nrm
>>> deleting _162r.frq
>>> deleting _162r.fnm
>>> deleting _162r.fdx
>>> deleting _162r.fdt
>>> deleting _162g_d.del
>>> deleting _162g.tis
>>> deleting _162g.tii
>>> deleting _162g.prx
>>> deleting _162g.nrm
>>> deleting _162g.frq
>>> deleting _162g.fnm
>>> deleting _162g.fdx
>>> deleting _162g.fdt
>>> deleting _1625_m.del
>>> deleting _1625.tis
>>> deleting _1625.tii
>>> deleting _1625.prx
>>> deleting _1625.nrm
>>> deleting _1625.frq
>>> deleting _1625.fnm
>>> deleting _1625.fdx
>>> deleting _1625.fdt
>>> deleting _161u_w.del
>>> deleting _161u.tis
>>> deleting _161u.tii
>>> deleting _161u.prx
>>> deleting _161u.nrm
>>> deleting _161u.frq
>>> deleting _161u.fnm
>>> deleting _161u.fdx
>>> deleting _161u.fdt
>>> deleting _161j_16.del
>>> ./
>>> _161j_17.del
>>> _164m.fdt
>>> _164m.fdx
>>> _164m.fnm
>>> _164m.frq
>>> _164m.nrm
>>> _164m.prx
>>> _164m.tii
>>> _164m.tis
>>> _164m_1.del
>>> _164x.fdt
>>> _164x.fdx
>>> _164x.fnm
>>> _164x.frq
>>> _164x.nrm
>>> _164x.prx
>>> _164x.tii
>>> _164x.tis
>>> _164x_1.del
>>> segments.gen
>>> segments_1inv
>>>
>>> sent 516 bytes  received 105864302 bytes  30247090.86 bytes/sec
>>> total size is 966107226  speedup is 9.13
>>> + [[ -z search1 ]]
>>> + [[ -z /opt/solr/logs ]]
>>> + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
>>> + [[ -z tomcat5 ]]
>>> ++ whoami
>>> + [[ tomcat5 != tomcat5 ]]
>>> ++ who -m
>>> ++ cut '-d ' -f1
>>> ++ sed '-es/^.*!//'
>>> + oldwhoami=
>>> + [[ '' == '' ]]
>>> +++ pgrep -g0 snapinstaller
>>> ++ tail -1
>>> ++ cut -f1 '-d '
>>> ++ ps h -Hfp 3621 3629 3630 3631
>>> + oldwhoami=tomcat5
>>> + [[ -z /opt/solr/data ]]
>>> ++ echo /opt/solr/data
>>> ++ cut -c1
>>> + [[ / != \/ ]]
>>> ++ echo /opt/solr/logs
>>> ++ cut -c1
>>> + [[ / != \/ ]]
>>> ++ date +%s
>>> + start=1189030205
>>> + logMessage started by tomcat5
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo 2007/09/05 15:10:05 started by tomcat5
>>> + [[ -n '' ]]
>>> + logMessage command: /opt/solr/bin/snapinstaller -M search1 -S /
>>> opt/solr/logs -d /opt/solr/data -V
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M
>>> search1 -S /opt/solr/logs -d /opt/solr/data -V
>>> + [[ -n '' ]]
>>> ++ ls /opt/solr/data
>>> ++ grep 'snapshot\.'
>>> ++ grep -v wip
>>> ++ sort -r
>>> ++ head -1
>>> + name=temp-snapshot.20070905150504
>>> + trap 'echo "caught INT/TERM, exiting now but partial installation
>>> may have already occured";/bin/rm -rf ${data_dir"/index.tmp$
>>> $;logExit aborted 13' INT TERM
>>> + [[ temp-snapshot.20070905150504 == '' ]]
>>> + name=/opt/solr/data/temp-snapshot.20070905150504
>>> ++ cat /opt/solr/logs/snapshot.current
>>> + [[ /opt/solr/data/temp-snapshot.20070905150504 == /opt/solr/data/
>>> snapshot.20070905150005 ]]
>>> + ssh -o StrictHostKeyChecking=no search1 mkdir -p /opt/solr/logs
>>> + logMessage installing snapshot /opt/solr/data/temp-snapshot.
>>> 20070905150504
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo 2007/09/05 15:10:05 installing snapshot /opt/solr/data/temp-
>>> snapshot.20070905150504
>>> + [[ -n '' ]]
>>> + cp -lr /opt/solr/data/temp-snapshot.20070905150504/ /opt/solr/
>>> data/index.tmp3621
>>> + /bin/rm -rf /opt/solr/data/index
>>> + mv -f /opt/solr/data/index.tmp3621 /opt/solr/data/index
>>> + echo /opt/solr/data/temp-snapshot.20070905150504
>>> ++ uname -n
>>> + scp -q -o StrictHostKeyChecking=no /opt/solr/logs/
>>> snapshot.current search1:/opt/solr/logs/snapshot.current.search2
>>> + logMessage notifing Solr to open a new Searcher
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo 2007/09/05 15:10:06 notifing Solr to open a new Searcher
>>> + [[ -n '' ]]
>>> + /opt/solr/bin/commit
>>> + [[ 0 != 0 ]]
>>> + logExit ended 0
>>> ++ date +%s
>>> + end=1189030210
>>> ++ expr 1189030210 - 1189030205
>>> + diff=5
>>> ++ timeStamp
>>> ++ date '+%Y/%m/%d %H:%M:%S'
>>> + echo '2007/09/05 15:10:10 ended (elapsed time: 5 sec)'
>>> + exit 0
>>>
>>>
>>>
>>> +--------------------------------------------------------+
>>>  | Matthew Runo
>>>  | Zappos Development
>>>  | mruno@zappos.com
>>>  | 702-943-7833
>>> +--------------------------------------------------------+
>>>
>>>
>>> On Aug 31, 2007, at 6:11 PM, Bill Au wrote:
>>>
>>>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113  
>>>>> already
>>>>> installed
>>>>
>>>> It looks like you have a directory named temp-snapshot. 
>>>> 20070816120113
>>>> in your data directory.  You should remove it.  One of the other
>>>> script might have left that behind somehow.
>>>>
>>>> I will update the snapinstaller script to ignore non-snapshot when
>>>> looking for the latest snapshot to install.
>>>>
>>>> Bill
>>>>
>>>> On 8/31/07, Matthew Runo <mruno@zappos.com> wrote:
>>>>> Hello!
>>>>>
>>>>> On a somewhat related note, our replication seems very much  
>>>>> broken.
>>>>> I've added -v to all my cron jobs, and I think I've seen the error
>>>>> (below).
>>>>>
>>>>> As you can see, it's rsyncing an updated index, but then doesn't
>>>>> seem
>>>>> to know to install it. I'm not sure why though.. no errors are
>>>>> reported anywhere via the -v. Any help would be most appreciated,
>>>>> I'm
>>>>> sure I'm just missing something.   You can see the cronjob
>>>>> command in
>>>>> the subject of the forwarded message.
>>>>>
>>>>> +--------------------------------------------------------+
>>>>>   | Matthew Runo
>>>>>   | Zappos Development
>>>>>   | mruno@zappos.com
>>>>>   | 702-943-7833
>>>>> +--------------------------------------------------------+
>>>>>
>>>>>
>>>>> Begin forwarded message:
>>>>>
>>>>>> From: root@web03.zappos.com (Cron Daemon)
>>>>>> Date: August 31, 2007 1:02:36 PM PDT
>>>>>> To: mruno@zappos.com
>>>>>> Subject: Cron <tomcat5@search2> /opt/solr/bin/snappuller -M
 
>>>>>> search1
>>>>>> -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data  
>>>>>> -v;/
>>>>>> opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
>>>>>> solr/data -v
>>>>>>
>>>>>> started by tomcat5
>>>>>> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/ 
>>>>>> solr/
>>>>>> data -S /opt/solr/logs -d /opt/solr/data -v
>>>>>> pulling snapshot snapshot.20070831130005
>>>>>> receiving file list ... done
>>>>>> deleting segments_19zm
>>>>>> deleting _10bg.tis
>>>>>> deleting _10bg.tii
>>>>>> deleting _10bg.prx
>>>>>> deleting _10bg.nrm
>>>>>> deleting _10bg.frq
>>>>>> deleting _10bg.fnm
>>>>>> deleting _10bg.fdx
>>>>>> deleting _10bg.fdt
>>>>>> ./
>>>>>> _14ff.fdt
>>>>>> _14ff.fdx
>>>>>> _14ff.fnm
>>>>>> _14ff.frq
>>>>>> _14ff.nrm
>>>>>> _14ff.prx
>>>>>> _14ff.tii
>>>>>> _14ff.tis
>>>>>> _14ff_9.del
>>>>>> _14fq.fdt
>>>>>> _14fq.fdx
>>>>>> _14fq.fnm
>>>>>> _14fq.frq
>>>>>> _14fq.nrm
>>>>>> _14fq.prx
>>>>>> _14fq.tii
>>>>>> _14fq.tis
>>>>>> _14fq_3.del
>>>>>> _14fr.fdt
>>>>>> _14fr.fdx
>>>>>> _14fr.fnm
>>>>>> _14fr.frq
>>>>>> _14fr.nrm
>>>>>> _14fr.prx
>>>>>> _14fr.tii
>>>>>> _14fr.tis
>>>>>> _14fr_2.del
>>>>>> _14fs.fdt
>>>>>> _14fs.fdx
>>>>>> _14fs.fnm
>>>>>> _14fs.frq
>>>>>> _14fs.nrm
>>>>>> _14fs.prx
>>>>>> _14fs.tii
>>>>>> _14fs.tis
>>>>>> _14fs_1.del
>>>>>> segments.gen
>>>>>> segments_1fza
>>>>>> write.lock
>>>>>>
>>>>>> sent 871 bytes  received 843185604 bytes  24440187.68 bytes/sec
>>>>>> total size is 843080453  speedup is 1.00
>>>>>> started by tomcat5
>>>>>> command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/ 
>>>>>> logs -
>>>>>> d /opt/solr/data -v
>>>>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113  
>>>>>> already
>>>>>> installed
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>


Mime
View raw message