lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Runo <mr...@zappos.com>
Subject Re: Replication broken.. no helpful errors?
Date Wed, 05 Sep 2007 22:30:14 GMT
If it helps anyone, this index is around a gig in size.

+--------------------------------------------------------+
  | Matthew Runo
  | Zappos Development
  | mruno@zappos.com
  | 702-943-7833
+--------------------------------------------------------+


On Sep 5, 2007, at 3:14 PM, Matthew Runo wrote:

> It seems that the scripts cannot open new searchers at the end of  
> the process, for some reason. Here's a message from cron, but I'm  
> not sure what to make of it... It looks like the files properly  
> copied over, but failed the install. I removed the temp* directory,  
> but still SOLR could not launch a new searcher. I don't see any  
> activity in catalina.out though...
>
>
> started by tomcat5
> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/ 
> data -S /opt/solr/logs -d /opt/solr/data -v
> pulling snapshot temp-snapshot.20070905150504
> receiving file list ... done
> deleting segments_1ine
> deleting _164h_1.del
> deleting _164h.tis
> deleting _164h.tii
> deleting _164h.prx
> deleting _164h.nrm
> deleting _164h.frq
> deleting _164h.fnm
> deleting _164h.fdx
> deleting _164h.fdt
> deleting _164g_1.del
> deleting _164g.tis
> deleting _164g.tii
> deleting _164g.prx
> deleting _164g.nrm
> deleting _164g.frq
> deleting _164g.fnm
> deleting _164g.fdx
> deleting _164g.fdt
> deleting _164f_1.del
> deleting _164f.tis
> deleting _164f.tii
> deleting _164f.prx
> deleting _164f.nrm
> deleting _164f.frq
> deleting _164f.fnm
> deleting _164f.fdx
> deleting _164f.fdt
> deleting _164e_1.del
> deleting _164e.tis
> deleting _164e.tii
> deleting _164e.prx
> deleting _164e.nrm
> deleting _164e.frq
> deleting _164e.fnm
> deleting _164e.fdx
> deleting _164e.fdt
> deleting _164d_1.del
> deleting _164d.tis
> deleting _164d.tii
> deleting _164d.prx
> deleting _164d.nrm
> deleting _164d.frq
> deleting _164d.fnm
> deleting _164d.fdx
> deleting _164d.fdt
> deleting _164c_1.del
> deleting _164c.tis
> deleting _164c.tii
> deleting _164c.prx
> deleting _164c.nrm
> deleting _164c.frq
> deleting _164c.fnm
> deleting _164c.fdx
> deleting _164c.fdt
> deleting _164b_1.del
> deleting _164b.tis
> deleting _164b.tii
> deleting _164b.prx
> deleting _164b.nrm
> deleting _164b.frq
> deleting _164b.fnm
> deleting _164b.fdx
> deleting _164b.fdt
> deleting _164a_1.del
> deleting _164a.tis
> deleting _164a.tii
> deleting _164a.prx
> deleting _164a.nrm
> deleting _164a.frq
> deleting _164a.fnm
> deleting _164a.fdx
> deleting _164a.fdt
> deleting _163z_3.del
> deleting _163z.tis
> deleting _163z.tii
> deleting _163z.prx
> deleting _163z.nrm
> deleting _163z.frq
> deleting _163z.fnm
> deleting _163z.fdx
> deleting _163z.fdt
> deleting _163o_3.del
> deleting _163o.tis
> deleting _163o.tii
> deleting _163o.prx
> deleting _163o.nrm
> deleting _163o.frq
> deleting _163o.fnm
> deleting _163o.fdx
> deleting _163o.fdt
> deleting _163d_4.del
> deleting _163d.tis
> deleting _163d.tii
> deleting _163d.prx
> deleting _163d.nrm
> deleting _163d.frq
> deleting _163d.fnm
> deleting _163d.fdx
> deleting _163d.fdt
> deleting _1632_6.del
> deleting _1632.tis
> deleting _1632.tii
> deleting _1632.prx
> deleting _1632.nrm
> deleting _1632.frq
> deleting _1632.fnm
> deleting _1632.fdx
> deleting _1632.fdt
> deleting _162r_7.del
> deleting _162r.tis
> deleting _162r.tii
> deleting _162r.prx
> deleting _162r.nrm
> deleting _162r.frq
> deleting _162r.fnm
> deleting _162r.fdx
> deleting _162r.fdt
> deleting _162g_d.del
> deleting _162g.tis
> deleting _162g.tii
> deleting _162g.prx
> deleting _162g.nrm
> deleting _162g.frq
> deleting _162g.fnm
> deleting _162g.fdx
> deleting _162g.fdt
> deleting _1625_m.del
> deleting _1625.tis
> deleting _1625.tii
> deleting _1625.prx
> deleting _1625.nrm
> deleting _1625.frq
> deleting _1625.fnm
> deleting _1625.fdx
> deleting _1625.fdt
> deleting _161u_w.del
> deleting _161u.tis
> deleting _161u.tii
> deleting _161u.prx
> deleting _161u.nrm
> deleting _161u.frq
> deleting _161u.fnm
> deleting _161u.fdx
> deleting _161u.fdt
> deleting _161j_16.del
> ./
> _161j_17.del
> _164m.fdt
> _164m.fdx
> _164m.fnm
> _164m.frq
> _164m.nrm
> _164m.prx
> _164m.tii
> _164m.tis
> _164m_1.del
> _164x.fdt
> _164x.fdx
> _164x.fnm
> _164x.frq
> _164x.nrm
> _164x.prx
> _164x.tii
> _164x.tis
> _164x_1.del
> segments.gen
> segments_1inv
>
> sent 516 bytes  received 105864302 bytes  30247090.86 bytes/sec
> total size is 966107226  speedup is 9.13
> + [[ -z search1 ]]
> + [[ -z /opt/solr/logs ]]
> + fixUser -M search1 -S /opt/solr/logs -d /opt/solr/data -V
> + [[ -z tomcat5 ]]
> ++ whoami
> + [[ tomcat5 != tomcat5 ]]
> ++ who -m
> ++ cut '-d ' -f1
> ++ sed '-es/^.*!//'
> + oldwhoami=
> + [[ '' == '' ]]
> +++ pgrep -g0 snapinstaller
> ++ tail -1
> ++ cut -f1 '-d '
> ++ ps h -Hfp 3621 3629 3630 3631
> + oldwhoami=tomcat5
> + [[ -z /opt/solr/data ]]
> ++ echo /opt/solr/data
> ++ cut -c1
> + [[ / != \/ ]]
> ++ echo /opt/solr/logs
> ++ cut -c1
> + [[ / != \/ ]]
> ++ date +%s
> + start=1189030205
> + logMessage started by tomcat5
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo 2007/09/05 15:10:05 started by tomcat5
> + [[ -n '' ]]
> + logMessage command: /opt/solr/bin/snapinstaller -M search1 -S / 
> opt/solr/logs -d /opt/solr/data -V
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo 2007/09/05 15:10:05 command: /opt/solr/bin/snapinstaller -M  
> search1 -S /opt/solr/logs -d /opt/solr/data -V
> + [[ -n '' ]]
> ++ ls /opt/solr/data
> ++ grep 'snapshot\.'
> ++ grep -v wip
> ++ sort -r
> ++ head -1
> + name=temp-snapshot.20070905150504
> + trap 'echo "caught INT/TERM, exiting now but partial installation  
> may have already occured";/bin/rm -rf ${data_dir"/index.tmp$ 
> $;logExit aborted 13' INT TERM
> + [[ temp-snapshot.20070905150504 == '' ]]
> + name=/opt/solr/data/temp-snapshot.20070905150504
> ++ cat /opt/solr/logs/snapshot.current
> + [[ /opt/solr/data/temp-snapshot.20070905150504 == /opt/solr/data/ 
> snapshot.20070905150005 ]]
> + ssh -o StrictHostKeyChecking=no search1 mkdir -p /opt/solr/logs
> + logMessage installing snapshot /opt/solr/data/temp-snapshot. 
> 20070905150504
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo 2007/09/05 15:10:05 installing snapshot /opt/solr/data/temp- 
> snapshot.20070905150504
> + [[ -n '' ]]
> + cp -lr /opt/solr/data/temp-snapshot.20070905150504/ /opt/solr/ 
> data/index.tmp3621
> + /bin/rm -rf /opt/solr/data/index
> + mv -f /opt/solr/data/index.tmp3621 /opt/solr/data/index
> + echo /opt/solr/data/temp-snapshot.20070905150504
> ++ uname -n
> + scp -q -o StrictHostKeyChecking=no /opt/solr/logs/ 
> snapshot.current search1:/opt/solr/logs/snapshot.current.search2
> + logMessage notifing Solr to open a new Searcher
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo 2007/09/05 15:10:06 notifing Solr to open a new Searcher
> + [[ -n '' ]]
> + /opt/solr/bin/commit
> + [[ 0 != 0 ]]
> + logExit ended 0
> ++ date +%s
> + end=1189030210
> ++ expr 1189030210 - 1189030205
> + diff=5
> ++ timeStamp
> ++ date '+%Y/%m/%d %H:%M:%S'
> + echo '2007/09/05 15:10:10 ended (elapsed time: 5 sec)'
> + exit 0
>
>
>
> +--------------------------------------------------------+
>  | Matthew Runo
>  | Zappos Development
>  | mruno@zappos.com
>  | 702-943-7833
> +--------------------------------------------------------+
>
>
> On Aug 31, 2007, at 6:11 PM, Bill Au wrote:
>
>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
>>> installed
>>
>> It looks like you have a directory named temp-snapshot.20070816120113
>> in your data directory.  You should remove it.  One of the other
>> script might have left that behind somehow.
>>
>> I will update the snapinstaller script to ignore non-snapshot when
>> looking for the latest snapshot to install.
>>
>> Bill
>>
>> On 8/31/07, Matthew Runo <mruno@zappos.com> wrote:
>>> Hello!
>>>
>>> On a somewhat related note, our replication seems very much broken.
>>> I've added -v to all my cron jobs, and I think I've seen the error
>>> (below).
>>>
>>> As you can see, it's rsyncing an updated index, but then doesn't  
>>> seem
>>> to know to install it. I'm not sure why though.. no errors are
>>> reported anywhere via the -v. Any help would be most appreciated,  
>>> I'm
>>> sure I'm just missing something.   You can see the cronjob  
>>> command in
>>> the subject of the forwarded message.
>>>
>>> +--------------------------------------------------------+
>>>   | Matthew Runo
>>>   | Zappos Development
>>>   | mruno@zappos.com
>>>   | 702-943-7833
>>> +--------------------------------------------------------+
>>>
>>>
>>> Begin forwarded message:
>>>
>>>> From: root@web03.zappos.com (Cron Daemon)
>>>> Date: August 31, 2007 1:02:36 PM PDT
>>>> To: mruno@zappos.com
>>>> Subject: Cron <tomcat5@search2> /opt/solr/bin/snappuller -M search1
>>>> -P 18080 -D /opt/solr/data -S /opt/solr/logs -d /opt/solr/data -v;/
>>>> opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -d /opt/
>>>> solr/data -v
>>>>
>>>> started by tomcat5
>>>> command: /opt/solr/bin/snappuller -M search1 -P 18080 -D /opt/solr/
>>>> data -S /opt/solr/logs -d /opt/solr/data -v
>>>> pulling snapshot snapshot.20070831130005
>>>> receiving file list ... done
>>>> deleting segments_19zm
>>>> deleting _10bg.tis
>>>> deleting _10bg.tii
>>>> deleting _10bg.prx
>>>> deleting _10bg.nrm
>>>> deleting _10bg.frq
>>>> deleting _10bg.fnm
>>>> deleting _10bg.fdx
>>>> deleting _10bg.fdt
>>>> ./
>>>> _14ff.fdt
>>>> _14ff.fdx
>>>> _14ff.fnm
>>>> _14ff.frq
>>>> _14ff.nrm
>>>> _14ff.prx
>>>> _14ff.tii
>>>> _14ff.tis
>>>> _14ff_9.del
>>>> _14fq.fdt
>>>> _14fq.fdx
>>>> _14fq.fnm
>>>> _14fq.frq
>>>> _14fq.nrm
>>>> _14fq.prx
>>>> _14fq.tii
>>>> _14fq.tis
>>>> _14fq_3.del
>>>> _14fr.fdt
>>>> _14fr.fdx
>>>> _14fr.fnm
>>>> _14fr.frq
>>>> _14fr.nrm
>>>> _14fr.prx
>>>> _14fr.tii
>>>> _14fr.tis
>>>> _14fr_2.del
>>>> _14fs.fdt
>>>> _14fs.fdx
>>>> _14fs.fnm
>>>> _14fs.frq
>>>> _14fs.nrm
>>>> _14fs.prx
>>>> _14fs.tii
>>>> _14fs.tis
>>>> _14fs_1.del
>>>> segments.gen
>>>> segments_1fza
>>>> write.lock
>>>>
>>>> sent 871 bytes  received 843185604 bytes  24440187.68 bytes/sec
>>>> total size is 843080453  speedup is 1.00
>>>> started by tomcat5
>>>> command: /opt/solr/bin/snapinstaller -M search1 -S /opt/solr/logs -
>>>> d /opt/solr/data -v
>>>> latest snapshot /opt/solr/data/temp-snapshot.20070816120113 already
>>>> installed
>>>>
>>>
>>>
>>
>


Mime
View raw message