lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carl Roberts <carl.roberts.zap...@gmail.com>
Subject Re: Running multiple full-import commands via curl in a script
Date Wed, 28 Jan 2015 13:59:15 GMT
Thanks Mikhail - synchronous=true works like a charm...:)

On 1/28/15, 5:16 AM, Mikhail Khludnev wrote:
> Literally, queue can be done by submitting as is (async) and polling
> command status. However, giving
> https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L200
> you can try to add &synchronous=true&... that should hang request until
> it's completed.
> The other question is how run requests in parallel which is explicitly
> violated by
> https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L173
>   The only workaround I can suggest is to duplicate DIH definitions in solr
> config
>    <requestHandler name="/dataimport" class="solr.DataImportHandler"> ...
>    <requestHandler name="/dataimport2" class="solr.DataImportHandler"> ...
>    <requestHandler name="/dataimport3" class="solr.DataImportHandler"> ...
>   ...
> then those guys should be able to handle own request in parallel. Nasty
> stuff..
> have a good hack
>
> On Wed, Jan 28, 2015 at 3:47 AM, Carl Roberts <carl.roberts.zapata@gmail.com
>> wrote:
>> Hi,
>>
>> I am attempting to run all these curl commands from a script so that I can
>> put them in a crontab job, however, it seems that only the first one
>> executes and the other ones return with an error (below):
>>
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2002"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2003"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2004"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2005"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2006"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2007"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2008"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2009"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2010"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2011"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2012"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2013"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2014"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> full-import&clean=false&entity=cve-2015"
>> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
>> delta-import&clean=false&entity=cve-last"
>>
>> error:
>>
>> *A command is still running...*
>>
>> Question:  Is there a way to queue the other requests in Solr so that they
>> run as soon as the previous one is done?  If not, how would you recommend I
>> do this?
>>
>> Many thanks in advance,
>>
>> Joe
>>
>>
>>
>


Mime
View raw message