lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <mkhlud...@griddynamics.com>
Subject Re: Running multiple full-import commands via curl in a script
Date Wed, 28 Jan 2015 10:16:38 GMT
Literally, queue can be done by submitting as is (async) and polling
command status. However, giving
https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L200
you can try to add &synchronous=true&... that should hang request until
it's completed.
The other question is how run requests in parallel which is explicitly
violated by
https://github.com/apache/lucene-solr/blob/trunk/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java#L173
 The only workaround I can suggest is to duplicate DIH definitions in solr
config
  <requestHandler name="/dataimport" class="solr.DataImportHandler"> ...
  <requestHandler name="/dataimport2" class="solr.DataImportHandler"> ...
  <requestHandler name="/dataimport3" class="solr.DataImportHandler"> ...
 ...
then those guys should be able to handle own request in parallel. Nasty
stuff..
have a good hack

On Wed, Jan 28, 2015 at 3:47 AM, Carl Roberts <carl.roberts.zapata@gmail.com
> wrote:

> Hi,
>
> I am attempting to run all these curl commands from a script so that I can
> put them in a crontab job, however, it seems that only the first one
> executes and the other ones return with an error (below):
>
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2002"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2003"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2004"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2005"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2006"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2007"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2008"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2009"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2010"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2011"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2012"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2013"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2014"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> full-import&clean=false&entity=cve-2015"
> curl "http://127.0.0.1:8983/solr/nvd-rss/dataimport?command=
> delta-import&clean=false&entity=cve-last"
>
> error:
>
> *A command is still running...*
>
> Question:  Is there a way to queue the other requests in Solr so that they
> run as soon as the previous one is done?  If not, how would you recommend I
> do this?
>
> Many thanks in advance,
>
> Joe
>
>
>


-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mkhludnev@griddynamics.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message