lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Lowe <jbl...@johnblowe.com>
Subject DIH problem passing HTTP parameters into data-config
Date Tue, 11 Aug 2009 23:25:34 GMT
I've read the documentation as carefully as I can, but I must be  
missing something.  I'm running Solr 1.3.

The doc sez that I can pass my own parameters in to DIH via the HTTP  
request:

http://wiki.apache.org/solr/DataImportHandler#head-520f8e527d9da55e8ed1e274e29709c8805c8eae

What I'd like to do is pass the URL of an RSS feed into DIH via HTTP (/ 
dataimport), and have it import the items in the feed in "the usual  
way" (i.e. as shown for example in the slashdot example).  (This way I  
don't have to hardcode my list of feeds)

SO if I have a URL like the following to dataimport:

http://localhost:8983/solr/rss/dataimport?command=full-import&clean=false&feed=http%3A%2F%2Ffeeds.wsjonline.com%2Fwsj%2Fxml%2Frss%2F3_7011.xml

(i.e. a normal dataimport command, plus a parameter 'feed' with the  
url-encoded value of an RSS feed)

AND, in my config file I have something like:

<dataConfig>
        <dataSource type="HttpDataSource" name="rss-feed" />
        <document>
                <entity name="article" dataSource="rss-feed"
                                pk="link"
                                url="${dataimporter.request.url}"
                                processor="XPathEntityProcessor"

[blah blah blah...looks pretty much like the slashdot RSS rss-data- 
config.xml example)]

It should just work.

However,

I get the following:

Aug 10, 2009 12:15:55 AM org.apache.solr.core.SolrCore execute
INFO: [rss] webapp=/solr path=/dataimport params={command=full- 
import&feed=http://feeds.wsjonline.com/wsj/xml/rss/3_7011.xml&clean=false&feedname=wsjhome

} status=0 QTime=0
Aug 10, 2009 12:15:55 AM  
org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO: Starting Full Import
Aug 10, 2009 12:15:55 AM  
org.apache.solr.handler.dataimport.HttpDataSource getData
SEVERE: Exception thrown while getting data
java.net.MalformedURLException: no protocol: null
	at java.net.URL.<init>(URL.java:583)
	at java.net.URL.<init>(URL.java:480)
	at java.net.URL.<init>(URL.java:429)
	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:90)
	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:47)
	at  
org 
.apache 
.solr 
.handler 
.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java: 
210)
	at  
org 
.apache 
.solr 
.handler 
.dataimport 
.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:180)
	at  
org 
.apache 
.solr 
.handler 
.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:163)
	at  
org 
.apache 
.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
	at  
org 
.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java: 
178)
	at  
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java: 
136)
	at  
org 
.apache 
.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java: 
334)
	at  
org 
.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java: 
386)
	at org.apache.solr.handler.dataimport.DataImporter 
$1.run(DataImporter.java:377)
Aug 10, 2009 12:15:55 AM org.apache.solr.handler.dataimport.DocBuilder  
buildDocument
SEVERE: Exception while processing: ${dataimporter.request.feedname}  
document : SolrInputDocumnt[{}]
org.apache.solr.handler.dataimport.DataImportHandlerException:  
Exception in invoking url null Processing Document # 1
	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:115)
	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:47)
	at  
org 
.apache 
.solr 
.handler 
.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java: 
210)
	at  
org 
.apache 
.solr 
.handler 
.dataimport 
.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:180)
	at  
org 
.apache 
.solr 
.handler 
.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:163)
	at  
org 
.apache 
.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
	at  
org 
.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java: 
178)
	at  
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java: 
136)
	at  
org 
.apache 
.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java: 
334)
	at  
org 
.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java: 
386)
	at org.apache.solr.handler.dataimport.DataImporter 
$1.run(DataImporter.java:377)
Caused by: java.net.MalformedURLException: no protocol: null
	at java.net.URL.<init>(URL.java:583)
	at java.net.URL.<init>(URL.java:480)
	at java.net.URL.<init>(URL.java:429)
	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:90)
	... 10 more
Aug 10, 2009 12:15:55 AM  
org.apache.solr.handler.dataimport.DataImporter doFullImport
SEVERE: Full Import failed

Seems like the HTTP parm is not getting passed. Or I'm not referring  
to it properly. Or something!

Any suggestions welcome...

Cheers, TIA,

John
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message