lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dyer, James" <James.D...@ingramcontent.com>
Subject RE: problem with data import handler delta import due to use of multiple datasource
Date Tue, 08 Oct 2013 14:18:02 GMT
Bill,

I do not believe there is any way to tell it to use a different datasource for the parent
delta query.  

If you used this approach, would it solve your problem:  http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport
?

James Dyer
Ingram Content Group
(615) 213-4311


-----Original Message-----
From: Bill Au [mailto:bill.w.au@gmail.com] 
Sent: Tuesday, October 08, 2013 8:50 AM
To: solr-user@lucene.apache.org
Subject: Re: problem with data import handler delta import due to use of multiple datasource

I am using 4.3.  It is not related to bugs related to last_index_time.  The
problem is caused by the fact that the parent entity and child entity use
different data source (different databases on different hosts).

>From the log output, I do see the the delta query of the child entity being
executed correctly and found all the rows that have been modified for the
child entity.  But it fails when it executed the parentDeltaQuery because
it is still using the database connection from the child entity (ie
datasource ds2 in my example above).

Is there a way to tell DIH to use a different datasource in the
parentDeltaQuery?

Bill


On Sat, Oct 5, 2013 at 10:28 PM, Alexandre Rafalovitch
<arafalov@gmail.com>wrote:

> Which version of Solr and what kind of SQL errors? There were some bugs in
> 4.x related to last_index_time, but it does not sound related.
>
> Regards,
>    Alex.
>
> Personal website: http://www.outerthoughts.com/
> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> - Time is the quality of nature that keeps events from happening all at
> once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD book)
>
>
> On Sun, Oct 6, 2013 at 8:51 AM, Bill Au <bill.w.au@gmail.com> wrote:
>
> > Here is my DIH config:
> >
> > <dataConfig>
> > <dataSource name="ds1" type="JdbcDataSource"
> driver="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://localhost1/dbname1" user="db_username1"
> > password="db_password1"/>
> > <dataSource name="ds2" type="JdbcDataSource"
> driver="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://localhost2/dbname2" user="db_username2"
> > password="db_password2"/>
> > <document name="products">
> >         <entity name="item" dataSource="ds1" query="select * from item">
> >             <field column="ID" name="id" />
> >             <field column="NAME" name="name" />
> >
> >             <entity name="feature" dataSource="ds2" query="select
> > description from feature where item_id='${item.ID}'">
> >                 <field name="features" column="description" />
> >             </entity>
> >         </entity>
> >     </document>
> > </dataConfig>
> >
> > I am having trouble with delta import.  I think it is because the main
> > entity and the sub-entity use different data source.  I have tried using
> > both a delta query:
> >
> > deltaQuery="select id from item where id in (select item_id as id from
> > feature where last_modified > '${dih.last_index_time}') or last_modified
> > &gt; '${dih.last_index_time}'"
> >
> > and a parentDeltaQuery:
> >
> > <entity name="feature" pk="ITEM_ID" query="select DESCRIPTION as features
> > from FEATURE where ITEM_ID='${item.ID}'" deltaQuery="select ITEM_ID from
> > FEATURE where last_modified > '${dih.last_index_time}'"
> > parentDeltaQuery="select ID from item where ID=${feature.ITEM_ID}"/>
> >
> > I ended up with an SQL error for both.  Is there any way to make delta
> > import work in my case?
> >
> > Bill
> >
>


Mime
View raw message