lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dyer, James" <James.D...@ingramcontent.com>
Subject RE: DIH nested entities don't work
Date Tue, 27 Nov 2012 15:48:21 GMT
The line numbers aren't matching up, but best I can tell, it looks like its probably getting
a NPE on this line:

DIHCacheTypes type = types[pkColumnIndex];

If this is the correct line, it seems the "types" array is NULL.  The "types" array is populated
from a properties file that gets written when the cache is first created.  You should be able
to find this file for the PRODUCTS cache at d:\cacheloc\PRODUCTS_cache.properties ... This
file should have 2 lines "CACHE_NAMES=" and "CACHE_TYPES=".  The values should be comma-separated
lists of column names and data types from your database.  My guess is this is not getting
populated properly, or somehow this properties file is entirely not accessible.

If this is indeed the problem, the first thing I would try is to both change your "SELECT
*" statements to statements with actual lists of the columns, and to also specify "persistCacheFieldNames"
and "persistCacheFieldTypes" on the cache-building DIH handlers.

Also, I forget which version of Solr you say you're using, but if you're using 3.6, 4.0-ALPHA
or 4.0-BETA, the "cacheKey" parameter was incorrectly renamed "cachePk".  For 4.0.0, this
was renamed back to "cacheKey".

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-----Original Message-----
From: mroosendaal [mailto:mroosendaal@yahoo.com] 
Sent: Tuesday, November 27, 2012 3:17 AM
To: solr-user@lucene.apache.org
Subject: RE: DIH nested entities don't work

Hi James,

I was out for a week hence the late response. I did as you suggested and i
can import several views in parallel. However running the data-import which
does the join gives the following errors:

27-nov-2012 10:11:24 org.apache.solr.common.SolrException log
SEVERE: Exception while processing: END_FRG_PRODUCTS_VW document :
SolrInputDocument[PDT_GLOBAL_ID=1000004000000000,
PDT_EAN_CODE=0077779703623, PDT_I
D=1000004000000000, PDT_AVAILABILITY=Leverbaar, AVAIL_CODE_ON_STOCK=200,
search_title=19621966 Red Album, AVAIL_CODE_OFF_STOCK=200,
PDT_TYP_CODE=POP]:
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.RuntimeException: java.lang.NullPointerException
        at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:63)
        at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:246)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:472)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:498)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:411)
        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:326)
        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:234)
        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:382)
        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:448)
        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
        at
org.apache.solr.handler.dataimport.BerkleyBackedCache.iterator(BerkleyBackedCache.java:674)
        at
org.apache.solr.handler.dataimport.DIHCacheProcessor.nextRow(DIHCacheProcessor.java:97)
        at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
        ... 8 more
Caused by: java.lang.NullPointerException
        at
org.apache.solr.handler.dataimport.BerkleyBackedCache$PrimaryKeyTupleBinding.objectToEntry(BerkleyBackedCache.java:1071)
        at
com.sleepycat.bind.tuple.TupleBinding.objectToEntry(TupleBinding.java:73)
        at
org.apache.solr.handler.dataimport.BerkleyBackedCache.iterator(BerkleyBackedCache.java:663)
        ... 10 more

27-nov-2012 10:11:24 org.apache.solr.handler.dataimport.BerkleyBackedCache
close
INFO: Total read/write time for cache: PRODUCTS was 3 ms
27-nov-2012 10:11:24 org.apache.solr.handler.dataimport.BerkleyBackedCache
close
WARNING: couldn't close db for cache: PRODUCTS
Problem: 1 locks left
---- LSN: 0x0/0x170585c----
 ThinLockAddr:391683389 Owner:985340868 -1_Thread-18_ThreadLocker Waiters:
(none)

Local Cache Usage = 108
Cache Layout: Allocation of resources in the cache.
        adminBytes=0
        cacheTotalBytes=31.358
        dataBytes=0
        lockBytes=108
        sharedCacheTotalBytes=78.245

27-nov-2012 10:11:24 org.apache.solr.handler.dataimport.BerkleyBackedCache
close
INFO: Total read/write time for cache: FEATURE was 3 ms
27-nov-2012 10:11:24 org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: [collection1] webapp=/solr path=/dataimport-join-all
params={command=full-import&optimize=false&clean=false&commit=false&verbose=false}
status=0
 QTime=240 {} 0 240
27-nov-2012 10:11:24 org.apache.solr.common.SolrException log
SEVERE: Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: java
.lang.RuntimeException: java.lang.NullPointerException
        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:273)
        at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:382)
        at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:448)
        at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:429)
Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.RuntimeException: java.lang.NullPointe
rException
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:413)
        at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:326)
        at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:234)
        ... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.RuntimeException: java.lang.NullPointerException
        at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:63)
        at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:246)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:472)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:498)
        at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:411)
        ... 5 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
        at
org.apache.solr.handler.dataimport.BerkleyBackedCache.iterator(BerkleyBackedCache.java:674)
        at
org.apache.solr.handler.dataimport.DIHCacheProcessor.nextRow(DIHCacheProcessor.java:97)
        at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
        ... 8 more
Caused by: java.lang.NullPointerException
        at
org.apache.solr.handler.dataimport.BerkleyBackedCache$PrimaryKeyTupleBinding.objectToEntry(BerkleyBackedCache.java:1071)
        at
com.sleepycat.bind.tuple.TupleBinding.objectToEntry(TupleBinding.java:73)
        at
org.apache.solr.handler.dataimport.BerkleyBackedCache.iterator(BerkleyBackedCache.java:663)
        ... 10 more

27-nov-2012 10:11:24 org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback{flags=0,_version_=0}
27-nov-2012 10:11:24 org.apache.solr.update.DefaultSolrCoreState
newIndexWriter
INFO: Creating new IndexWriter...
27-nov-2012 10:11:24 org.apache.solr.update.DefaultSolrCoreState
newIndexWriter
INFO: Waiting until IndexWriter is unused... core=collection1
27-nov-2012 10:11:24 org.apache.solr.update.DefaultSolrCoreState
newIndexWriter
INFO: Rollback old IndexWriter... core=collection1
27-nov-2012 10:11:24 org.apache.solr.core.CachingDirectoryFactory close
INFO: Closing
directory:D:\apache-solr-4.1-2012-10-30_09-13-06\example\solr\collection1\data\index
27-nov-2012 10:11:24 org.apache.solr.core.CachingDirectoryFactory get
INFO: return new directory for
D:\apache-solr-4.1-2012-10-30_09-13-06\example\solr\collection1\data\index
forceNew:true
27-nov-2012 10:11:24 org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
       
commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@D:\apache-solr-4.1-2012-10-30_09-13-06\example\solr\collection1\data\inde
x lockFactory=org.apache.lucene.store.NativeFSLockFactory@74f2db2d;
maxCacheMB=48.0
maxMergeSizeMB=4.0),segFN=segments_te,generation=1058,filenames=[_
1qy_Lucene41_0.doc, _1qy_nrm.cfe, segments_te, _1qy.fnm,
_1qy_Lucene41_0.tip, _1qy_Lucene41_0.pos, _1qy.fdx, _1qy_nrm.cfs,
_1qy_Lucene41_0.tim, _1qy.s
i, _1qy.fdt]

My data-import-join:
<dataConfig>
	<document name="JOIN_ALL">
        	<entity name="END_FRG_PRODUCTS_VW" 
           	
processor="org.apache.solr.handler.dataimport.DIHCacheProcessor"
           		cacheKey="PDT_ID"
            		rootEntity="true"
            		persistCacheBaseDir="d:\cacheloc"
            	
persistCacheImpl="org.apache.solr.handler.dataimport.BerkleyBackedCache"
            		persistCacheName="PRODUCTS"
            		berkleyInternalCacheSize="1000000"
            		berkleyInternalShared="true">
            		<entity name="END_FRG_FEATURES_VW"
				processor="org.apache.solr.handler.dataimport.DIHCacheProcessor"
				persistCacheImpl="org.apache.solr.handler.dataimport.BerkleyBackedCache"
				persistCacheName="FEATURE" 
				persistCacheBaseDir="d:\cacheloc"
				berkleyInternalCacheSize="1000000"
                                berkleyInternalShared="true"
				cacheKey="PDT_ID"
				cacheLookup="END_FRG_PRODUCTS_VW.PDT_ID"/>
			<entity name="END_FRG_TAGS_VW"
				processor="org.apache.solr.handler.dataimport.DIHCacheProcessor"
				persistCacheImpl="org.apache.solr.handler.dataimport.BerkleyBackedCache"
				persistCacheName="TAGS" 
				persistCacheBaseDir="d:\cacheloc"
				berkleyInternalCacheSize="1000000"
                                berkleyInternalShared="true"
				cacheKey="PDT_ID"
				cacheLookup="END_FRG_PRODUCTS_VW.PDT_ID"/>
		</entity>
	</document>
</dataConfig>

Just to be clear products *can* have a feature and/or tag but is not
required.

Hope you have an idea?

Thanks,
Maarten



--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-nested-entities-don-t-work-tp4015514p4022566.html
Sent from the Solr - User mailing list archive at Nabble.com.



Mime
View raw message