lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeniba Johnson <Jeniba.John...@lntinfotech.com>
Subject Solr 4.4.0 on hadoop 2.2.0
Date Fri, 01 Aug 2014 06:42:44 GMT
Hi,

Iam new to solr. I have integrated solr4.9.0 and Hadoop 2.3.0.
I have changed the solrconfig.xml file  so that it can index and store the data on hdfs.

Solrconfig.xml
<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
      <str name="solr.hdfs.home">hdfs://xxx.xx.xx.xx:50070/user/solr/data</str>
      <bool name="solr.hdfs.blockcache.enabled">true</bool>
      <int name="solr.hdfs.blockcache.slab.count">1</int>
      <bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
      <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
      <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
      <bool name="solr.hdfs.blockcache.write.enabled">true</bool>
      <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
      <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
      <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
    </directoryFactory>

Commands
java -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs -Dsolr.data.dir=hdfs://
xxx.xx.xx.xx:50070/user/solr/data/collection1 -
Dsolr.updatelog=hdfs:// xxx.xx.xx.xx:50070/user/solr/data/collection1  -jar start.jar

Error

48454 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.core.CoreContainer  â Unable to
create core: collection1
org.apache.solr.common.SolrException: Problem creating directory: hdfs://172.29.17.40:50070/user/solr/data/collection1
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:868)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:643)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:261)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:253)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: Problem creating directory: hdfs://172.29.17.40:50070/user/solr/data/collection1
        at org.apache.solr.store.hdfs.HdfsDirectory.<init>(HdfsDirectory.java:87)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:148)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:267)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:479)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:774)
        ... 12 more
Caused by: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException:
Protocol message end-group tag did not match expected tag.; Host Details : local host is:
"cldx-1310-1182/172.29.17.40"; destination host is: "cldx-1310-1182":50070;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
        at org.apache.hadoop.ipc.Client.call(Client.java:1351)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1679)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
       at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
        at org.apache.solr.store.hdfs.HdfsDirectory.<init>(HdfsDirectory.java:62)
        ... 17 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group
tag did not match expected tag.
        at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
        at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
        at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
        at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
        at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
        at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
48485 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.core.CoreContainer  â null:org.apache.solr.common.SolrException:
Unable to create core: collection1
        at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:911)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:568)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:261)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:253)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.solr.common.SolrException: Problem creating directory: hdfs://172.29.17.40:50070/user/solr/data/collection1
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:868)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:643)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:556)
        ... 10 more
Caused by: java.lang.RuntimeException: Problem creating directory: hdfs://172.29.17.40:50070/user/solr/data/collection1
        at org.apache.solr.store.hdfs.HdfsDirectory.<init>(HdfsDirectory.java:87)
        at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:148)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:267)
        at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:479)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:774)
        ... 12 more
Caused by: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException:
Protocol message end-group tag did not match expected tag.; Host Details : local host is:
"cldx-1310-1182/172.29.17.40"; destination host is: "cldx-1310-1182":50070;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
        at org.apache.hadoop.ipc.Client.call(Client.java:1351)
        at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1679)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
        at org.apache.solr.store.hdfs.HdfsDirectory.<init>(HdfsDirectory.java:62)
        ... 17 more
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group
tag did not match expected tag.
        at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
        at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
        at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
        at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
        at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:2364)
        at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:996)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)

48535 [main] INFO  org.apache.solr.servlet.SolrDispatchFilter  â user.dir=/home/huser1/solr-4.9.0/example
48536 [main] INFO  org.apache.solr.servlet.SolrDispatchFilter  â SolrDispatchFilter.init()
done
49200 [main] INFO  org.eclipse.jetty.server.AbstractConnector  â Started SocketConnector@0.0.0.0:8983<mailto:SocketConnector@0.0.0.0:8983>

Can anyone look into this and help me with the solution.



Regards,
Jeniba


________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information
for the intended recipient(s). Unintended recipients are prohibited from taking action on
the basis of information in this e-mail and using or disseminating the information, and must
notify the sender and delete it from their system. L&T Infotech will not accept responsibility
or liability for the accuracy or completeness of, or the presence of any virus or disabling
code in this e-mail"

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message