lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-7948) MapReduceIndexerTool of solr 5.2.1 doesn't work with hadoop 2.7.1
Date Thu, 20 Aug 2015 13:22:45 GMT

    [ https://issues.apache.org/jira/browse/SOLR-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704864#comment-14704864
] 

Mark Miller commented on SOLR-7948:
-----------------------------------

Thanks for the report.

I actually ran into this issue a couple weeks ago while trying to get the map reduce contrib
back up to speed.

You can see this issue when you try and run the example: https://github.com/markrmiller/solr-map-reduce-example

I think the issue is that a Kite Morphlines jar is using a Solr class that has changed. If
so, the answer is that Kite Morphlines should not use Solr classes outside of it's couple
Solr modules. I'll look into getting that changed very soon, and then we will have to update
versions.

> MapReduceIndexerTool of solr 5.2.1 doesn't work with hadoop 2.7.1
> -----------------------------------------------------------------
>
>                 Key: SOLR-7948
>                 URL: https://issues.apache.org/jira/browse/SOLR-7948
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - MapReduce
>    Affects Versions: 5.2.1
>         Environment: OS:suse 11
> JDK:java version "1.7.0_65" 
> Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
> HADOOP:hadoop 2.7.1 
> SOLR:5.2.1
>            Reporter: davidchiu
>
> When I used MapReduceIndexerTool of 5.2.1 to index files, I got follwoing errors,but
I used 4.9.0's MapReduceIndexerTool, it did work with hadoop 2.7.1.
> Exception ERROR as following:
> INFO  - 2015-08-20 11:44:45.155; [   ] org.apache.solr.hadoop.HeartBeater; Heart beat
reporting class is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
> INFO  - 2015-08-20 11:44:45.161; [   ] org.apache.solr.hadoop.SolrRecordWriter; Using
this unpacked directory as solr home: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
> INFO  - 2015-08-20 11:44:45.162; [   ] org.apache.solr.hadoop.SolrRecordWriter; Creating
embedded Solr server with solrHomeDir: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip,
fs: DFS[DFSClient[clientName=DFSClient_attempt_1440040092614_0004_r_000001_0_1678264055_1,
ugi=root (auth:SIMPLE)]], outputShardDir: hdfs://127.0.0.1:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001
> INFO  - 2015-08-20 11:44:45.194; [   ] org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader
for directory: '/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/'
> INFO  - 2015-08-20 11:44:45.206; [   ] org.apache.solr.hadoop.HeartBeater; HeartBeat
thread running
> INFO  - 2015-08-20 11:44:45.207; [   ] org.apache.solr.hadoop.HeartBeater; Issuing heart
beat for 1 threads
> INFO  - 2015-08-20 11:44:45.418; [   ] org.apache.solr.hadoop.SolrRecordWriter; Constructed
instance information solr.home /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
(/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip),
instance dir /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/,
conf dir /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/conf/,
writing index to solr.data.dir hdfs://127.0.0.1:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001/data,
with permdir hdfs://127.0.0.10:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001
> INFO  - 2015-08-20 11:44:45.426; [   ] org.apache.solr.core.SolrXmlConfig; Loading container
configuration from /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/solr.xml
> INFO  - 2015-08-20 11:44:45.474; [   ] org.apache.solr.core.CorePropertiesLocator; Config-defined
core root directory: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
> INFO  - 2015-08-20 11:44:45.503; [   ] org.apache.solr.core.CoreContainer; New CoreContainer
1656436773
> INFO  - 2015-08-20 11:44:45.503; [   ] org.apache.solr.core.CoreContainer; Loading cores
into CoreContainer [instanceDir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/]
> INFO  - 2015-08-20 11:44:45.503; [   ] org.apache.solr.core.CoreContainer; loading shared
library: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/lib
> WARN  - 2015-08-20 11:44:45.504; [   ] org.apache.solr.core.SolrResourceLoader; Can't
find (or read) directory to add to classloader: lib (resolved as: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/lib).
> INFO  - 2015-08-20 11:44:45.520; [   ] org.apache.solr.handler.component.HttpShardHandlerFactory;
created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections
: 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue :
-1,fairnessPolicy : false,useRetries : false,
> FATAL - 2015-08-20 11:44:45.526; [   ] org.apache.hadoop.mapred.YarnChild; Error running
child : java.lang.VerifyError: Bad return type
> Exception Details:
>   Location:
>     org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/http/conn/ClientConnectionManager;)Lorg/apache/http/impl/client/CloseableHttpClient;
@62: areturn
>   Reason:
>     Type 'org/apache/http/impl/client/DefaultHttpClient' (current frame, stack[0]) is
not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
>   Current Frame:
>     bci: @62
>     flags: { }
>     locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/http/conn/ClientConnectionManager',
'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/DefaultHttpClient'
}
>     stack: { 'org/apache/http/impl/client/DefaultHttpClient' }
>   Bytecode:
>     0000000: bb00 0359 2ab7 0004 4db2 0005 b900 0601
>     0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
>     0000020: b600 0a2c b600 0bb6 000c b900 0d02 00bb
>     0000030: 0011 592b b700 124e 2d2c b800 102d b0  
>   Stackmap Table:
>     append_frame(@47,Object[#127])
> 	at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:166)
> 	at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:49)
> 	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
> 	at org.apache.solr.hadoop.SolrRecordWriter.createEmbeddedSolrServer(SolrRecordWriter.java:163)
> 	at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:119)
> 	at org.apache.solr.hadoop.SolrOutputFormat.getRecordWriter(SolrOutputFormat.java:163)
> 	at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
> 	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message