lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "davidchiu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-7948) MapReduceIndexerTool of solr 5.2.1 doesn't work with hadoop 2.7.1
Date Fri, 21 Aug 2015 02:01:46 GMT

    [ https://issues.apache.org/jira/browse/SOLR-7948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706108#comment-14706108
] 

davidchiu commented on SOLR-7948:
---------------------------------

Do you mean that I should add "mapreduce.job.user.classpath.first=true" into mapred-site.xml?

> MapReduceIndexerTool of solr 5.2.1 doesn't work with hadoop 2.7.1
> -----------------------------------------------------------------
>
>                 Key: SOLR-7948
>                 URL: https://issues.apache.org/jira/browse/SOLR-7948
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - MapReduce
>    Affects Versions: 5.2.1
>         Environment: OS:suse 11
> JDK:java version "1.7.0_65" 
> Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
> HADOOP:hadoop 2.7.1 
> SOLR:5.2.1
>            Reporter: davidchiu
>            Assignee: Mark Miller
>
> When I used MapReduceIndexerTool of 5.2.1 to index files, I got follwoing errors,but
I used 4.9.0's MapReduceIndexerTool, it did work with hadoop 2.7.1.
> Exception ERROR as following:
> INFO  - 2015-08-20 11:44:45.155; [   ] org.apache.solr.hadoop.HeartBeater; Heart beat
reporting class is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
> INFO  - 2015-08-20 11:44:45.161; [   ] org.apache.solr.hadoop.SolrRecordWriter; Using
this unpacked directory as solr home: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
> INFO  - 2015-08-20 11:44:45.162; [   ] org.apache.solr.hadoop.SolrRecordWriter; Creating
embedded Solr server with solrHomeDir: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip,
fs: DFS[DFSClient[clientName=DFSClient_attempt_1440040092614_0004_r_000001_0_1678264055_1,
ugi=root (auth:SIMPLE)]], outputShardDir: hdfs://127.0.0.1:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001
> INFO  - 2015-08-20 11:44:45.194; [   ] org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader
for directory: '/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/'
> INFO  - 2015-08-20 11:44:45.206; [   ] org.apache.solr.hadoop.HeartBeater; HeartBeat
thread running
> INFO  - 2015-08-20 11:44:45.207; [   ] org.apache.solr.hadoop.HeartBeater; Issuing heart
beat for 1 threads
> INFO  - 2015-08-20 11:44:45.418; [   ] org.apache.solr.hadoop.SolrRecordWriter; Constructed
instance information solr.home /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
(/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip),
instance dir /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/,
conf dir /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/conf/,
writing index to solr.data.dir hdfs://127.0.0.1:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001/data,
with permdir hdfs://127.0.0.10:9000/tmp/outdir/reducers/_temporary/1/_temporary/attempt_1440040092614_0004_r_000001_0/part-r-00001
> INFO  - 2015-08-20 11:44:45.426; [   ] org.apache.solr.core.SolrXmlConfig; Loading container
configuration from /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/solr.xml
> INFO  - 2015-08-20 11:44:45.474; [   ] org.apache.solr.core.CorePropertiesLocator; Config-defined
core root directory: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip
> INFO  - 2015-08-20 11:44:45.503; [   ] org.apache.solr.core.CoreContainer; New CoreContainer
1656436773
> INFO  - 2015-08-20 11:44:45.503; [   ] org.apache.solr.core.CoreContainer; Loading cores
into CoreContainer [instanceDir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/]
> INFO  - 2015-08-20 11:44:45.503; [   ] org.apache.solr.core.CoreContainer; loading shared
library: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/lib
> WARN  - 2015-08-20 11:44:45.504; [   ] org.apache.solr.core.SolrResourceLoader; Can't
find (or read) directory to add to classloader: lib (resolved as: /usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1440040092614_0004/container_1440040092614_0004_01_000004/82f2eca9-d6eb-483b-960f-0d3b3b93788c.solr.zip/lib).
> INFO  - 2015-08-20 11:44:45.520; [   ] org.apache.solr.handler.component.HttpShardHandlerFactory;
created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections
: 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue :
-1,fairnessPolicy : false,useRetries : false,
> FATAL - 2015-08-20 11:44:45.526; [   ] org.apache.hadoop.mapred.YarnChild; Error running
child : java.lang.VerifyError: Bad return type
> Exception Details:
>   Location:
>     org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/http/conn/ClientConnectionManager;)Lorg/apache/http/impl/client/CloseableHttpClient;
@62: areturn
>   Reason:
>     Type 'org/apache/http/impl/client/DefaultHttpClient' (current frame, stack[0]) is
not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
>   Current Frame:
>     bci: @62
>     flags: { }
>     locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/http/conn/ClientConnectionManager',
'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/DefaultHttpClient'
}
>     stack: { 'org/apache/http/impl/client/DefaultHttpClient' }
>   Bytecode:
>     0000000: bb00 0359 2ab7 0004 4db2 0005 b900 0601
>     0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
>     0000020: b600 0a2c b600 0bb6 000c b900 0d02 00bb
>     0000030: 0011 592b b700 124e 2d2c b800 102d b0  
>   Stackmap Table:
>     append_frame(@47,Object[#127])
> 	at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:166)
> 	at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:49)
> 	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
> 	at org.apache.solr.hadoop.SolrRecordWriter.createEmbeddedSolrServer(SolrRecordWriter.java:163)
> 	at org.apache.solr.hadoop.SolrRecordWriter.<init>(SolrRecordWriter.java:119)
> 	at org.apache.solr.hadoop.SolrOutputFormat.getRecordWriter(SolrOutputFormat.java:163)
> 	at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
> 	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message