kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Ciceron <...@triggit.com>
Subject Re: jobtracker / hadoop comsumer
Date Wed, 31 Aug 2011 19:02:46 GMT
ok i could live with setting mapred.job.tracker manually for the code for now.
This way it can connect now to the proper jobtracker.

> The hadoop map tasks will need to connect to the kafka server port (the
> broker uri/port).

i run the hadoop soncumer on the same hostA where the kafka-server is running.

each of the host in the hadoop cluster can telnet/nmap to port 9092 on
hostA where the kafka-server is running.
also HostA can connect to port 5181 on any host in the cluster.

but each task fails with a similar connection issue :
java.io.IOException: java.net.ConnectException: Connection refused
	at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:155)
	at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:14)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:326)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.connect(Native Method)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
	at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:54)
	at kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:193)
	at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:156)
	at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:65)
	at kafka.etl.KafkaETLContext.getOffsetRange(KafkaETLContext.java:209)
	at kafka.etl.KafkaETLContext.<init>(KafkaETLContext.java:97)
	at kafka.etl.KafkaETLRecordReader.next(KafkaETLRecordReader.java:115)
	... 11 more

Mime
View raw message