kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Kupstaitis-Dunkler <stefan....@gmail.com>
Subject Pushing from Pig to Kafka
Date Mon, 28 Sep 2015 08:11:39 GMT
Hi all!

I accidently posted this on the dev mailing list yesterday, when it's much
better off here. Can anybody help me?

When I run a pig script I get:

java.lang.InstantiationError: org.apache.avro.io.BinaryEncoder
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.InstantiationError: org.apache.avro.io.BinaryEncoder
at
kafka.bridge.pig.AvroKafkaStorage.prepareToWrite(AvroKafkaStorage.java:77)

I am using a HDP 2.2 installation which comes with Kafka 0.8.1. I try to
push a single string from Pig into Kafka and follow these instructions:
https://github.com/apache/kafka/blob/43b92f8b1ce8140c432edf11b0c842f5fbe04120/contrib/hadoop-producer/README.md

The script looks like:
REGISTER
/usr/hdp/current/kafka-broker/contrib/lib/kafka-hadoop-producer-0.8.1.2.2.0.0-2041.jar;
REGISTER /usr/hdp/current/pig-client/lib/avro-1.4.1.jar;
REGISTER /usr/hdp/current/pig-client/piggybank.jar;
REGISTER
/usr/hdp/current/kafka-broker/libs/kafka_2.10-0.8.1.2.2.0.0-2041.jar;
REGISTER /usr/hdp/current/pig-client/lib/jackson-core-asl-1.8.8.jar;
REGISTER /usr/hdp/current/pig-client/lib/jackson-mapper-asl-1.8.8.jar;
REGISTER /usr/hdp/current/kafka-broker/libs/scala-library-2.10.4.jar;
STORE my_data INTO 'kafka://MY-BROKER-HOST-NAME:6667/myTopic' USING
kafka.bridge.pig.AvroKafkaStorage('"string"');

MY-BROKER-HOST-NAME is one of my Kafka brokers, myTopic is an existing
topic. The version 1.4.1 of avro is not deployed by default, but I deployed
it and registered it instead of avro 1.7.5, because I first thought this
will solve my problem, but the same problem occurs. Also, I noticed, that
the same problom still occurs, when I do not register avro at all. When I
look into the avro documentation I can see, that the BinaryEncoder class is
abstract, but nevertheless is tried to be instantiated in the
AvroKakfaStorage class. So the error is not a surprise. Is there anything I
am missing to make this work? Am I doing something wrong?

Best regards, Stefan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message