spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rachana Srivastava <Rachana.Srivast...@markmonitor.com>
Subject Spark process failing to receive data from the Kafka queue in yarn-client mode.
Date Fri, 05 Feb 2016 17:38:18 GMT
I am trying to run following code using yarn-client mode in but getting slow readprocessor
error mentioned below but the code works just fine in the local mode.  Any pointer is really
appreciated.

Line of code to receive data from the Kafka Queue:
JavaPairReceiverInputDStream<String, String> messages =  KafkaUtils.createStream(jssc,
String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, kafkaTopicMap,
StorageLevel.MEMORY_ONLY());

JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>,
String>() {
      public String call(Tuple2<String, String> tuple2) {
                              LOG.info(" &&&&&&&&&&&&&&&&&&&&
Input json stream data  " +  tuple2._2);
        return tuple2._2();
      }
    });


Error Details:
016-02-05 11:44:00 WARN DFSClient:975 - Slow ReadProcessor read fields took 30
011ms (threshold=30000ms); ack: seqno: 1960 reply: 0 reply: 0 reply: 0 downstrea
mAckTimeNanos: 1227280, targets: [DatanodeInfoWithStorage[10.0.0.245:50010,DS-a5
5d9212-3771-4936-bbe7-02035e7de148,DISK], DatanodeInfoWithStorage[10.0.0.243:500
10,DS-231b9915-c2e2-4392-b075-8a52ba1820ac,DISK], DatanodeInfoWithStorage[10.0.0
.244:50010,DS-6b8b5814-7dd7-4315-847c-b73bd375af0e,DISK]]
2016-02-05 11:44:00 INFO BlockManager:59 - Removing RDD 1954
2016-02-05 11:44:00 INFO MapPartitionsRDD:59 - Removing RDD 1955 from persisten

Mime
View raw message