spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Spark Streaming + Kafka + scala job message read issue
Date Thu, 24 Dec 2015 10:21:39 GMT
Hi All,

We are using Bitnami Kafka 0.8.2 + spark 1.5.2 in Google cloud platform. Our spark streaming
job(consumer) not receiving all the messages sent to the specific topic. It receives 1 out
of ~50 messages(added log in the job stream and identified). We are not seeing any errors
in the kafka logs. Unable to debug further from kafka layer. The console consumer shows the
INPUT topic is received in the console. it is not reaching the spark-kafka integration stream.
Any thoughts how to debug this issue. Another topic is working fine in same setup.

Again tried with spark 1.3.0, kafka which is also has same issue. All these jobs are
working fine in our local lab servers

Vivek M
The information contained in this electronic message and any attachments to this message are
intended for the exclusive use of the addressee(s) and may contain proprietary, confidential
or privileged information. If you are not the intended recipient, you should not disseminate,
distribute or copy this e-mail. Please notify the sender immediately and destroy all copies
of this message and any attachments. WARNING: Computer viruses can be transmitted via email.
The recipient should check this email and any attachments for the presence of viruses. The
company accepts no liability for any damage caused by any virus transmitted by this email.

View raw message