spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Triones,Deng(vip.com)" <triones.d...@vipshop.com>
Subject spark kafka receiver with different partition the consumer speed is unbanlance in one same executor
Date Sat, 27 Feb 2016 13:56:10 GMT
Hi spark dev/user.

         I am running spark 1.4.1 with kafka high level consumer. With spark.streaming.receiver.maxRate=10000
and with consumer thread 1(topicMap.put(topic, 1))
         I found a problem is that the consumer speed for different partition in one executor
is different far away.

See the prod data below:
         For partition 180: the first consumer length is 4597280216, the second time is :
 4597462979     , the diff is : 182763
         For partition 181: the first consumer length is 4616334930 , the second time is :
 4616635037    , the diff is : 300107

There are big diff for different partition. So any one knows why , and any suggestions about
this, I just plan to increase consumer thread. (topicMap.put(topic, 2))

The LAG Info:

Step 1:

./bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect *********160:2181/kafka
--group spark-mercury-trace-product  | grep *****************************114-93.idc.vipshop.com-145325889
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
spark-mercury-trace-product trace                          180 4597280216      4603394607
     6114391         spark-mercury-trace-product_gd6-bigdata-spark2-114-93.idc.vipshop.com-1453258898970-abd247a8-0
spark-mercury-trace-product trace                          181 4616334930      4616335540
     610             spark-mercury-trace-product_gd6-bigdata-spark2-114-93.idc.vipshop.com-1453258898970-abd247a8-0

step 2:

./bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect *********160:2181/kafka
--group spark-mercury-trace-product  | grep *****************************114-93.idc.vipshop.com-145325889
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
spark-mercury-trace-product trace                          180 4597462979      4603695222
     6232243         *****************************114-93.idc.vipshop.com-145325889
spark-mercury-trace-product trace                          181 4616635037      4616636392
     1355            *****************************114-93.idc.vipshop.com-145325889


the stack Info:

[cid:image001.png@01D171A9.A9D64A60]


本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人,谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容,或将其用于其他任何目的或向任何人披露。谢谢您的合作!
This communication is intended only for the addressee(s) and may contain information that
is privileged and confidential. You are hereby notified that, if you are not an intended recipient
listed above, or an authorized employee or agent of an addressee of this communication responsible
for delivering e-mail messages to an intended recipient, any dissemination, distribution or
reproduction of this communication (including any attachments hereto) is strictly prohibited.
If you have received this communication in error, please notify us immediately by a reply
e-mail addressed to the sender and permanently delete the original e-mail communication and
any attachments from all storage devices without making or otherwise retaining a copy.
Mime
View raw message