kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Yeargers <jon.yearg...@cedexis.com>
Subject kafka consumers as kubernetes pods - best practice
Date Mon, 01 May 2017 22:09:44 GMT
Im looking for suggestions as to how to manage k-consumers when they are
run as kubernetes pods - especially in an auto-scaling environment. Looking
at the output of our logging it seems like we're dropping data when a pod
is moved between hosts despite doing (what I believe is) an orderly
shutdown.

Im seeing values for consumer lag drop precipitously when a pod is
restarted - as if it were committing an offset at the end of the queue
instead of where it should have left off / picked up.

This is an investigation-in-progress. Not entirely sure what's happening
here - it just looks wrong.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message