samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Yu <david...@optimizely.com>
Subject RecordTooLargeException recovery
Date Thu, 06 Oct 2016 16:30:57 GMT
Hi,

Our Samza job (0.10.1) throws RecordTooLargeExceptions when flushing the KV
store change to the changelog topic, as well as sending outputs to Kafka.
We have two questions to this problem:

1. It seems that after the affected containers failed multiple times, the
job was able to recover and move on. This is a bit hard to understand. How
could this be recoverable? We were glad it actually did, but are
uncomfortable not knowing the reason behind it.
2. We would be the best way to prevent this from happening? Since Samza
serde happens behind the scenes, there does not seem to be a good way to
find out the payload size in bytes before putting into the KV store. Any
suggestions on this?

Thanks,
David

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message