samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xinyu liu <xinyuliu...@gmail.com>
Subject Re: RecordTooLargeException recovery
Date Thu, 06 Oct 2016 19:47:42 GMT
Hi, David,

For your questions:

1) In this case Samza recovered but the changelog message was lost. In
0.10.1 KafkaSystemProducer has a race condition: there is small chance the
later send success might override the previous failure. The bug is fixed in
the upcoming 0.11.0 release (SAMZA-1019). The fix allows you to catch the
exception and then you can decide to ignore or rethrow it. In the latter
case the container will fail and Samza will guarantee the message will be
reprocessed after it restarts.

2) There are several ways that might help in your case: First you can turn
on compression for your checkpoint stream. That usually saves about 20% -
30%. Second, you can also bump up the max.requst.size for the producer. In
this case you need to make sure the broker also set up the corresponding
max message size. Last, you might also try to split the key into subkeys so
the value will be smaller.

Thanks,
Xinyu

On Thu, Oct 6, 2016 at 9:30 AM, David Yu <david.yu@optimizely.com> wrote:

> Hi,
>
> Our Samza job (0.10.1) throws RecordTooLargeExceptions when flushing the KV
> store change to the changelog topic, as well as sending outputs to Kafka.
> We have two questions to this problem:
>
> 1. It seems that after the affected containers failed multiple times, the
> job was able to recover and move on. This is a bit hard to understand. How
> could this be recoverable? We were glad it actually did, but are
> uncomfortable not knowing the reason behind it.
> 2. We would be the best way to prevent this from happening? Since Samza
> serde happens behind the scenes, there does not seem to be a good way to
> find out the payload size in bytes before putting into the KV store. Any
> suggestions on this?
>
> Thanks,
> David
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message