samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bae, Jae Hyeon" <metac...@gmail.com>
Subject Re: Question on nullEnvelop
Date Fri, 06 Feb 2015 20:30:47 GMT
What I am doing is, consuming two topics, samza_input and samza_command.
samza_command will have some control command something like "shutdown,all"
because kill-yarn-job.sh does not gracefully shutdown SamzaContainer. Am I
correct? If so, what's the best way to shutdown the container without using
command topic?

10ms explains why 50 null envelops were consumed per second. Does it hurt
overall processing performance? I don't think so, but I want to confirm.

Thank you
Best, Jae

On Fri, Feb 6, 2015 at 12:16 PM, Chris Riccomini <criccomini@apache.org>
wrote:

> Hey Jae,
>
> SamzaContainer polls for new messages by calling
> consumerMultiplexer.choose. In a case where there are no messages
> available, choose will return null. The next time choose is called, it will
> be invoked with a timeout (the default is 10ms). This time, the poll call
> will block until 1) the timeout is hit 2) there is a new message available
> to process. This is to prevent a tight loop.
>
> > its frequency is too high, in my testing environment, it's more than 50
> per second.
>
> Why do you think this is too high? It either has to do this, or sleep for
> longer. The longer the container sleeps, the more latency that's introduced
> when there *is* a message available. 10ms is what we use by default.
>
> Cheers,
> Chris
>
> On Fri, Feb 6, 2015 at 11:11 AM, Bae, Jae Hyeon <metacret@gmail.com>
> wrote:
>
> > Could you explain why consumerMultiplexer.choose returns null?
> >
> > Can it happen when there's no message in the kafka topic?
> >
> > If my theory is correct, its frequency is too high, in my testing
> > environment, it's more than 50 per second.
> >
> > Thank you
> > Best, Jae
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message