samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com.INVALID>
Subject Re: A few questions on Samza
Date Mon, 01 Dec 2014 20:46:59 GMT
Hey Jae,

> how can I change the number of containers on the fly?

This is not currently supported. You have to restart your job with a new
yarn.container.count setting. SAMZA-348 is the ticket that will allow this.

> Can I use kill-yarn-job.sh?

If your job is running in YARN, you can use that script. You can also have
the job shut itself down using coordinator.shutdown().

> How long does Samza wait for close() method to return? Is there any
>timeout configuration? or will it be blocked forever?

It will block forever. Your task must handle any timeouts itself, and
return cleanly form close().

> Is there any guideline for running samza jobs without auto-create topics?

Not really. In such a case, you'd just need to pre-create your output
topics before running your job. The checkpoint topic should be
auto-created for you (Samza will try to do this, even if you have Kafka's
auto create feature disabled).

Cheers,
Chris

On 12/1/14 12:35 PM, "Bae, Jae Hyeon" <metacret@gmail.com> wrote:

>Hello
>
>I am very new at samza and doing PoC now. So far, I really like Samza
>especially its architecture on tight integration with Kafka and
>checkpointing.
>
>1. Dynamically change the number of containers
>
>I saw the following from the documentation:
>
>"The number of tasks is determined automatically from the number of
>partitions in the input and is fixed, but the number of containers (and
>the
>CPU and memory resources associated with them) is specified by the user at
>run time and can be changed at any time."
>
>Is this true? If so, how can I change the number of containers on the fly?
>
>2. How to shutdown the job
>
>I will need to shutdown the job and restart it. How can I shutdown the
>job?
>Can I use kill-yarn-job.sh?
>
>3. Timeout for close() of ClosableTask
>
>I guess that if the job implements ClosableTask, it can be shutdown
>gracefully. How long does Samza wait for close() method to return? Is
>there
>any timeout configuration? or will it be blocked forever?
>
>4. Auto-create topics
>
>We're not running our kafka clusters with auto.create.topics.enabled=true
>to prevent any abuse of the broker. We observed that auto-created topics
>fell into unstable status before, we're not sure it's a bug on kafka
>broker
>or it was a just coincidence. Is there any guideline for running samza
>jobs
>without auto-create topics?
>
>Thank you
>Best, Jae


Mime
View raw message