samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Bär <nicolas.b...@gmail.com>
Subject Task Coordinator Shutdown
Date Sun, 15 Jun 2014 19:57:43 GMT
Hi All

I tried the new shutdown feature from SAMZA-253.

It works well in case one container is started with multiple threads on a
single node. But
`taskCoordinator.shutdown(TaskCoordinator.RequestScope.CURRENT_TASK);`
seems not to work running a Samza Stream Task with 20 containers on 4
machines using YARN. Each container handles one partition only, therefore
I'm consuming from a Kafka topic with 20 partitions.

As far as I understood, this would mean on every call of
`taskCoordinator.shutdown(TaskCoordinator.RequestScope.CURRENT_TASK);` the
corresponding container would finish. This is not the case. In fact all
containers are still running after calling this command in every task.

In case it matters: the command is called in the window function of the
WindowableTask.

Yarn: 2.2.0
Samza: 0.7.0 branch (from last friday:
052de224a3256cc652032de5e804338b4dc92fe0)

Any hints on how to further debug this?

Second question: Is there any chance to receive the container name / number
within the task instance? It would make debugging a lot easier :)


Cheers
Nicolas

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message