samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Shkolnik <bor...@gmail.com>
Subject Re: API for containers?
Date Thu, 05 Nov 2015 21:40:31 GMT
Here is couple of ideas (not sure if that exactly what you want).
You can read JobModel of a job (form JobCoordinator server) and this job
model will contain container to tasks (and to partitions) mapping.

In order to find containers and the nodes - you can use yarn web api.
http://RM/cluster, then go to ApplicationMaster and list all the containers.

But remember this mapping can change, if a container fails, it will be
restarted on the same or different node.

On Wed, Nov 4, 2015 at 11:42 AM, Rick Mangi <rick@chartbeat.com> wrote:

> Hello,
>
> I’m wondering if there’s currently any API (or other way) to find out
> which container would handle a theoretical key and what node that container
> is running on.
>
> The use-case I’m thinking of is if I wanted to run a rest service on each
> node which could handle queries against the current state of a job by
> talking to a shared kv store. I saw that there is an elastic search kv
> implementation in the codebase but didn’t see any way to direct a query to
> a specific instance of elastic search running on the cluster. To do this
> you would want to look at a key and determine which container would handle
> that key in the stream, then hit a service on that node (potentially there
> could be more than 1 container on the node of course…).
>
> Thanks,
>
> Rick
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message