samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jagadish Venkatraman <jagadish1...@gmail.com>
Subject [DISCUSS] Re-thinking the Samza Job Coordinator
Date Tue, 01 Mar 2016 16:48:20 GMT
Hi all,

Currently, the only way to run distributed Samza containers is using Yarn.
There have been requests to run Samza with other resource managers like
Mesos, to run Samza as an embedded library, and to run in Docker containers
managed by ECS/Kubernetes.

I created SAMZA-881 for this proposing a re-design of the Samza
coordinator.


   - The new re-design views the JobCoordinator as a component that can
   provide the containerId, and JobModel to the SamzaContainer.
   - There are pluggable interfaces defined for Leader Election, and
   interacting with a Resource Manager like Yarn/Mesos.
   - Primitives for dynamically refreshing the job model.

I've attached a design doc with the Jira ticket, and I'd really appreciate
your feedback on the ticket.

Thanks,
Jagadish

-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message