samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elias Levy <>
Subject Executing Samza jobs natively in Kubernetes
Date Sun, 29 Nov 2015 19:18:15 GMT
I've been exploring Samza for stream processing as well as Kubernetes as a
container orchestration system and I wanted to be able to use one with the
other.  The prospect of having to execute YARN either along side or on top
of Kubernetes did not appeal to me, so I developed a KubernetesJob
implementation of SamzaJob.

You can find the details at,
but in summary KubernetesJob executes and generates a serialized JobModel.
Instead of interacting with Kubernetes directly to create the
SamzaContainers (as the YarnJob's SamzaApplicationMaster may do with the
YARN RM), it output a config YAML file that can be used to create the
SamzaContainers in Kubernetes by using Resource Controllers.  For this you
require to package your job as a Docker image.  You can reach the README at
the above repo for details.

A few observations:

It would be useful if SamzaContainer accepted the JobModel via an
environment variable.  Right not it expects a URL to download it from.  I
get around this by using a entry point script that copies the model from an
environment variable into a file, then passes a file URL to SamzaContainer.

SamzaContainer doesn't allow you to configure the JMX port.  It selects a
port at random from the ephemeral range as it expects to execute in YARN
where a static port could result in a conflict.  This is not the case in
Kubernetes where each Pod (i.e. SamzaContainer) is given its own IP

This implementation doesn't provide a Samza dashboard, which in the YARN
implementation is hosted in the Application Master.  There didn't seem to
be much value provided by the dashboard that is not already provided by the
Kubernetes tools for monitoring pods.

I've successfully executed the hello-samza jobs in Kubernetes:

$ kubectl get po
NAME                       READY     STATUS    RESTARTS   AGE
kafka-1-jjh8n              1/1       Running   0          2d
kafka-2-buycp              1/1       Running   0          2d
kafka-3-tghkp              1/1       Running   0          2d
wikipedia-feed-0-4its2     1/1       Running   0          1d
wikipedia-parser-0-l0onv   1/1       Running   0          17h
wikipedia-parser-1-crrxh   1/1       Running   0          17h
wikipedia-parser-2-1c5nn   1/1       Running   0          17h
wikipedia-stats-0-3gaiu    1/1       Running   0          16h
wikipedia-stats-1-j5qlk    1/1       Running   0          16h
wikipedia-stats-2-2laos    1/1       Running   0          16h
zookeeper-1-1sb4a          1/1       Running   0          2d
zookeeper-2-dndk7          1/1       Running   0          2d
zookeeper-3-46n09          1/1       Running   0          2d

Finally, accessing services within the Kubernetes cluster from the outside
is quite cumbersome unless one uses an external load balancer.  This makes
it difficult to bootstrap a job, as SamzaJob must connect to Zookeeper and
Kafka to find out the number of partitions on the topics it will subscribe
to, so it can assign them statically among the number of containers

Ideally Samza would operate along the lines of the Kafka high-level
consumer, which dynamically coordinate to allocate work among members of a
consumer group.  This would do away with the new to execute SamzaJob a
priori to generate the JobModel to pass to the SamzaContainers.  It would
also allow for dynamically changing the number of containers without having
the shutdown the job.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message