kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com.INVALID>
Subject Re: KOYA vs. Samza?
Date Sat, 17 Jan 2015 00:16:45 GMT
Hey Otis,

I think the key phrase in Samza's description is:

"Apache Samza is a distributed stream processing framework"

Kafka is not a stream processing framework. It's a message queue/broker
system. Stream processing frameworks (e.g. Storm, Spark Streaming, Samza,
etc) use message queueing/brokering systems to pass messages within the
stream processing framework.

Samza is more akin to Storm or Spark Streaming. KOYA is just putting Kafka
brokers in a YARN grid.

At least, that's my understanding.

Cheers,
Chris

On 1/16/15 4:05 PM, "Otis Gospodnetic" <otis.gospodnetic@gmail.com> wrote:

>Hm.  My understanding was that both are aimed at basically the same thing
>-
>Kafka on YARN.  From Samza site:
>"Apache Samza is a distributed stream processing framework. It uses Apache
>Kafka <http://kafka.apache.org/> for messaging, and Apache Hadoop YARN
><http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.h
>tml>
>to
>provide fault tolerance, processor isolation, security, and resource
>management."
>
>And KOYA:
>"KOYA is a YARN application that launches Kafka within YARN. It then
>manages the resource negotiation with Resource Manager, and ensures that
>Kafka operates in a YARN native way. For an external publisher or
>subscriber, KOYA would not look any different than Kafka since the same
>code is being run as a YARN application."
>
>It looks like they were never mentioned together.... until now :)
>
>Otis
>--
>Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>Solr & Elasticsearch Support * http://sematext.com/
>
>
>On Fri, Jan 16, 2015 at 3:34 PM, Chris Riccomini <
>criccomini@linkedin.com.invalid> wrote:
>
>> Hey Otis,
>>
>> I'm not terribly familiar with KOYA, but my understanding is that it's a
>> tool for deploying Kafka brokers to YARN, and administering them. I
>>don't
>> think that it has any stream processing functionality built into it. As
>> such, it seems to me that KOYA and Samza could be used together: you
>>could
>> use KOYA to deploy Kafka in YARN, and Samza to read/write messages from
>> the brokers that have been deployed.
>>
>> Samza provides containers that have consumers/producers in them, and
>>allow
>> you to plug in processing logic as new messages arrive. Samza's goal is
>>to
>> provide features that are useful when you're processing the messages,
>>such
>> as fault tolerance (restarting consumers/producers when they fail),
>> checkpointing (saving offsets), state management (if you're counting
>> messages, you want to make sure your count is accurate even if you
>>fail),
>> etc.
>>
>> Cheers,
>> Chris
>>
>> On 1/16/15 12:14 PM, "Otis Gospodnetic" <otis.gospodnetic@gmail.com>
>> wrote:
>>
>> >Hi,
>> >
>> >I was wondering if anyone can compare and contrast KOYA and Samza?
>> >
>> >Thanks,
>> >Otis
>> >--
>> >Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> >Solr & Elasticsearch Support * http://sematext.com/
>>
>>


Mime
View raw message