kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Clarke <liam.cla...@adscale.co.nz>
Subject Re: Questions for platform to choose
Date Wed, 21 Aug 2019 07:14:58 GMT
 C Hi Eliza,

Kafka Streaming, Spark Streaming, Flink and Storm are all good. They also
all have their caveats. It's really hard to say that X is the best.

For example, Kafka Streaming can't read from one Kafka cluster and write to
another, but Spark can.

But then Spark offers two flavours of streaming, the low level and fiddly
to integrate with Kafka RDD based streaming, or the higher level Dataframe
based structured streaming that integrates a lot easier with Kafka, but
currently doesn't support a group by followed by a group by.

Spark requires either manually creating and managing a cluster to scale, or
else using Yarn or EMR, whereas Kafka Streaming is straightforward to scale
by deploying another copy of the app.

I can keep going...

You really need to analyse what you're trying to achieve, what existing
expertise you have in your organisation, and then just try the various
technologies.

On Wed, 21 Aug. 2019, 6:42 pm Eliza, <eli@chinabuckets.com> wrote:

> Hello,
>
> We have all of spark, flink, storm, kafka installed.
> For realtime streaming calculation, which one is the best above?
> Like other big players, the logs in our stack are huge.
>
> Thanks.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message