kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Kafka Connect concept question
Date Thu, 07 Apr 2016 20:41:42 GMT
On Wed, Apr 6, 2016 at 8:56 AM, Uber Slacker <cubeslacker@gmail.com> wrote:

> Hi folks. I'm pretty new to Kafka. I have spent a fair amount of time so
> far understanding the Kafka system in general and how producers and
> consumers work. I'm now trying to get a grasp on how Kafka Connect
> compares/contrasts to Producers/Consumers written via the Java API.
>
> When might someone want to write their own Java Producer/Consumer versus
> using a connector in Kafka Connect?  How does Kafka Connect use producers
> and consumers behind the scenes? Why wouldn't we simply want a
> producer/consumer library that contains producers and consumers written to
> work with various external systems such as HDFS? Why this new framework?
> Thanks for any clarification!
>

Internally Connect does use the producer and consumer. However, the
framework adds a lot of support for functionality you want specifically
when you are copying data from another system to Kafka or from Kafka to
another system. Connect handles distribution and fault tolerance for you at
the framework level. It provides a schema/data API and abstracts away
details of serialization such that you can write a single connector and
support multiple formats.

If you're trying to copy data to/from another system, we'd generally
recommend using the Connect framework since it adds all this extra support
and allows you to focus only on how you get the data into/out of the other
system. You'll want to use producers and consumers directly if you need
more control that Connect hides from you, but then you'll need to create
your own implementation of features Connect provides (or simply not support
them).

-Ewen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message