kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Uber Slacker <cubeslac...@gmail.com>
Subject Re: Kafka Connect concept question
Date Mon, 11 Apr 2016 15:33:52 GMT
Thanks for the explanations guys. It would be cool to see a section in the
documentation that explicitly compares and contrasts Kafka Connect versus
working directly with the producer and consumer APIs. That's just my
perspective as a newb - perhaps it's clear to others. Thanks again!

On Thu, Apr 7, 2016 at 4:41 PM, Ewen Cheslack-Postava <ewen@confluent.io>

> On Wed, Apr 6, 2016 at 8:56 AM, Uber Slacker <cubeslacker@gmail.com>
> wrote:
> > Hi folks. I'm pretty new to Kafka. I have spent a fair amount of time so
> > far understanding the Kafka system in general and how producers and
> > consumers work. I'm now trying to get a grasp on how Kafka Connect
> > compares/contrasts to Producers/Consumers written via the Java API.
> >
> > When might someone want to write their own Java Producer/Consumer versus
> > using a connector in Kafka Connect?  How does Kafka Connect use producers
> > and consumers behind the scenes? Why wouldn't we simply want a
> > producer/consumer library that contains producers and consumers written
> to
> > work with various external systems such as HDFS? Why this new framework?
> > Thanks for any clarification!
> >
> Internally Connect does use the producer and consumer. However, the
> framework adds a lot of support for functionality you want specifically
> when you are copying data from another system to Kafka or from Kafka to
> another system. Connect handles distribution and fault tolerance for you at
> the framework level. It provides a schema/data API and abstracts away
> details of serialization such that you can write a single connector and
> support multiple formats.
> If you're trying to copy data to/from another system, we'd generally
> recommend using the Connect framework since it adds all this extra support
> and allows you to focus only on how you get the data into/out of the other
> system. You'll want to use producers and consumers directly if you need
> more control that Connect hides from you, but then you'll need to create
> your own implementation of features Connect provides (or simply not support
> them).
> -Ewen

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message