kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eugene miretsky <eugene.miret...@gmail.com>
Subject Re: Kakfa Connect Converter per Connector
Date Tue, 02 Feb 2016 17:15:20 GMT
Hi Eric,

We have a slightly different use case where we publish to Kafka using a
(modified) Connect Source and are using Spark Streaming to read the data
from Kafka and write to C* - it was really easy to write  simple code to
parse SchemaAndValue objects.

Setting up Spark Streaming is extremely easy (it has a built-in Kafka
consumer), and it provides all the fault tolerance, and distribution you
need. That said, it may be an overkill if all you need is to copy the data
over (we do a bunch of ETL in Spark Streaming).

A Connect Sink for Cassandra would be great. Let me know if you are indeed
planing to build and open source it, I'll be interested in contributing.

Cheers,

Eugene
On Jan 27, 2016 5:32 PM, "Gwen Shapira" <gwen@confluent.io> wrote:

> Hi Eric,
>
> 1. You are correct that the way to handle custom data formats in Kafka is
> to use a custom convertor.
> 2. You are also correct that we are currently assuming one converter per
> Connect instance / cluster that all connectors share (in the hope that each
> organization has one common data format...)
> If you need more data formats, the current work around is to run multiple
> KafkaConnect instances or clusters.
>
> I hope you have plans to open source your Cassandra connector! It will be
> very popular :)
> And if you decide to write your own Convertor, let us know if you have
> specific questions on how to proceed.
>
> Gwen
>
> On Wed, Jan 27, 2016 at 11:41 AM, Eric Lachman <
> eric.lachman@spottradingllc.com> wrote:
>
> > Hi,
> >
> > I am trying out Kafka connect and have a couple questions. We are
> directly
> > publishing raw binary data to kafka from one of our apps and wanted to
> > create a Kafka Connector Sink to move the raw data to something like
> > Cassandra. Since this data is directly published to Kafka it doesn't have
> > any of the Kafka Connect metadata such as Schema. So our Cassandra Sink
> > Connector fails at parsing it. It seems we could write a custom converter
> > to handle this raw data and fill out SchemaAndValue in very basic way to
> > make it work. I'm not sure if this is the correct approach. If it is, it
> > seems the only way to use the converter, at least in the stand alone
> mode,
> > is to use the same converter for all of Connect, because it doesn't look
> > like we could overwrite the converter config on a connector basis. So if
> we
> > were to write new Sinks and Sources it would have to use our custom
> > converter instead of the default.
> >
> > Thanks,
> >
> > Eric
> >
> >
> > ________________________________
> >
> >     Eric Lachman
> >      Software Developer I
> >
> >      Spot Trading L.L.C
> >      440 South LaSalle St., Suite 2800
> >      Chicago, IL 60605
> >      Office: 312.362.4550
> >      Direct:
> >      Fax: 312.362.4551
> >      eric.lachman@spottradingllc.com
> >      www.spottradingllc.com<http://www.spottradingllc.com/>
> >
> > ________________________________
> >
> > The information contained in this message may be privileged and
> > confidential and protected from disclosure. If the reader of this message
> > is not the intended recipient, or an employee or agent responsible for
> > delivering this message to the intended recipient, you are hereby
> notified
> > that any dissemination, distribution or copying of this communication is
> > strictly prohibited. If you have received this communication in error,
> > please notify us immediately by replying to the message and deleting it
> > from your computer. Thank you. Spot Trading, LLC
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message