kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arya Ketan <ketan.a...@gmail.com>
Subject Re: [ANN] sqlstream: Simple MySQL binlog to Kafka stream
Date Wed, 18 Mar 2015 02:26:40 GMT
AFAIK , linkedin uses databus to do the same. Aesop is built on top of
databus , extending its beautiful capabilities to mysql n hbase
On Mar 18, 2015 7:37 AM, "Xiao" <lixiao1983@gmail.com> wrote:

> Hi, all,
>
> Do you know how Linkedin team publishes changed rows in Oracle to Kafka? I
> believe they already knew the whole problem very well.
>
> Using triggers? or directly parsing the log? or using any Oracle
> GoldenGate interfaces?
>
> Any lesson or any standard message format? Could the Linkedin people share
> it with us? I believe it can help us a lot.
>
> Thanks,
>
> Xiao Li
>
>
> On Mar 17, 2015, at 12:26 PM, James Cheng <jcheng@tivo.com> wrote:
>
> > This is a great set of projects!
> >
> > We should put this list of projects on a site somewhere so people can
> more easily see and refer to it. These aren't Kafka-specific, but most seem
> to be "MySQL CDC." Does anyone have a place where they can host a page?
> Preferably a wiki, so we can keep it up to date easily.
> >
> > -James
> >
> > On Mar 17, 2015, at 8:21 AM, Hisham Mardam-Bey <
> hisham.mardambey@gmail.com> wrote:
> >
> >> Pretty much a hijack / plug as well (=
> >>
> >> https://github.com/mardambey/mypipe
> >>
> >> "MySQL binary log consumer with the ability to act on changed rows and
> >> publish changes to different systems with emphasis on Apache Kafka."
> >>
> >> Mypipe currently encodes events using Avro before pushing them into
> Kafka
> >> and is Avro schema repository aware. The project is young; and patches
> for
> >> improvements are appreciated (=
> >>
> >> On Mon, Mar 16, 2015 at 10:35 PM, Arya Ketan <ketan.arya@gmail.com>
> wrote:
> >>
> >>> Great work.
> >>> Sorry for kinda hijacking this thread, but I though that we had built
> >>> some-thing on mysql bin log event propagator and wanted to share it .
> >>> You guys can also look into Aesop ( https://github.com/Flipkart/aesop
> ).
> >>> Its
> >>> a change propagation frame-work. It has relays which listens to bin
> logs of
> >>> Mysql, keeps track of SCNs  and has consumers which can then
> (transform/map
> >>> or interpret as is) the bin log-event to a destination. Consumers also
> keep
> >>> track of SCNs and a slow consumer can go back to a previous SCN if it
> wants
> >>> to re-listen to events  ( similar to kafka's consumer view ).
> >>>
> >>> All the producers/consumers are extensible and you can write your own
> >>> custom consumer and feed off the data to it.
> >>>
> >>> Common use-cases:
> >>> a) Archive mysql based data into say hbase
> >>> b) Move mysql based data to say a search store for serving reads.
> >>>
> >>> It has a decent ( not an awesome :) ) console too which gives a nice
> human
> >>> readable view of where the producers and consumers are.
> >>>
> >>> Current supported producers are mysql bin logs, hbase wall-edits.
> >>>
> >>>
> >>> Further insights/reviews/feature reqs/pull reqs/advices are all
> welcome.
> >>>
> >>> --
> >>> Arya
> >>>
> >>> Arya
> >>>
> >>> On Tue, Mar 17, 2015 at 1:48 AM, Gwen Shapira <gshapira@cloudera.com>
> >>> wrote:
> >>>
> >>>> Really really nice!
> >>>>
> >>>> Thank you.
> >>>>
> >>>> On Mon, Mar 16, 2015 at 7:18 AM, Pierre-Yves Ritschard <
> pyr@spootnik.org
> >>>>
> >>>> wrote:
> >>>>> Hi kafka,
> >>>>>
> >>>>> I just wanted to mention I published a very simple project which
can
> >>>>> connect as MySQL replication client and stream replication events
to
> >>>>> kafka: https://github.com/pyr/sqlstream
> >>>>>
> >>>>> When you don't have control over an application, it can provide
a
> >>> simple
> >>>>> way of consolidating SQL data in kafka.
> >>>>>
> >>>>> This is an early release and there are a few caveats (mentionned
in
> the
> >>>>> README), mostly the poor partitioning which I'm going to evolve
> quickly
> >>>>> and the reconnection strategy which doesn't try to keep track of
> binlog
> >>>>> position, other than that, it should work as advertised.
> >>>>>
> >>>>> Cheers,
> >>>>> - pyr
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Hisham Mardam-Bey
> >> http://hisham.cc/
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message