kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Young, Ben" <Ben.Yo...@fisglobal.com.INVALID>
Subject RE: Copy topic to offline file
Date Fri, 08 Jun 2018 09:48:49 GMT
Thanks Francis,

I think it's because of the ad-hoc request nature of what we want. It's not something we always
want on. I'll take a look at KSQL

Thanks,
Ben

-----Original Message-----
From: Francis Siefken [mailto:francis@axual.io]
Sent: 08 June 2018 09:21
To: users@kafka.apache.org
Subject: Re: Copy topic to offline file

Hi Ben, you mentioned 'We can't connect them via connect', but connect with it's filestream
export and import was my first thought. If you want to export/import time segments you could
use KSQL, would this not fulfil your requirement?

Francis

On Thu, Jun 7, 2018 at 7:47 PM, Young, Ben <Ben.Young@fisglobal.com.invalid> wrote:
> Hi
>
> We have a requirement that we need to be able to copy a portion of a Kafka topic to a
file (perhaps between a range of timestamps) which can then be loaded into a Kafka instance
on a different machine. This machine may have a different number of partitions etc for the
topic. We can't connect them via connect or anything, this is more for offline analysis of
data that produce issues in our software, or for providing data for regression tests.
>
> I'm happy to write a tool to do this (I think the OffsetsForTimestamp
> API does what I need), but does anyone know of any tools out there
> that do something like this? I've had an initial google, but I can't
> find anything
>
> Thanks,
> Ben
>
> Ben Young
> Principal Software Engineer - Risk
> Risk & Compliance
> T:  +44 (0) 1865 578140
> E: ben.young@fisglobal.com
> FIS | Empowering the Financial World
> FIS Systems Limited * Registered in England and Wales No. 1937159 *
> Registered Office: Level 39, 25 Canada Square, London E14 5LQ
>
> The information contained in this message is proprietary and/or confidential. If you
are not the intended recipient, please: (i) delete the message and all copies; (ii) do not
disclose, distribute or use the message in any manner; and (iii) notify the sender immediately.
In addition, please be aware that any message addressed to our domain is subject to archiving
and review by persons other than the intended recipient. Thank you.
The information contained in this message is proprietary and/or confidential. If you are not
the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose,
distribute or use the message in any manner; and (iii) notify the sender immediately. In addition,
please be aware that any message addressed to our domain is subject to archiving and review
by persons other than the intended recipient. Thank you.
Mime
View raw message