kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francis Siefken <fran...@axual.io>
Subject Re: Copy topic to offline file
Date Fri, 08 Jun 2018 08:21:27 GMT
Hi Ben, you mentioned 'We can't connect them via connect', but connect
with it's filestream export and import was my first thought. If you
want to export/import time segments you could use KSQL, would this not
fulfil your requirement?

Francis

On Thu, Jun 7, 2018 at 7:47 PM, Young, Ben
<Ben.Young@fisglobal.com.invalid> wrote:
> Hi
>
> We have a requirement that we need to be able to copy a portion of a Kafka topic to a
file (perhaps between a range of timestamps) which can then be loaded into a Kafka instance
on a different machine. This machine may have a different number of partitions etc for the
topic. We can't connect them via connect or anything, this is more for offline analysis of
data that produce issues in our software, or for providing data for regression tests.
>
> I'm happy to write a tool to do this (I think the OffsetsForTimestamp API does what I
need), but does anyone know of any tools out there that do something like this? I've had an
initial google, but I can't find anything
>
> Thanks,
> Ben
>
> Ben Young
> Principal Software Engineer - Risk
> Risk & Compliance
> T:  +44 (0) 1865 578140
> E: ben.young@fisglobal.com
> FIS | Empowering the Financial World
> FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered Office:
Level 39, 25 Canada Square, London E14 5LQ
>
> The information contained in this message is proprietary and/or confidential. If you
are not the intended recipient, please: (i) delete the message and all copies; (ii) do not
disclose, distribute or use the message in any manner; and (iii) notify the sender immediately.
In addition, please be aware that any message addressed to our domain is subject to archiving
and review by persons other than the intended recipient. Thank you.

Mime
View raw message