kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject Re: does kafka support "COMMIT" of a batch ?
Date Mon, 26 Jan 2015 21:55:41 GMT
well I guess this is what I need
https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka

anyone knows how far this effort has gone?

thanks
Yang

On Mon, Jan 26, 2015 at 12:06 PM, Yang <teddyyyy123@gmail.com> wrote:

> we are pulling from a DB, maybe 100k rows at a time. for each row we send
> it to kafka.
>
> now the problem is , if any errors happen during the db pulling (such as
> an earthquake), we stop.
> next time we wake up. we can't write the same record again into kafka,
> otherwise there might be over counting.
>
>
> does kafka support a "BATCH mode" with commit ? so basically at the start
> of my session, I declare "start transaction", and after all records have
> been pulled and sent to kafka, I declare "commit" .
>
> Thanks
> Yang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message