kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject does kafka support "COMMIT" of a batch ?
Date Mon, 26 Jan 2015 20:06:39 GMT
we are pulling from a DB, maybe 100k rows at a time. for each row we send
it to kafka.

now the problem is , if any errors happen during the db pulling (such as an
earthquake), we stop.
next time we wake up. we can't write the same record again into kafka,
otherwise there might be over counting.

does kafka support a "BATCH mode" with commit ? so basically at the start
of my session, I declare "start transaction", and after all records have
been pulled and sent to kafka, I declare "commit" .


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message