kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pranavi Chandramohan <prana...@campaignmonitor.com>
Subject Re: Transactional markers are not deleted from log segments when policy is compact
Date Mon, 03 Jun 2019 06:02:08 GMT
Hi Jonathan,

After looking at the changes in PR, I would like to ensure the fix would
work for our scenario too.
We create a new TransactionId for every producer instance every time it
restarts and this would result in creating new ProducerIds. Old producers
and their transactions will get expired.
Our expectation is that the markers of inactive ProducerIds should be
deleted. Does this hold true?

Thanks,
Pranavic

On Sat, Jun 1, 2019 at 8:45 AM Pranavi Chandramohan <
pranavic@campaignmonitor.com> wrote:

> Thanks Jonathan! That should help.
>
> On Fri, 31 May 2019, 6:44 pm Jonathan Santilli, <
> jonathansantilli@gmail.com> wrote:
>
>> Hello Pranavi,
>>
>> it sounds like this was solved in the release candidate 2.2.1-RC1 (
>> https://issues.apache.org/jira/browse/KAFKA-8335)
>> Take a look at it.
>>
>> Hope that helps.
>>
>>
>> Cheers!
>> --
>> Jonathan
>>
>>
>>
>>
>> On Fri, May 31, 2019 at 8:59 AM Pranavi Chandramohan <
>> pranavic@campaignmonitor.com> wrote:
>>
>> > Hi all,
>> >
>> > We use Kafka version 2.11-1.1.1. We produce and consume transactional
>> > messages and recently we noticed that 2 partitions of the
>> __consumer_offset
>> > topic have very high disk usage (256GB)
>> > When we looked at the log segments for these 2 partitions, there were
>> files
>> > that were 6 months old.
>> > By dumping the content of an old log segment using the following command
>> >
>> > kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration
>> > --print-data-log  --files 00000000003949894887.log | less
>> >
>> >
>> > we found that all the records were COMMIT transaction markers.
>> >
>> > offset: 1924582627 position: 183 CreateTime: 1548972578376 isvalid:
>> > true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId:
>> > 126015 producerEpoch: 0 sequence: -1 isTransactional: true headerKeys:
>> > [] endTxnMarker: COMMIT coordinatorEpoch: 28
>> >
>> >
>> > Why are the commit transaction markers not compacted and deleted?
>> >
>> >
>> > Log cleaner config
>> >
>> > max.message.bytes 10000120
>> > min.cleanable.dirty.ratio 0.1
>> > compression.type uncompressed
>> > cleanup.policy compact
>> > retention.ms 2160000000
>> > segment.bytes 104857600
>> >
>> > # By default the log cleaner is disabled and the log retention policy
>> > will default to just delete segments after their retention expires.
>> > # If log.cleaner.enable=true is set the cleaner will be enabled and
>> > individual logs can then be marked for log compaction.
>> > log.cleaner.enable=true
>> > # give larger heap space to log cleaner
>> > log.cleaner.dedupe.buffer.size=1342177280
>> >
>>
>>
>> --
>> Santilli Jonathan
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message