kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barot, Abhishek" <abhishek.ba...@bankofamerica.com>
Subject Topic with retention period of 10 mins
Date Tue, 24 Jan 2017 15:59:26 GMT
Hi Kafka-users,

We've set up a KAFKA(0.10.0.0) topic to feed to elasticsearch so that we can view the stuff
on Kibana dashboard but the problem I'm running into is that topic apparently gets flooded
with loads and loads of data and we've got limited
Disk space on the server to utilize. So in order to optimize the issue we thoughts of reducing
the default retention period for the topic from 7 days ( default ) to say only 10 mins the
reason for this is once we've data feed to elasticsearch we're good.
But now we ran into the issue with elasticsearch not being able to log messages and inspecting
server.log shows many such below lines.

[2017-01-24 15:04:05,191] INFO Scheduling log segment 349979273 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,203] INFO Scheduling log segment 360540990 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,207] INFO Scheduling log segment 369270578 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,244] INFO Scheduling log segment 376770736 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,244] INFO Scheduling log segment 384360562 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,244] INFO Scheduling log segment 391971484 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,244] INFO Scheduling log segment 399590110 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,244] INFO Scheduling log segment 407306178 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,245] INFO Scheduling log segment 415076641 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,256] INFO Scheduling log segment 422831782 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,256] INFO Scheduling log segment 430580404 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,256] INFO Scheduling log segment 438309156 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,257] INFO Scheduling log segment 446040902 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,257] INFO Scheduling log segment 453769192 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,257] INFO Scheduling log segment 461472996 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,257] INFO Scheduling log segment 469160801 for log logstash_qa-0 for
deletion. (kafka.log.Log)
[2017-01-24 15:04:05,257] INFO Scheduling log segment 476830839 for log logstash_qa-0 for
deletion. (kafka.log.Log)

Can anyone suggest what could be the better solution for this to continue feed the stream
of messages to elastic search without running into disk space issue with kafka topic :(.

Thanks!

----------------------------------------------------------------------
This message w/attachments (message) is intended solely for the use of the intended recipient(s)
and may contain information that is privileged, confidential or proprietary.  If you are not
an intended recipient, please notify the sender, and then please delete and destroy all copies
and attachments, and be advised that any review or dissemination of, or the taking of any
action in reliance on, the information contained in or attached to this message is prohibited.

Unless specifically indicated, this message is not an offer to sell or a solicitation of any
investment products or other financial product or service, an official confirmation of any
transaction, or an official statement of Sender.  Subject to applicable law, Sender may intercept,
monitor, review and retain e-communications (EC) traveling through its networks/systems and
may produce any such EC to regulators, law enforcement, in litigation and as required by law.

The laws of the country of each sender/recipient may impact the handling of EC, and EC may
be archived, supervised and produced in countries other than the country in which you are
located. This message cannot be guaranteed to be secure or free of errors or viruses.  Attachments
that are part of this EC may have additional important disclosures and disclaimers, which
you should read.   By messaging with Sender you consent to the foregoing.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message