kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sa Li <sal...@gmail.com>
Subject Re: messages lost
Date Fri, 02 Jan 2015 18:51:52 GMT
Thanks a lot, Tim, this is the config of brokers

----------
broker.id=1
port=9092
host.name=10.100.70.128
num.network.threads=4
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
auto.leader.rebalance.enable=true
auto.create.topics.enable=true
default.replication.factor=3

log.dirs=/tmp/kafka-logs-1
num.partitions=8

log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=536870912
log.cleanup.interval.mins=1

zookeeper.connect=10.100.70.128:2181,10.100.70.28:2181,10.100.70.29:2181
zookeeper.connection.timeout.ms=1000000

-----------------------


We actually play around request.required.acks in producer config, -1 cause
long latency, 1 is the parameter to cause messages lost. But I am not sure,
if this is the reason to lose the records.


thanks

AL







On Fri, Jan 2, 2015 at 9:59 AM, Timothy Chen <tnachen@gmail.com> wrote:

> What's your configured required.acks? And also are you waiting for all
> your messages to be acknowledged as well?
>
> The new producer returns futures back, but you still need to wait for
> the futures to complete.
>
> Tim
>
> On Fri, Jan 2, 2015 at 9:54 AM, Sa Li <salicn@gmail.com> wrote:
> > Hi, all
> >
> > We are sending the message from a producer, we send 100000 records, but
> we
> > see only 99573 records for that topics, we confirm this by consume this
> > topic and check the log size in kafka web console.
> >
> > Any ideas for the message lost, what is the reason to cause this?
> >
> > thanks
> >
> > --
> >
> > Alec Li
>



-- 

Alec Li

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message