kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joris Peeters <j.peet...@wintoncapital.com>
Subject high level consumer timeout?
Date Wed, 23 Sep 2015 08:17:16 GMT
I have a topic with three partitions, to which I send 5 messages (very rapidly after one another)
and they get partitioned well ([1,2,2] as it happens).
I'm launching three identical high level java customers (kafka 8.2.1), single-threaded, to
consume those messages. Once a message is received, the consumers have to do a fair amount
of work, which typically takes a few minutes. Right now I'm committing the offset after every
message has been completed (so I have auto.commit.enable to false *).

What I am seeing, with this setup, is that the three consumers each work through their first
message and then just hang (in the subsequent "it.hasNext()"). The remaining two messages
do not arrive. When, on the other hand, I set auto.commit.enable to true, everything goes
well and the three consumers churn through the five messages without trouble.

This makes me suspect there is some sort of heartbeat issue going on, that is triggered by
the auto.commit? We don't want to auto-commit in this case, though, so does anyone know of
a different solution that could be tried?

My situation sounds somewhat similar to http://grokbase.com/t/kafka/users/14ck70w38q/kafka-consumer-session-timeouts
- but I am not sure if anyone got to the bottom of that.

As always, any help would be much appreciated!

(*) These are the consumer properties I have set explicitly:

        props.put("auto.commit.interval.ms", "1000");
        props.put("auto.commit.enable", "false"); // or true.
        props.put("offsets.storage", "kafka");

Winton Capital Management Limited ("Winton") is a limited company registered in England and
Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No.
3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United
Kingdom, registered as an investment adviser with the US Securities and Exchange Commission,
registered with the US Commodity Futures Trading Commission and a member of the National Futures
Association in the United States.

This communication, including any attachments, is confidential and may be privileged. This
email is for use by the intended recipient only. If you receive it in error, please notify
the sender and delete it. You should not copy or disclose all or any part of this email.

This email does not constitute an offer or solicitation and nothing contained in this email
constitutes, and should not be construed as, investment advice. Prospective investors should
request offering materials and consult their own advisers with respect to investment decisions
and inform themselves as to applicable legal requirements, exchange control regulations and
taxes in the countries of their citizenship, residence or domicile. Past performance is not
indicative of future results.

Winton takes reasonable steps to ensure the accuracy and integrity of its communications,
including emails. However Winton accepts no liability for any materials transmitted. Emails
are not secure and cannot be guaranteed to be error free.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message