kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Laursen <llaur...@rubiconproject.com>
Subject Re: Hitting integer limit when setting log segment.bytes
Date Thu, 14 May 2015 00:57:03 GMT
Hey folks,

Any update on this?

On Thu, Apr 30, 2015 at 5:34 PM, Lance Laursen <llaursen@rubiconproject.com>
wrote:

> Hey all,
>
> I am attempting to create a topic which uses 8GB log segment sizes, like
> so:
> ./kafka-topics.sh --zookeeper localhost:2181 --create --topic perftest6p2r
> --partitions 6 --replication-factor 2 --config max.message.bytes=655360
> --config segment.bytes=8589934592
>
> And am getting the following error:
> Error while executing topic command For input string: "8589934592"
> java.lang.NumberFormatException: For input string: "8589934592"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:583)
> ...
> ...
>
> Upon further testing with --alter topic, it would appear that
> segment.bytes will not accept a value higher than 2,147,483,647, which is
> the upper limit for a signed 32bit int. This then restricts log segment
> size to an upper limit of ~2GB.
>
> We run Kafka on hard drive dense machines, each with 10gbit uplinks. We
> can set ulimits higher in order to deal with all the open file handles
> (since Kafka keeps all log segment file handles open), but it would be
> preferable to minimize this number, as well as minimize the amount of log
> segment rollover experienced at high traffic (ie: a rollover every 1-2
> seconds or so when saturating 10gbe).
>
> Is there a reason (performance or otherwise) that a 32 bit integer is used
> rather than something larger?
>
> Thanks,
> -Lance
>
>


-- 

[image: elogo.png]

Leading the Automation of Advertising

LANCE LAURSEN | Systems Architect

••• (M) 310.903.0546

12181 BLUFF CREEK DRIVE, 4TH FLOOR, PLAYA VISTA, CA 90094

RUBICONPROJECT.COM <http://www.rubiconproject.com/> | @RUBICONPROJECT
<http://twitter.com/rubiconproject>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message