kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sven Ludwig" <s_lud...@gmx.de>
Subject Aw: Re: Broker-side Compression and Migration to lz4
Date Fri, 07 Jul 2017 18:14:39 GMT
Hi,

thanks. On another aspect in this: On a cluster that has topics with data, is it safe to gracefully
shutdown a broker, change the configured compression from snappy to lz4 and start the broker?
What will haben if one does that? All afterwards created new topics will come with lz4 compression.
But what happens to the existing topics in snappy?

Regards
Sven
 

Gesendet: Montag, 26. Juni 2017 um 13:10 Uhr
Von: "Ismael Juma" <ismael@juma.me.uk>
An: "Kafka Users" <users@kafka.apache.org>
Betreff: Re: Broker-side Compression and Migration to lz4
Hi Sven,

If you change the topic config, any new data received by that broker will
be in the new compression type. However, followers don't uncompress data,
so they will store the data as it was in the leader. An easier way to test
what you are trying to test is to use MirrorMaker to mirror the data to
another cluster.

Ismael

On Mon, Jun 26, 2017 at 11:59 AM, Sven Ludwig <s_ludwig@gmx.de> wrote:

> Hi,
>
>
> 1. I would like to test lz4 compression on a new broker node that I want
> add to my cluster, while the other nodes remain in snappy, in order to
> compare disk usage etc. I am not sure if this scenario is even possible
> with only one cluster, since in the docs it is mentioned that
> compression.type is a topic-level setting, and thus maybe not really per
> broker. I want to use Broker-side compression only. The situation is:
>
> broker 1 - Status: Up - compression.type snappy
> broker 2 - Status: Up - compression.type snappy
> broker 3 - Status: Down - compression.type lz4
>
> The topic for the test already exists on brokers 1 and 2 which are up and
> has some 20GB content.
>
> When I start broker 3, will it use snappy or lz4 for the topic?
>
>
> 2. How can one migrate a cluster (or a topic) from one broker-side
> compression type to another?
>
>
> Kind Regards,
> Sven
>

Mime
View raw message