kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Powis <spo...@salesforce.com>
Subject Re: Mirrormaker producing to only one partition in a topic
Date Wed, 30 May 2018 00:09:23 GMT
Hey Ryan,

I ran into a similar issue and it was how the RoundRobinAssignor/Partitioner
was hashing the keys in my messages.  You may want to look at how thats
implemented and see if its causing all of your messages to end up in the
same partition.

For what its worth, this ticket has the implementation I put together to
deal with it: https://issues.apache.org/jira/browse/KAFKA-3333

Stephen

On Wed, May 30, 2018 at 5:28 AM, Ryan En <Ryan_526@hotmail.com> wrote:

> Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the
> messages from one Kafka cluster to another.
>
>
> The source and target Kafka cluster are pretty much set up the same...
> replication factor is 3, number of partitions is 3,
> auto.create.topics.enable is true.
>
>
> I am finding that in the target Kafka cluster, all the messages produced
> by Mirrormaker are going into one partition only.
>
>
> Here are my configs:
>
>
> mirrormaker producer
>
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required username="xxxxxxxx" password="xxxxxxxxxx";
> ssl.truststore.location=/path/to/truststore.jks
> ssl.truststore.password=xxxxxxxxxxx
> ssl.keystore.location=/path/to/keystore.jks
> ssl.keystore.password=xxxxxxxxx
> ssl.key.password=xxxxxxxx
> retries=5000000
> max.in.flight.requests.per.connection=1
> client.id=xxxxxxxx
> compression.type=lz4
> bootstrap.servers=<aggregate-broker>:9292
> acks=all
>
>
> mirrormaker consumer
>
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required username="xxxxxxxx" password="xxxxxxxxxx";
> ssl.truststore.location=/path/to/truststore.jks
> ssl.truststore.password=xxxxxxxxxx
> ssl.keystore.location=/path/to/keystore.jks
> ssl.keystore.password=xxxxxxxxxx
> ssl.key.password=xxxxxxxxx
> client.id=xxxxxxxxxxxx
> partition.assignment.strategy=org.apache.kafka.clients.
> consumer.RoundRobinAssignor
> auto.commit.interval.ms=1000
> auto.offset.reset=earliest
> bootstrap.servers=<local-broker>:9192
> session.timeout.ms=30000
> group.id=xxxxxxxxxxxxxx
> enable.auto.commit=false
>
>
> Any ideas why this may happen?
>
>
> Thanks in advance
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message