kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: kafka-reassign-partitions.sh leaves data on source broker?
Date Sun, 01 Jun 2014 00:32:15 GMT
You may take a look at the broker configs for replication factor:

http://kafka.apache.org/documentation.html#brokerconfigs

With replication factor 1 (default value) the broker holding the old leader
should received StopReplica Request to stop the replica and delete the
segment files. Do you see this log entries on broker 2's request logs?

Guozhang


On Sat, May 31, 2014 at 5:15 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi Guozhang,
>
> I don't have any replication-related properties in my config file and I
> don't see any mentions of replication at startup in the log, so I don't
> really know.
>
> The only time I saw replication mentioned in the log was during some errors
> that said my replication factor was 1 and that was > the number of brokers.
>  So based on this error message I'd say my replication factor is 1.
>
> Is there a replication property I could explicitly set?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Sat, May 31, 2014 at 7:05 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>
> > Hello Otis,
> >
> > What is your replication factor?
> >
> > Guozhang
> >
> >
> > On Sat, May 31, 2014 at 12:39 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I have 2 pretty unbalanced brokers running 0.8.1.1 and I'm trying to
> > > balance them (read: free disk space on the overloaded broker).
> > >
> > > Is there a tool that will just rebalance everything?  I couldn't find
> > one,
> > > so I ran this:
> > >
> > > $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> > > zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> > > --execute
> > >
> > > This seems to have *copied* about half of the partitions of the topic
> > > specified in the given JSON file from broker 2 (source) to broker 1
> > > (target). Good.
> > >
> > > I see this in broker 2 (source broker):
> > >
> > > [2014-05-31 19:24:59,994] INFO [ReplicaFetcherManager on broker 2]
> > Removed
> > > fetcher for partitions [spm_new_cluster_free_system_topic,1]
> > > (kafka.server.ReplicaFetcherManager)
> > >
> > > ... which looks promising.... but:
> > >
> > > My broker 2 (source) *still* has all partitions of the problematic
> topic.
> > > I thought it would remove the moved partitions from the source
> broker....
> > > no?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Performance Monitoring * Log Analytics * Search Analytics
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message