ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Rebalancing speedup (IGNITE-1093)
Date Wed, 05 Aug 2015 15:50:46 GMT
Anton,

Good points!

Can you please copy the design points made in this discussion to the Jira
ticket?

D.

On Wed, Aug 5, 2015 at 2:57 AM, Anton Vinogradov <avinogradov@gridgain.com>
wrote:

> Parallel Demand pool seems to be ready.
> Working on parallel Supply pool.
>
> General idea is to use systemPool to handle demand requests. Amount ot
> parallel workers will be limited indirectly by Demand poll's parallelism
> limits.
> But overload still can happens in this case. Will write load tests.
>
> Next idea is to go through whole partition and when
> s.messageSize() >= cctx.config().getRebalanceBatchSize()
> send supply batch, store current iterator, swapLsnr and etc. and return
> from supply thread.
> When same node will make next request then supplyPool just continue
> iterating and give next or last batch for requested partition.
>
> Also speed can be increased if SupplyPool will prepare N batches on first
> demand request and will generate additional on each new request.
>
> On Thu, Jul 30, 2015 at 9:41 PM, Dmitriy Setrakyan <dsetrakyan@apache.org>
> wrote:
>
> > Anton,
> >
> > Very good point. Valentin is already doing per-partition topic in his
> work
> > for Continuous Queries:
> > https://issues.apache.org/jira/browse/IGNITE-104
> >
> > Perhaps it makes sense for both of you to sync up.
> >
> > Also, as a side note, whenever starting a dev list discussion about a
> > ticket, it always helps to include a link to the dev list thread into
> Jira
> > (I already did it for IGNITE-1093).
> >
> > D.
> >
> > On Thu, Jul 30, 2015 at 11:31 AM, Anton Vinogradov <
> > avinogradov@gridgain.com
> > > wrote:
> >
> > > Hello,
> > >   I think that speed of rebalancing can be increased by parallel
> handling
> > > of supply messages.
> > >   Main Idea is to use per-partition topics.
> > >
> > >   At the start of DemandPool we can create topics for every partition
> and
> > > handle each partition's supply messages at separated listener. In this
> > case
> > > only one thread will request lock for partition it have to relocate.
> > >
> > >   Also we can do the same at SupplyPool, and make supply messages in
> > > parallel way.
> > >
> > >   Amount of simultaneous messages should be limited to prevent
> > overloading.
> > > Current idea is to request N partitions at start and request one next
> > > partition when one of prevous relocated.
> > >
> > >   Per-partition listeners can be created this way:
> > > for (int p = 0; p <= maxP; p++) {
> > >     cctx.io().addOrderedHandler(topic(p), new CI2<UUID,
> > > GridDhtPartitionSupplyMessage>() {
> > >         @Override public void apply(UUID nodeId,
> > > GridDhtPartitionSupplyMessage msg) {
> > >             handleSupplyMessage(new SupplyMessage(nodeId, msg));
> > >         }
> > >     });
> > > }
> > >
> > > Thoughts?
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message