lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trey Grainger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud
Date Wed, 22 Jun 2016 02:36:57 GMT

    [ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343228#comment-15343228
] 

Trey Grainger commented on SOLR-9241:
-------------------------------------

I'm also very excited to see this patch. For the next evolution of Solr's scalability (and
ultimately auto-scaling), these are exactly the kinds of core capabilities we need for seamlessly
scaling up/down, resharding, and redistributing shards and replicas across a cluster. 

The smart merge looks interesting - seems like effectively a way to index into a larger number
of shards (for indexing throughput) while merging them into a smaller number of shards for
searching, enabling scaling of indexing and searching resourced independently. This obviously
won't work well with Near-Realtime Searching, but I'd be curious to hear more explanation
about how this works in practice for SolrCloud clusters that don't need NRT search.

Agreed with Joel's comments about the update to trunk vs. 4.6.1. One thing that seems to have
been added since 4.6.1 that probably overlaps with this patch is the Replica Placement Strategies
(SOLR-6220) vs. the Allocation Strategies implemented here.

The rest of the patch seems like all new objects that don't overlap much with the current
code base. Would be interesting to know how much has changed between 4.6.1 to 6.1 collections/SolrCloud-wise
that would create conflicts with this patch. Am obviously hoping not too much...

Either way, very excited about the contribution and about the potential for getting these
capabilities integrated into Solr.

> Rebalance API for SolrCloud
> ---------------------------
>
>                 Key: SOLR-9241
>                 URL: https://issues.apache.org/jira/browse/SOLR-9241
>             Project: Solr
>          Issue Type: New Feature
>          Components: SolrCloud
>    Affects Versions: 4.6.1
>         Environment: Ubuntu, Mac OsX
>            Reporter: Nitin Sharma
>              Labels: Cluster, SolrCloud
>             Fix For: 4.6.1
>
>         Attachments: rebalance.diff
>
>   Original Estimate: 2,016h
>  Remaining Estimate: 2,016h
>
> This is the v1 of the patch for Solrcloud Rebalance api (as described in http://engineering.bloomreach.com/solrcloud-rebalance-api/)
, built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API  is to provide
a zero downtime mechanism to perform data manipulation and  efficient core allocation in solrcloud.
This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling
platform. (and work in unison with other complementing monitoring and scaling features).
> Patch Status:
> ===============
> The patch is work in progress and incremental. We have done a few rounds of code clean
up. We wanted to get the patch going first to get initial feed back.  We will continue to
work on making it more open source friendly and easily testable.
>  Deployment Status:
> ====================
> The platform is deployed in production at bloomreach and has been battle tested for large
scale load. (millions of documents and hundreds of collections).
>  Internals:
> =============
> The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/
> It is built on top of the admin collections API as an action (with various flavors).
At a high level, the rebalance api provides 2 constructs:
> Scaling Strategy:  Decides how to move the data.  Every flavor has multiple options which
can be reviewed in the api spec.
> Re-distribute  - Move around data in the cluster based on capacity/allocation.
> Auto Shard  - Dynamically shard a collection to any size.
> Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller
one.  (the source should be divisible by destination)
> Scale up -  Add replicas on the fly
> Scale Down - Remove replicas on the fly
> Allocation Strategy:  Decides where to put the data.  (Nodes with least cores, Nodes
that do not have this collection etc). Custom implementations can be built on top as well.
One other example is Availability Zone aware. Distribute data such that every replica is placed
on different availability zone to support HA.
>  Detailed API Spec:
> ====================
>   https://github.com/bloomreach/solrcloud-rebalance-api
>  Contributors:
> =====================
>   Nitin Sharma
>   Suruchi Shah
>  Questions/Comments:
> =====================
>   You can reach me at nitin.sharma@bloomreach.com



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message