lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Best practice for saving state of large cluster?
Date Fri, 28 Jun 2019 14:01:02 GMT
I agree - and it would provide you the opportunity to use snapshots for backups on S3.

> Am 28.06.2019 um 15:06 schrieb Kyle Fransham <kyle.fransham@superna.net>:
> 
> Just my two cents, but why not put your data on EBS volumes and decouple
> from the AMI? This way you're storing the collections in the "amazon
> suggested" way:
> https://aws.amazon.com/premiumsupport/knowledge-center/instance-store-vs-ebs/
> 
> 
> Also saves the (potentially error-prone) step of creating a new AMI when
> data has changed.
> 
> Kyle
> 
>> On Fri, Jun 28, 2019 at 8:27 AM chris <chris@yeeplusplus.com> wrote:
>> 
>> I have a cluster of 100 shards on 100 nodes, with solr 7.5, running in
>> AWS.The use case is read-dominant, with ingestion performed about once per
>> week. There are about 84 billion documents in the cluster. It is unused on
>> weekends and only used during normal business hours M-F.What I do now is
>> after each round of ingestion, create a new set of AMIs, then terminate
>> each instance.The next morning, the cluster is restarted by creating a new
>> set of spot requests, using the most recent AMIs. At the end of the day,
>> the cluster is turned off by terminating the instances (if no data was
>> changed), or by creating a new set of AMIs and then terminating the
>> instances.Is there a better way to do this? I'm not facing any real
>> problems with this setup, but I want to make sure I'm not missing something
>> obvious.Thanks,Chris
> 
> 
> 
> -- 
> 
> Kyle Fransham
> 
> Vice President, Research & Development
> 
> *Business Protection You Can Count On*
> 
> 104 Schneider Road | Kanata, Ontario K2K 1Y2
> <https://www.google.ca/maps/place/104+Schneider+Rd,+Kanata,+ON+K2K+1Y2/@45.340043,-75.9036994,17z/data=!3m1!4b1!4m2!3m1!1s0x4cd1fffa70d4d3f3:0xddf444ae3573dcc5>
> 
> tel 613-729-1100 | mobile 613-897-9414 <613-xxx-yyyy> |
> www.supernaeyeglass.com
> 
> <https://www.facebook.com/SupernaNET>  <https://twitter.com/SupernaNet>
> 
> -- 
> CONFIDENTIALITY NOTICE: The information contained in this email is 
> privileged and confidential and intended only for the use of the individual 
> or entity to whom it is addressed.   If you receive this message in error, 
> please notify the sender immediately at 613-729-1100 and destroy the 
> original message and all copies. Thank you.

Mime
View raw message