ode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sudharma subasinghe <suba...@cse.mrt.ac.lk>
Subject Re: Deploying process in the cluster
Date Thu, 28 May 2015 16:03:53 GMT
Hi,

I understood the problem within dynamic master/slave configuration. In my
approach, when a deployment request is routed to a slave node there will
not be a deployment. I suggest two options to avoid it.
1) Have static master/slave configuration only for deploy process
2) Modify the deployment web service to complie and verify the process and
then copy it to the deploy folder irrespective of whether its a master or
slave, then deployment poller should take care of the deployment


On 28 May 2015 at 14:43, Sathwik B P <sathwik.bp@gmail.com> wrote:

> Sudharma,
>
> We definitely need a master/slave in the hazelcast cluster. This is
> probably needed for the job migration in the Scheduler to migrate the jobs
> associated with a down node. Let hold on this topic for future discussion.
>
> Going by the explanation where the master/slave nodes have certain
> predefined tasks to perform is perfectly fine.
>
> I have this scenario,
>
> I am using HAProxy as my load balancer and configured 3 nodes in the
> cluster.
>
> Node1 - Active
> Node2 - Active
> Node3 - Backup
>
> Load balancing algorithm: RoundRobin
>
> A Backup node (Node3) is one which the load balancer will not route
> requests to, until one of the Active node i.e either Node1 or Node2 has
> gone down.
>
> All these 3 nodes are also part of the hazelcast cluster as well.
>
> In the hazelcast cluster, assume Node1 is elected as the leader/master and
> Node2,Node3 as slaves.
>
> I initiate the deploy operation on the DeploymentWebService which the load
> balancer routes it to one of the Active nodes in the cluster, lets say it's
> the Node1. Since Node1 is also the master in the hazelcast cluster,
> deployment is a success.
>
> I initiate another deploy operation on the DeploymentWebService which the
> load balancer routes it to the next active node which is Node2. Since Node2
> is a slave in the Hazelcast cluster, What happens to the deployment?
>
> regards,
> sathwik
>
> On Wed, May 27, 2015 at 10:55 PM, sudharma subasinghe <
> suba.11@cse.mrt.ac.lk
> > wrote:
>
> > Hi,
> >
> > I will explain my approach as much as possible. The oldest node in the
> > hazelcast cluster is elected as the master node. In the failure of the
> > master node, next oldest node will be elected as the master node. This
> > master-slave configuration is just for deployment. When the hazelcast
> > cluster elected the master node, that node becomes a master node for
> > deploying process. So it will do the deploying artifacts. If you want to
> > get the idea of electing master node please refer the code which I have
> > located in the github. (
> > https://github.com/Subasinghe/ode/tree/ode_clustering)
> >
> > I identified separated actions which should be followed by the master and
> > salve nodes.
> > Actions which are followed by master node only
> > 1) create deployment unit
> > 2) set the version nu to deployment unit
> > 3) compile deployment unit
> > 4) scan deployment unit
> > 5) retire previous versions
> > Master node and slave nodes should create _processes which stores
> > ProcessConfImpl
> > Only master node will write the version nu to database, create .deployed
> > file
> >
> > So there are some actions which should be followed only by master node
> > while other actions should be followed by all the nodes.The idea of
> having
> > a master node is deploying artifacts and avoid others from writing the
> > version nu to database.
> > Whether a node is active or passive, all nodes should do the
> > deployment.Master
> > and slaves will follow necessary actions as in above.
> >
> >
> >
> >
> >
> > On 27 May 2015 at 15:49, Sathwik B P <sathwik.bp@gmail.com> wrote:
> >
> > > Nandika,
> > >
> > > I very well understand what you have put across, but it's secondary to
> me
> > > now.
> > >
> > > Sudharma,
> > > My primary concern is to understand at a high level the deployment
> > > architecture and how would master-slave configuration fit in. Are there
> > any
> > > restrictions imposed by the in-progress design?
> > >
> > > Firstly, how would ODE process deployment work under these cluster
> > > configurations?
> > >
> > > Sample Cluster configurations: A load balancer is frontending the
> > servers.
> > > 1) Cluster consisting of 2 nodes all Active-Active.
> > > 2) Cluster consisting of 2 nodes Active-Passive.
> > > 3) Cluster with 2+ nodes with additional nodes either in Active or
> > Passive.
> > >
> > > regards,
> > > sathwik
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, May 27, 2015 at 3:04 PM, Nandika Jayawardana <
> jayawark@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Sathwik,
> > > >
> > > > According to my understanding, in the clustering scenario, the master
> > > node
> > > > should perform all the deployment actions and the slave nodes also
> need
> > > to
> > > > perform some deployment actions. For example, the slave nodes also
> > should
> > > > handle the process ACTIVATED event so that the process configuration
> is
> > > > added to the engine and necessary web services are created so that
> when
> > > the
> > > > load balancer send requests to any node in the cluster, it is ready
> to
> > > > accept those requests.
> > > >
> > > > Regards
> > > > Nandika
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 27, 2015 at 12:30 PM, Sathwik B P <sathwik.bp@gmail.com>
> > > > wrote:
> > > >
> > > > > Sudharma,
> > > > >
> > > > > Where are you going to configure the master-slaves, is it in the
> web
> > > > > application level or at the load balancer?
> > > > >
> > > > > regards,
> > > > > sathwik
> > > > >
> > > > > On Tue, May 26, 2015 at 7:42 PM, sudharma subasinghe <
> > > > > suba.11@cse.mrt.ac.lk>
> > > > > wrote:
> > > > >
> > > > > > Hi Tammo,
> > > > > >
> > > > > > Can you suggest the best method from these to implement? As
> first I
> > > > > > suggested the master-slaves scenario I think it is easy to
> > implement
> > > > than
> > > > > > distributed lock scenario. However if you can suggest one from
> > these
> > > > two,
> > > > > > then I can think about it.
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > > On 21 May 2015 at 12:40, Sathwik B P <sathwik.bp@gmail.com>
> wrote:
> > > > > >
> > > > > > > With respect to the hotdeployment,
> > > > > > >
> > > > > > > We can drop the deployment archive onto the deployment
folder.
> > > Since
> > > > > the
> > > > > > > DeploymentPoller are acquiring the distributed lock for
the
> > > > > > DeploymentUnit,
> > > > > > > only one of the nodes will get the lock and initiate the
> > > deployment.
> > > > > > > DeploymentPollers on other nodes will fail in acquiring
the
> lock
> > > and
> > > > > > hence
> > > > > > > will silently ignore it.
> > > > > > >
> > > > > > > On Thu, May 21, 2015 at 12:30 PM, Sathwik B P <
> > > sathwik.bp@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Tammo,
> > > > > > > >
> > > > > > > > The distributed lock acquisition on the DeploymentUnit
should
> > be
> > > > > added
> > > > > > to
> > > > > > > > both DeploymentWebService and DeploymentPoller.
> > > > > > > >
> > > > > > > > When a deployment operation is initiated through the
> > > > > > > DeploymentWebService,
> > > > > > > > The load balancer routes it to any of the available
nodes.
> > > > > > > >
> > > > > > > > On the routed node, the DeploymentWebService acquires
the
> > > > Distributed
> > > > > > > > lock. On the remaining nodes the DeploymentPoller
will try to
> > > > acquire
> > > > > > the
> > > > > > > > distributed lock and will not get it and hence will
silently
> > > ignore
> > > > > it.
> > > > > > > >
> > > > > > > > Once the routed node completes the deployment, it
will
> release
> > > the
> > > > > > lock.
> > > > > > > > This way we don't have to stall the DeploymentPoller
in other
> > > > nodes.
> > > > > > > >
> > > > > > > > Does it answer the concerns?
> > > > > > > >
> > > > > > > >
> > > > > > > > Now, if we give the responsibility of identifying
the master
> > node
> > > > to
> > > > > > the
> > > > > > > > hazelcast, how do we plan to intimate the load balancer
to
> > change
> > > > > it's
> > > > > > > > configuration about the master node?
> > > > > > > > Assuming there are 3 nodes in the cluster,
> > > > > > > > node1 -master
> > > > > > > > node2 - slave
> > > > > > > > node3 - slave
> > > > > > > >
> > > > > > > > Node1 goes down, the LB will promote Node2 as master
node,
> but
> > > > > > hazelcast
> > > > > > > > might promote Node3 as master node. They are out of
sync.
> > > > > > > >
> > > > > > > > Is this argument valid?
> > > > > > > >
> > > > > > > > regards,
> > > > > > > > sathwik
> > > > > > > >
> > > > > > > > On Wed, May 20, 2015 at 1:51 PM, Tammo van Lessen
<
> > > > > > tvanlessen@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Sudharma,
> > > > > > > >>
> > > > > > > >> what do you expect from the "other nodes deployment"?
> > > Compilation
> > > > is
> > > > > > not
> > > > > > > >> needed since the CBP file is written to the (shared)
FS.
> > > > > Registration
> > > > > > is
> > > > > > > >> also not needed, since it is done via the shared
database.
> So
> > > the
> > > > > only
> > > > > > > >> thing that might be needed is to tell the engine
that there
> > is a
> > > > new
> > > > > > > >> deployment. I'd need to check that. If this is
needed, I
> > revert
> > > my
> > > > > > last
> > > > > > > >> statement, then it is perhaps better to just send
an event
> > over
> > > > > > > Hazelcast
> > > > > > > >> to all nodes that the deployment has changed.
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >>   Tammo
> > > > > > > >>
> > > > > > > >> On Wed, May 20, 2015 at 10:13 AM, sudharma subasinghe
<
> > > > > > > >> suba.11@cse.mrt.ac.lk
> > > > > > > >> > wrote:
> > > > > > > >>
> > > > > > > >> > Hi Tammo,
> > > > > > > >> >
> > > > > > > >> > The master node writes meta data. But runtime
information
> > must
> > > > be
> > > > > > > >> available
> > > > > > > >> > in all nodes.Since the folder is shared,
all nodes will
> see
> > > the
> > > > > > > >> > availability of a new process. My idea is
for master node
> to
> > > > write
> > > > > > the
> > > > > > > >> meta
> > > > > > > >> > data and other nodes to just read the meta
data and load
> > > > > process.So
> > > > > > we
> > > > > > > >> need
> > > > > > > >> > a small delay between master node deployment
and other
> nodes
> > > > > > > deployment.
> > > > > > > >> >
> > > > > > > >> > Is there anyway to set the delay between
master node and
> > > slaves
> > > > > > until
> > > > > > > >> > master node finish the deployment?
> > > > > > > >> >
> > > > > > > >> > Thank you
> > > > > > > >> > Sudharma
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 20 May 2015 at 13:01, Tammo van Lessen
<
> > > tvanlessen@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Sathwik,
> > > > > > > >> > >
> > > > > > > >> > > On Wed, May 20, 2015 at 6:40 AM, Sathwik
B P <
> > > > > > sathwik.bp@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Sudharma/Tammo,
> > > > > > > >> > > >
> > > > > > > >> > > > 1) How do we plan to decide which
is the master node
> in
> > > the
> > > > > > > cluster?
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > > I think the easiest approach is to always
elect the
> oldest
> > > > node
> > > > > in
> > > > > > > the
> > > > > > > >> > > cluster to be the master. AFAIK Hazelcast
can easily
> asked
> > > for
> > > > > > this
> > > > > > > >> > > information.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > > 2) Don't we need to stall the Deployment
Pollers in
> the
> > > > slave
> > > > > > > nodes?
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > Absolutely.
> > > > > > > >> > >
> > > > > > > >> > > Suggestion:
> > > > > > > >> > > > I am not sure whether do we need
Master-SLaves. Why
> not
> > > give
> > > > > > every
> > > > > > > >> node
> > > > > > > >> > > in
> > > > > > > >> > > > the cluster the same status (Active-Active).
> > > > > > > >> > > >
> > > > > > > >> > > > When a new deployment is made,
the load balancer can
> > push
> > > it
> > > > > to
> > > > > > > any
> > > > > > > >> of
> > > > > > > >> > > the
> > > > > > > >> > > > available nodes. That node will
probably acquire a
> > > > distributed
> > > > > > > lock
> > > > > > > >> on
> > > > > > > >> > > the
> > > > > > > >> > > > deployment unit and acts as master
for that
> deployment.
> > > This
> > > > > > > ensures
> > > > > > > >> > > > optimum usage of the cluster nodes.
Probably no static
> > > > > > > >> configuration of
> > > > > > > >> > > > Master-Slave in the load balancer
nor in the
> hazelcast.
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > > But this would not allow to have the
hotdeployment via
> > > > > filesystem
> > > > > > > >> still
> > > > > > > >> > > enabled, right?
> > > > > > > >> > >
> > > > > > > >> > > Best,
> > > > > > > >> > >   Tammo
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > --
> > > > > > > >> > > Tammo van Lessen - http://www.taval.de
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Tammo van Lessen - http://www.taval.de
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message