cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Kimlaychuk <Vadim.Kimlayc...@Elion.ee>
Subject RE: Status: update from 4.4.0 -> 4.4.1 still broken ?
Date Thu, 30 Oct 2014 07:08:14 GMT
I have tried to do upgrade 2nd time and failed again. The sequence of actions is described
here: http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgrade-4.4.html

Followed instructions until the end. VM upgrade has taken more than 5 hrs.  The result of
update is:
-----------------
Stopping and starting 1 secondary storage vm(s)...
ERROR: Failed to stop secondary storage vm with id 25

Done stopping and starting secondary storage vm(s)

Stopping and starting 1 console proxy vm(s)...
ERROR: Failed to stop console proxy vm with id 1

Done stopping and starting console proxy vm(s) .

Stopping and starting 1 running routing vm(s)...
2
Done restarting router(s).
-------------------

After restarting managment server workers are not querying status and resources anymore. The
last lines at management server log are:

-------------------

2014-10-30 08:48:10,207 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin UcsManagerImpl
2014-10-30 08:48:10,208 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin VpcVirtualRouterElement
2014-10-30 08:48:10,208 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin BaremetalPxeManagerImpl
2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin BaremetalManagerImpl
2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ApiDiscoveryServiceImpl
2014-10-30 08:48:10,209 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ManagementServerImpl
2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin DedicatedResourceManagerImpl
2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin InternalLoadBalancerElement
2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin LdapManagerImpl
2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ApiRateLimitServiceImpl
2014-10-30 08:48:10,211 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin BaremetalDhcpManagerImpl
2014-10-30 08:48:10,212 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ContrailVpcElementImpl
2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin ContrailElementImpl
2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin MidoNetElement
2014-10-30 08:48:10,215 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin NetscalerElement
2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin NiciraNvpElement
2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin OpenDaylightControllerResourceManagerImpl
2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin OvsElement
2014-10-30 08:48:10,216 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin PaloAltoExternalFirewallElement
2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin VirtualRouterElement
2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin SspElement
2014-10-30 08:48:10,217 DEBUG [c.c.a.ApiServer] (main:null) Discovered plugin BigSwitchVnsElement
2014-10-30 08:48:10,250 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (NetworkStatsUpdater-1:ctx-ca0c3513)
Successfully updated aggregate network stats
2014-10-30 08:48:10,379 INFO  [c.c.a.ApiServer] (Thread-11:null) ApiServer listening on port
8096
------------------------------------------

Full log is here: http://pastebin.com/Wpe7wsAV

Log is not growing and changing. Cloudstack console is missing with 404 error.  I thing VM
update should not influence this. Any System VM in fact can be just re-created by "killing"
from management GUI. 

Vadim.


-----Original Message-----
From: Andrija Panic [mailto:andrija.panic@gmail.com] 
Sent: Wednesday, October 29, 2014 10:21 AM
To: users@cloudstack.apache.org
Subject: Re: Status: update from 4.4.0 -> 4.4.1 still broken ?

If it makes it any easier - it failed for me, but with no error simply got "transaction revoked..."
error, no description, nothing...

On 28 October 2014 23:19, motty cruz <motty.cruz@gmail.com> wrote:

> Thanks Andrija,
> but that did not fixed for me:
> mysql> describe snapshot_policy;
> +-----------+---------------------+------+-----+---------+----------------+
> | Field     | Type                | Null | Key | Default | Extra          |
> +-----------+---------------------+------+-----+---------+----------------+
> | id        | bigint(20) unsigned | NO   | PRI | NULL    | auto_increment |
> | uuid      | varchar(40)         | YES  | UNI | NULL    |                |
> | volume_id | bigint(20) unsigned | NO   | MUL | NULL    |                |
> | schedule  | varchar(100)        | NO   |     | NULL    |                |
> | timezone  | varchar(100)        | NO   |     | NULL    |                |
> | interval  | int(4)              | NO   |     | 4       |                |
> | max_snaps | int(8)              | NO   |     | 0       |                |
> | active    | tinyint(1) unsigned | NO   |     | NULL    |                |
> | display   | tinyint(1)          | NO   |     | 1       |                |
> +-----------+---------------------+------+-----+---------+----------------+
> 9 rows in set (0.00 sec)
>
> I changed it as suggest:
> ALTER TABLE  `snapshot_policy` ADD  `display` TINYINT( 1 ) NOT NULL 
> DEFAULT  '1'
>
> am I doing something wrong? this is my 3rd attempt to upgrade.failed 
> with the same result.
>
> - Please help!
>
>
>
>
> On Tue, Oct 28, 2014 at 2:56 PM, Andrija Panic 
> <andrija.panic@gmail.com>
> wrote:
>
> > Motty, here is copy/paste from someone's email here on mailing list:
> > Maybe check if this is your issue...
> > I did not manage to upgrade, and this solution came to late for 
> > me... did clean install, since this was test install in my case...
> >
> > <thomas.schneider@euskill.com> wrote:
> > > Hi,
> > >
> > > I solved the problem.
> > >
> > > The column display from the snapshot _policy was missing after update.
> > >
> > > Just do: ALTER TABLE  `snapshot_policy` ADD  `display` TINYINT( 1 
> > > ) NOT NULL DEFAULT  '1'
> > >
> > > Regards,
> >
> > On 28 October 2014 22:13, motty cruz <motty.cruz@gmail.com> wrote:
> >
> > > forgot to mentioned about this logs in /etc/cloudstack/sysvm.log
> > > nohup: ignoring input
> > >
> > > Stopping and starting 1 secondary storage vm(s)...
> > > curl: (7) couldn't connect to host
> > > ERROR: Failed to stop secondary storage vm with id 15
> > >
> > > Done stopping and starting secondary storage vm(s)
> > >
> > > Stopping and starting 1 console proxy vm(s)...
> > > curl: (7) couldn't connect to host
> > > ERROR: Failed to stop console proxy vm with id 16
> > >
> > > Done stopping and starting console proxy vm(s) .
> > >
> > > Stopping and starting 1 running routing vm(s)...
> > > curl: (7) couldn't connect to host
> > > 2
> > > Done restarting router(s).
> > >
> > >
> > > Also /etc/cloudstack/cloud.log:
> > > [2014.10.28-13.48.43] Stopping and starting 1 secondary storage
> vm(s)...
> > > [2014.10.28-13.48.43] INFO: Stopping secondary storage vm with id 
> > > 15 [2014.10.28-13.48.43] ERROR: Failed to stop secondary storage 
> > > vm with
> id
> > 15
> > > [2014.10.28-13.48.43] Done stopping and starting secondary storage
> vm(s).
> > > [2014.10.28-13.48.43] Stopping and starting 1 console proxy vm(s)...
> > > [2014.10.28-13.48.43] INFO: Stopping console proxy with id 16 
> > > [2014.10.28-13.48.43] ERROR: Failed to stop console proxy vm with 
> > > id 16 [2014.10.28-13.48.43] Done stopping and starting console proxy vm(s)
.
> > > [2014.10.28-13.48.43] Stopping and starting 1 running routing vm(s)...
> > > [2014.10.28-13.48.43] INFO: Restarting router with id 17 
> > > [2014.10.28-13.48.43] ERROR: Failed to restart domainRouter with 
> > > id 17; unable to submit the job [2014.10.28-13.48.53] Done 
> > > restarting router(s).
> > >
> > > Thanks
> > > -Motty
> > >
> > > On Tue, Oct 28, 2014 at 2:08 PM, motty cruz <motty.cruz@gmail.com>
> > wrote:
> > >
> > > > Thanks Vadim,
> > > > I was able to restore back to ACS 4.4.0. upgraded again with the 
> > > > same
> > bad
> > > > result. has anybody successsfully upgrade from ACS 4.4.0 to
> ACS-4.4.1?
> > if
> > > > yes do you mind sharing notes?
> > > > I had used this URL:
> > > >
> > >
> >
> http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgr
> ade-4.4.html
> > > >
> > > > here is my logs:
> > > > upgradeACS4.4.1-http-status-404
> > > > *http://pastebin.com/raw.php?i=eeYQsVyD
> > > > <http://pastebin.com/raw.php?i=eeYQsVyD>*
> > > >
> > > > *I really apreciate if someone can help me! * *in /etc/sudoers*
> > > >
> > > > *cloud ALL=NOPASSWD : ALL*
> > > >
> > > > *my suspecion, the issue have to do with database but am not 
> > > > certain
> > and
> > > > won't know how to fix it. *
> > > >
> > > > *Thanks, *
> > > > *-Motty*
> > > >
> > > > On Tue, Oct 28, 2014 at 12:30 AM, Vadim Kimlaychuk < 
> > > > Vadim.Kimlaychuk@elion.ee> wrote:
> > > >
> > > >> Motty,
> > > >>
> > > >>         I got 404 after downgrading as well, but this was 
> > > >> database
> > > issue.
> > > >> You need to restore your DB first and then restart your 
> > > >> cloudstack management server.  Better way (from my point of 
> > > >> view) - having DB
> and
> > > >> Management servers as VMs and restore them from snapshots. It 
> > > >> is
> much
> > > >> faster :)
> > > >>
> > > >>         I will try to upgrade once again as Rohit Yadav suggested.
> > > >> Didn't think about old VM-s running at hypervisors. They were 
> > > >> not
> > > upgraded
> > > >> that is true.  Will keep in mind issue 7781 pointed by Nux!
> > > >>         Thank you all. Be back after update.
> > > >>
> > > >> Vadim.
> > > >>
> > > >> -----Original Message-----
> > > >> From: motty cruz [mailto:motty.cruz@gmail.com]
> > > >> Sent: Tuesday, October 28, 2014 12:39 AM
> > > >> To: users@cloudstack.apache.org
> > > >> Subject: Re: Status: update from 4.4.0 -> 4.4.1 still broken ?
> > > >>
> > > >> Nux,
> > > >> I just finished downgrading back to 4.4.0. but the error I get is:
> > > >> HTTP STATUS 404
> > > >> type status report
> > > >>
> > > >> I will try again to upgrade as soon as is fully restore back to
> > 4.4.0. I
> > > >> am running XenServer 6.2 hypervison with Centos 6.5 as manager. 
> > > >> has
> > > anybody
> > > >> run into this issue?
> > > >>
> > > >> Vadim, I don't meant to hijack your post.
> > > >>
> > > >> Thanks,
> > > >> Motty
> > > >>
> > > >> On Mon, Oct 27, 2014 at 3:09 PM, Nux! <nux@li.nux.ro> wrote:
> > > >>
> > > >> > Motty,
> > > >> >
> > > >> > Please provide DEBUG logs from management server and hypervisors.
> > > >> > It's possible you may be hitting this (as I remember similar
> > symptoms)
> > > >> > https://issues.apache.org/jira/browse/CLOUDSTACK-7781
> > > >> >
> > > >> > Lucian
> > > >> >
> > > >> > --
> > > >> > Sent from the Delta quadrant using Borg technology!
> > > >> >
> > > >> > Nux!
> > > >> > www.nux.ro
> > > >> >
> > > >> > ----- Original Message -----
> > > >> > > From: "motty cruz" <motty.cruz@gmail.com>
> > > >> > > To: users@cloudstack.apache.org
> > > >> > > Sent: Monday, 27 October, 2014 20:39:54
> > > >> > > Subject: Re: Status: update from 4.4.0 -> 4.4.1 still
broken ?
> > > >> >
> > > >> > > Hello Vadim,
> > > >> > > did you find the solution?
> > > >> > >
> > > >> > > I am having similar issue. I am running CentOS 6.5
> > > >> > > "nohup: ignoring input
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Stopping and starting 1 secondary storage vm(s)...
> > > >> > >
> > > >> > > curl: (56) Failure when receiving data from the peer
> > > >> > >
> > > >> > > ERROR: Failed to stop secondary storage vm with id 14
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Done stopping and starting secondary storage vm(s)
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Stopping and starting 1 console proxy vm(s)...
> > > >> > >
> > > >> > > curl: (7) couldn't connect to host
> > > >> > >
> > > >> > > ERROR: Failed to stop console proxy vm with id 13
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Done stopping and starting console proxy vm(s) .
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Stopping and starting 1 running routing vm(s)...
> > > >> > >
> > > >> > > curl: (7) couldn't connect to host
> > > >> > >
> > > >> > > 2
> > > >> > >
> > > >> > > Done restarting router(s)."
> > > >> > >
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > Motty
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Mon, Oct 27, 2014 at 8:52 AM, Rohit Yadav 
> > > >> > > <rohit.yadav@shapeblue.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> Hi Vadim,
> > > >> > >>
> > > >> > >> See my reply in-line;
> > > >> > >>
> > > >> > >> > On 27-Oct-2014, at 8:46 pm, Vadim Kimlaychuk
> > > >> > <Vadim.Kimlaychuk@Elion.ee>
> > > >> > >> wrote:
> > > >> > >> >
> > > >> > >> > Yes, I did follow this guide.  APT does not have

> > > >> > >> > pre-compiled packages
> > > >> > >> so I have compiled them by myself. This is the only
difference.
> > > >> > >> >
> > > >> > >> > cloudstack-sysvmadm  script that is mentioned in
the 
> > > >> > >> > last
> step
> > > >> > >> > has the
> > > >> > >> first line as follows:
> > > >> > >> >
> > > >> > >> > . /etc/rc.d/init.d/functions
> > > >> > >> >
> > > >> > >> > But there is no /etc/rc.d/  folder at my Ubuntu
server.
> Script
> > > >> > >> > failed
> > > >> > at
> > > >> > >> the very beginning so I haven't waited until VMs are
going 
> > > >> > >> to
> > > >> migrate.
> > > >> > >>
> > > >> > >> You’re correct. This is supposed to be CentOS/RHEL/Fedora
> > specific.
> > > >> > >> I think your upgrade went fine it’s just this script
that 
> > > >> > >> may
> > need
> > > >> > >> to be fixed. Can you create a ticket on CloudStack’s
JIRA? 
> > > >> > >> As a workaround you may try to remove this line in the

> > > >> > >> script, make
> > sure
> > > >> > >> “curl" is
> > > >> > installed on
> > > >> > >> Ubuntu and try again.
> > > >> > >>
> > > >> > >> > I have downgraded back to 4.4.0 until DEB packages
will 
> > > >> > >> > be
> > > >> released.
> > > >> > >> Problem may be there though I have no errors building
CS 
> > > >> > >> from
> the
> > > >> > source.
> > > >> > >>
> > > >> > >> I think you should be able to upgrade; until sysvmadm

> > > >> > >> script is fixed so that it works on Debian based distro,

> > > >> > >> after the
> upgrade
> > > >> > >> you may reboot
> > > >> > or
> > > >> > >> kill the existing/old 4.4.0 based systemvms using 
> > > >> > >> something
> like
> > a
> > > >> > custom
> > > >> > >> bash/python script or a tool like CloudMonkey or manually

> > > >> > >> if
> the
> > > >> > >> deploy size is small.
> > > >> > >>
> > > >> > >> Regards.
> > > >> > >>
> > > >> > >> >
> > > >> > >> > Vadim.
> > > >> > >> >
> > > >> > >> > -----Original Message-----
> > > >> > >> > From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
> > > >> > >> > Sent: Monday, October 27, 2014 5:00 PM
> > > >> > >> > To: users@cloudstack.apache.org
> > > >> > >> > Subject: Re: Status: update from 4.4.0 -> 4.4.1
still 
> > > >> > >> > broken
> ?
> > > >> > >> >
> > > >> > >> > Hi Vadim,
> > > >> > >> >
> > > >> > >> > Did you follow the upgrade instructions (especially
the
> > systemvm
> > > >> > >> upgrading):
> > > >> > >> >
> > > >> > >>
> > > >> >
> > > http://cloudstack-release-notes.readthedocs.org/en/4.4.1/upgrade/u
> > > pgra
> > > >> > de-4.4.html
> > > >> > >> >
> > > >> > >> > If consoles are not coming up, please upgrade systemvms

> > > >> > >> > to
> > 4.4.1
> > > >> > >> > ones
> > > >> > >> and delete existing console proxy VM(s) and retry.
> > > >> > >> >
> > > >> > >> > Any error log would help as well.
> > > >> > >>
> > > >> > >> Regards,
> > > >> > >> Rohit Yadav
> > > >> > >> Software Architect, ShapeBlue M. +91 88 262 30892 |

> > > >> > >> rohit.yadav@shapeblue.com
> > > >> > >> Blog: bhaisaab.org | Twitter: @_bhaisaab
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> Find out more about ShapeBlue and our range of CloudStack
> related
> > > >> > services
> > > >> > >>
> > > >> > >> IaaS Cloud Design & Build< 
> > > >> > >> http://shapeblue.com/iaas-cloud-design-and-build//>
> > > >> > >> CSForge – rapid IaaS deployment 
> > > >> > >> framework<http://shapeblue.com/csforge/
> > > >> > >
> > > >> > >> CloudStack Consulting<
> > http://shapeblue.com/cloudstack-consultancy/
> > > >
> > > >> > >> CloudStack Infrastructure Support< 
> > > >> > >> http://shapeblue.com/cloudstack-infrastructure-support/>
> > > >> > >> CloudStack Bootcamp Training Courses< 
> > > >> > >> http://shapeblue.com/cloudstack-training/>
> > > >> > >>
> > > >> > >> This email and any attachments to it may be confidential

> > > >> > >> and
> are
> > > >> > intended
> > > >> > >> solely for the use of the individual to whom it is addressed.
> Any
> > > >> > >> views
> > > >> > or
> > > >> > >> opinions expressed are solely those of the author and
do 
> > > >> > >> not necessarily represent those of Shape Blue Ltd or

> > > >> > >> related
> > companies.
> > > >> > >> If you are not
> > > >> > the
> > > >> > >> intended recipient of this email, you must neither take

> > > >> > >> any
> > action
> > > >> > >> based upon its contents, nor copy or show it to anyone.

> > > >> > >> Please contact the
> > > >> > sender
> > > >> > >> if you believe you have received this email in error.

> > > >> > >> Shape
> Blue
> > > >> > >> Ltd is
> > > >> > a
> > > >> > >> company incorporated in England & Wales. ShapeBlue

> > > >> > >> Services
> India
> > > >> > >> LLP
> > > >> > is a
> > > >> > >> company incorporated in India and is operated under

> > > >> > >> license
> from
> > > >> > >> Shape
> > > >> > Blue
> > > >> > >> Ltd. Shape Blue Brasil Consultoria Ltda is a company
> incorporated
> > > >> > >> in
> > > >> > Brasil
> > > >> > >> and is operated under license from Shape Blue Ltd. 
> > > >> > >> ShapeBlue SA
> > Pty
> > > >> > >> Ltd
> > > >> > is
> > > >> > >> a company registered by The Republic of South Africa
and 
> > > >> > >> is
> > traded
> > > >> > >> under license from Shape Blue Ltd. ShapeBlue is a 
> > > >> > >> registered
> > > >> trademark.
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Thanks for your support,
> > > >> > > Motty
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Thanks for your support,
> > > >> Motty
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks for your support,
> > > > Motty
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks for your support,
> > > Motty
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> > --------------------------------------
> >   http://admintweets.com
> > --------------------------------------
> >
>
>
>
> --
> Thanks for your support,
> Motty
>



-- 

Andrija Panić
--------------------------------------
  http://admintweets.com
--------------------------------------
Mime
View raw message