cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Date Mon, 17 Aug 2020 06:16:51 GMT
2020-07-25 03:23:51,093 WARN  [c.c.v.SystemVmLoadScanner]
(secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception Failed to
fetch any free public IP address
com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any free
public IP address

?

On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedammad83@gmail.com> wrote:

> I have checked further, the key is successfully injected in systemvm.iso on
> the management node and copied successfully to xenserver. I have checked on
> systemVM there is no public key found in authorized_keys of root.
>
> How can I troubleshoot this further ? how can I enable trace logging for
> management server to see if there is something problematic happening.
>
> Ammad Ali
>
> On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedammad83@gmail.com> wrote:
>
> > Hi,
> >
> > I have setup my test environment, here is what I did:
> >
> > - Installed ACS 4.11.1 and added xenserver 7.0 host in it. SystemVMs are
> > up in the zone and agents are up.
> > - Then upgraded the system to 4.11.3. Recreated systemVMs, agent is up
> and
> > systemVM are up with 4.11.3 systemVM version.
> > - Then upgraded the system to 4.13.1, recreated systemVMs are running but
> > agents are not up.
> >
> > I have checked md5sum of systemvm.iso on xenserver and management server,
> > both are same.
> >
> > [root@xenserver iso]# md5sum /opt/xensource/packages/iso/systemvm.iso
> > baba18f156395da3a5d8208727d8f421
> /opt/xensource/packages/iso/systemvm.iso
> >
> > [root@cloudstack-upgrade vms]# md5sum
> > /usr/share/cloudstack-common/vms/systemvm.iso
> > baba18f156395da3a5d8208727d8f421
> >  /usr/share/cloudstack-common/vms/systemvm.iso
> >
> > Also the private key on xenserver root and management server are same.
> >
> > management server
> > /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud
> >
> > xenserver path
> > /root/.ssh/id_rsa.cloud
> >
> > - Ammad Ali
> >
> > On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <syedammad83@gmail.com>
> wrote:
> >
> >> Here is the link for download management logs.
> >>
> >>
> >>
> https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing
> >>
> >> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <syedammad83@gmail.com>
> >> wrote:
> >>
> >>> I have reverted the version back to 4.11.3. But I have saved logs
> >>> starting from upgrade.
> >>>
> >>> I think the key has been copied successfully in system vm iso.
> >>>
> >>> 2020-07-25 00:34:17,214 INFO  [c.c.s.ConfigurationServerImpl]
> >>> (main:null) (logid:) Going to update systemvm iso with generated
> keypairs
> >>> if needed
> >>> 2020-07-25 00:34:17,214 INFO  [c.c.s.ConfigurationServerImpl]
> >>> (main:null) (logid:) Trying to inject public and private keys into
> systemvm
> >>> iso
> >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) (logid:)
> >>> Looking for vms/systemvm.iso in the classpath
> >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) (logid:)
> >>> System resource: file:/usr/share/cloudstack-common/vms/systemvm.iso
> >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) (logid:)
> >>> Absolute path =  /usr/share/cloudstack-common/vms/systemvm.iso
> >>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl]
> >>> (main:null) (logid:) Executing: /bin/bash
> >>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> >>> /var/cloudstack/management/.ssh/id_rsa.pub
> >>> /var/cloudstack/management/.ssh/id_rsa
> >>> /usr/share/cloudstack-common/vms/systemvm.iso
> >>> 2020-07-25 00:34:17,636 INFO  [c.c.s.ConfigurationServerImpl]
> >>> (main:null) (logid:) Injected public and private keys into systemvm iso
> >>> with result : null
> >>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying
> >>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> >>> to /opt/xensource/packages/iso on 172.16.2.22 with permission 0644
> >>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying
> >>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission 0644
> >>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying
> >>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> >>> to /opt/xensource/packages/iso on 172.16.2.9 with permission 0644
> >>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying
> >>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission 0644
> >>>
> >>> I have attached complete management logs. The start of logs is the
> start
> >>> of management server after upgrade of management server from 4.11.3 to
> >>> 4.13.1.
> >>>
> >>> Ammad Ali
> >>>
> >>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic <andrija.panic@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Do you get an error while trying to inject ssh key into the
> systemvm.iso
> >>>> (mgmt logs) , or can you confirm that the systemvm.iso on XS host and
> >>>> the
> >>>> one on the mgmt server are identical, md5sum them (i.e. has the iso
> been
> >>>> copied over to the XS successfully) - this might explain not being
> able
> >>>> to
> >>>> login via ssh with your private key.
> >>>>
> >>>> Best,
> >>>>
> >>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <syedammad83@gmail.com>
wrote:
> >>>>
> >>>> > Yes this is exactly the same issue that i faced.
> >>>> >
> >>>> > Sent from my iPhone
> >>>> >
> >>>> > > On 12-Aug-2020, at 8:35 AM, Eric Lee Green <
> >>>> eric.lee.green@gmail.com>
> >>>> > wrote:
> >>>> > >
> >>>> > > Correct, 4.11.3 template is used for 4.11.3, 4.12, and
4.13. 4.14
> >>>> moves
> >>>> > to the 4.14.0 template.
> >>>> > >
> >>>> > > There seems to be something odd happening key-wise sometimes
with
> >>>> > upgrades from 4.11.3 to 4.13.1 or 4.14.0.   I managed an upgrade
> from
> >>>> > 4.11.3 to 4.13.1 that *almost* worked, but the secondary storage
VM
> >>>> > wouldn't work and thus I couldn't spawn new virtual machines. Same
> >>>> symptom
> >>>> > -- key error when the agent tried to ssh into it. And deleting
it
> and
> >>>> > making it respawn didn't help. Then I tried 4.11.3 to 4.14.0 and
> >>>> *all* the
> >>>> > VM's failed at that point (of course, that was with the new
> template).
> >>>> > >
> >>>> > > Right now I'm back at 4.11.3 until this can be figured out.
> >>>> > >
> >>>> > >> On 8/11/2020 5:53 AM, Ammad Syed wrote:
> >>>> > >> Hi,
> >>>> > >>
> >>>> > >> I think 4.12 and 4.13 uses same systemVM template i.e
4.11.3
> >>>> version,
> >>>> > which
> >>>> > >> I already have registered. Currently I am running 4.11.3
version
> >>>> of ACS.
> >>>> > >>
> >>>> > >> MariaDB [cloud]> SELECT id,name,type,cross_zones,state
FROM
> >>>> > >> cloud.vm_template WHERE name like '%systemvm-xenserver%'
AND
> >>>> removed IS
> >>>> > >> NULL;
> >>>> > >>
> >>>>
> +------+-----------------------------+--------+-------------+----------+
> >>>> > >> | id   | name                        | type   | cross_zones
|
> >>>> state    |
> >>>> > >>
> >>>>
> +------+-----------------------------+--------+-------------+----------+
> >>>> > >> |  337 | systemvm-xenserver-3.0.0    | SYSTEM |      
    0 |
> >>>> Inactive |
> >>>> > >> |  418 | systemvm-xenserver-4.2      | SYSTEM |      
    0 |
> >>>> Active   |
> >>>> > >> |  472 | systemvm-xenserver-4.3      | USER   |      
    1 |
> >>>> Inactive |
> >>>> > >> |  473 | systemvm-xenserver-4.3      | USER   |      
    1 |
> >>>> Inactive |
> >>>> > >> |  474 | systemvm-xenserver-4.3      | USER   |      
    1 |
> >>>> Inactive |
> >>>> > >> |  475 | systemvm-xenserver-4.3      | USER   |      
    1 |
> >>>> Inactive |
> >>>> > >> |  476 | systemvm-xenserver-4.3      | USER   |      
    0 |
> >>>> Inactive |
> >>>> > >> |  479 | systemvm-xenserver-4.3-2    | USER   |      
    1 |
> >>>> Inactive |
> >>>> > >> |  480 | systemvm-xenserver-4.3      | SYSTEM |      
    0 |
> >>>> Active   |
> >>>> > >> |  549 | systemvm-xenserver-4.5.1    | USER   |      
    0 |
> >>>> Active   |
> >>>> > >> |  550 | systemvm-xenserver-4.5.1    | SYSTEM |      
    0 |
> >>>> Active   |
> >>>> > >> |  651 | systemvm-xenserver-4.7.0    | USER   |      
    0 |
> >>>> Inactive |
> >>>> > >> |  652 | systemvm-xenserver-4.7.0    | USER   |      
    0 |
> >>>> Inactive |
> >>>> > >> |  653 | systemvm-xenserver-4.7.0    | SYSTEM |      
    0 |
> >>>> Inactive |
> >>>> > >> |  737 | systemvm-xenserver-4.9.2    | SYSTEM |      
    1 |
> >>>> Inactive |
> >>>> > >> |  739 | systemvm-xenserver-4.9.2-sb | SYSTEM |      
    1 |
> >>>> Active   |
> >>>> > >> | 1245 | systemvm-xenserver-4.11.1   | SYSTEM |      
    1 |
> >>>> Active   |
> >>>> > >> | 1584 | systemvm-xenserver-4.11.2   | SYSTEM |      
    1 |
> >>>> Active   |
> >>>> > >> | 1677 | systemvm-xenserver-4.11.3   | SYSTEM |      
    1 |
> >>>> Active   |
> >>>> > >>
> >>>>
> +------+-----------------------------+--------+-------------+----------+
> >>>> > >>
> >>>> > >> - Ammad
> >>>> > >>
> >>>> > >> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion <
> >>>> pdion891@apache.org>
> >>>> > wrote:
> >>>> > >> db.
> >>>> > >>> Hi Syed,
> >>>> > >>> From 4.12, the systemvm template had to be upgraded
because of
> OS
> >>>> > change in
> >>>> > >>> the template, moved to a latest version of Debian.
Because of
> >>>> that,
> >>>> > some VR
> >>>> > >>> scripts have changed and make obsolete older version
of VRs, so
> >>>> you
> >>>> > will
> >>>> > >>> most likely have to register an updated systemvm templates
and
> >>>> upgrade
> >>>> > your
> >>>> > >>> system VMs and VRs.
> >>>> > >>>
> >>>> > >>> Regards,
> >>>> > >>>
> >>>> > >>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed <
> >>>> syedammad83@gmail.com>
> >>>> > wrote:
> >>>> > >>>
> >>>> > >>>> Hi Guys,
> >>>> > >>>>
> >>>> > >>>> I was previously on 4.9.3 cloudstack and upgraded
to 4.11.1
> then
> >>>> > 4.11.3.
> >>>> > >>>> The version 4.11.3 is working fine since six months.
> >>>> > >>>>
> >>>> > >>>> Now I have tried to upgrade my system from 4.11.3
to 4.13.1.
> The
> >>>> > upgrade
> >>>> > >>>> goes successful. I didn't uploaded any system
VM template.
> >>>> However the
> >>>> > >>>> problem occured when I recreated my systemVM of
POD, the VM
> >>>> recreated
> >>>> > and
> >>>> > >>>> its state was running but agent state was not
getting up, its
> >>>> showing
> >>>> > >>> blank
> >>>> > >>>> in column.
> >>>> > >>>>
> >>>> > >>>> Digging further via job logs, the job is failed
with error that
> >>>> > unable to
> >>>> > >>>> execute command via ssh. Below are the logs.
> >>>> > >>>>
> >>>> > >>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper]
> >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece)
SSH execution
> of
> >>>> > command
> >>>> > >>>> /opt/cloud/bin/router_proxy.sh keystore-s
> >>>> > >>>> etup 169.254.2.199
> >>>> /usr/local/cloud/systemvm/conf/agent.properties
> >>>> > >>>> /usr/local/cloud/systemvm/conf/cloud.jks TJaQYChYBwKh7Cx9
365
> >>>> > >>>> /usr/local/cloud/systemvm/conf/clou
> >>>> > >>>> d.csr has an error status code in return. Result
output:
> >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.m.DirectAgentAttache]
> >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece)
Seq
> >>>> > >>>> 906-3195585410596077730: Response Received:
> >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece)
Seq
> >>>> > >>>> 906-3195585410596077730: Processing:  { Ans: ,
MgmtId:
> 779271079
> >>>> > >>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1,
Flags: 10,
> >>>> > >>>> [{"org.apache.cloudstack.ca
> >>>> > >>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}]
> >>>> > >>>> }
> >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258
> >>>> > ctx-df740f75)
> >>>> > >>>> (logid:9fa7dece) Seq 906-319558541059607773
> >>>> > >>>> 0: Received:  { Ans: , MgmtId: 77927107943497,
via:
> >>>> > >>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, {
> >>>> SetupKeystoreAnswer } }
> >>>> > >>>> 2020-07-25 02:30:48,127 ERROR [c.c.v.VirtualMachineManagerImpl]
> >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258
> >>>> > ctx-df740f75)
> >>>> > >>>> (logid:9fa7dece) Failed to setup keystore and
generate CSR for
> >>>> system
> >>>> > vm:
> >>>> > >>>> s-24142-VM
> >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258
> >>>> > ctx-df740f75)
> >>>> > >>>> (logid:9fa7dece) Done executing VM work job:
> >>>> > >>>>
> >>>> > >>>>
> >>>> > >>>
> >>>> >
> >>>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"}
> >>>> > >>>> 2020-07-25 02:30:48,128 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258
> >>>> > ctx-df740f75)
> >>>> > >>>> (logid:9fa7dece) Complete async job-1208258, jobStatus:
> >>>> SUCCEEDED,
> >>>> > >>>> resultCode: 0, result: null
> >>>> > >>>>
> >>>> > >>>> I tried to dig it further, I was unable to login
systemVM via
> >>>> ssh from
> >>>> > >>>> xenserver host with key /root/.ssh/id_rsa.cloud
placed. Look
> like
> >>>> > private
> >>>> > >>>> key issue. However I am able to login on my old
systemVMs ( i.e
> >>>> > created
> >>>> > >>> on
> >>>> > >>>> ACS 4.11.3)
> >>>> > >>>>
> >>>> > >>>> Also I have SSL certificate enabled for console
proxy on my ACS
> >>>> 4.11.3
> >>>> > >>> and
> >>>> > >>>> I am using only xenserver 7.0 hosts.
> >>>> > >>>>
> >>>> > >>>> I tried to disable SSL on secstorage and console
proxy from
> >>>> global
> >>>> > >>>> settings, but still didn't worked.
> >>>> > >>>>
> >>>> > >>>> I had a fresh installation of ACS 4.13.1 with
xenserver 7.0,
> >>>> systemVMs
> >>>> > >>> are
> >>>> > >>>> working fine in it.
> >>>> > >>>>
> >>>> > >>>> Please advise.
> >>>> > >>>> --
> >>>> > >>>> Regards,
> >>>> > >>>>
> >>>> > >>>>
> >>>> > >>>> Syed Ammad Ali
> >>>> > >>>>
> >>>> > >>
> >>>> >
> >>>>
> >>>
> >>>
> >>> --
> >>> Regards,
> >>>
> >>>
> >>> Syed Ammad Ali
> >>>
> >>
> >>
> >> --
> >> Regards,
> >>
> >>
> >> Syed Ammad Ali
> >>
> >
> >
> > --
> > Regards,
> >
> >
> > Syed Ammad Ali
> >
>
>
> --
> Regards,
>
>
> Syed Ammad Ali
>


-- 

Andrija Panić

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message