cloudstack-users mailing list archives

Site index · List index
Message view
Top
Subject Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Date Thu, 27 Aug 2020 08:29:51 GMT
```Guys, any help to troubleshoot this further would be highly appreciated.

>
> ﻿
> Hi,
>
> The systemVM is accessible with 169.x.x.x IP from xenserver host. But login with private
key is denied. I am testing this on my test environment.
>
> [root@xenserver ~]# ping 169.254.0.121
> PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data.
> 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms
> 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms
> 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms
> 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms
> ^C
> --- 169.254.0.121 ping statistics ---
> 4 packets transmitted, 4 received, 0% packet loss, time 2997ms
> rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms
> [root@xenserver ~]#
> [root@xenserver ~]#
> [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922
> Permission denied (publickey).
>
> I have checked by logging into the systemVM, the public key is not found in /root/.ssh/authorized_keys.

>
> I have digged further, the systemvm.iso is same on the management node and xenserver
host. I have checked by mounting the ISO as well, the public key is there in authorized_keys
in ISO.
>
> Restarting / recreating systemVM isn't fixed the issue.
>
> The systemVM is already running on HVM mode on xenserver host as the template OS type
is Debian 8 64bit is selected.
>
> On my PRD upgrade, I have tested on one zone only. Currently I am testing on my test
ACS environment which have only one zone.
>
>
>> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic <andrija.panic@gmail.com> wrote:
>> It was my tool (GUI based tool, that still doesn't return any line (???)
>> but I see it.
>>
>> Problem is with  169.254.1.178 not being reachable over SSH (2020-07-25
>> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in VR:
>> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178
>> /usr/local/cloud/systemvm/conf/agent.properties
>> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365
>> /usr/local/cloud/systemvm/conf/cloud.csr)
>>
>> Can you try to ssh from that particular XenServer (where the SSVM is
>> running) to the IP 169.xxxxx on port 3922 (ssh port) using the private key
>> - if you log in to the VM via it's console (root/password) - do you see
>> with ifconfig the interface being started with 169..xxxx IP address? I
>> assume that destroying the SSVM and starting a new one doesn't fix the
>> issue?
>>
>> You can also try changing the OS type of the systemVM template to Other
>> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need to see ID
>> for that OS type from the guest_os table, and set the guest_os_id field in
>> the vm_template table for your systemVM template.
>>
>> Does the issue occur in other zones as well? (assuming same hypervisor type)
>>
>>
>> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedammad83@gmail.com> wrote:
>>
>> > I have rechecked the logs are there. Please review below lines.
>> >
>> >
>> >
>> > (27 hits)
>> > Line 409969: 2020-07-25 01:45:52,320 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334
>> > job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to setup
>> > keystore and generate CSR for system vm: s-25023-VM
>> > Line 437413: 2020-07-25 01:51:33,845 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-70:ctx-457ce4ee
>> > job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to setup
>> > keystore and generate CSR for system vm: s-25024-VM
>> > Line 549538: 2020-07-25 02:06:44,833 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-4634f7e0
>> > job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to setup
>> > keystore and generate CSR for system vm: v-25027-VM
>> > Line 663111: 2020-07-25 02:23:43,783 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-fe0b857b
>> > job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to setup
>> > keystore and generate CSR for system vm: v-20771-VM
>> > Line 667060: 2020-07-25 02:24:30,221 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-662cfe5d
>> > job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to setup
>> > keystore and generate CSR for system vm: v-24763-VM
>> > Line 674269: 2020-07-25 02:25:47,345 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-b6529613
>> > job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to setup
>> > keystore and generate CSR for system vm: v-20773-VM
>> > Line 678169: 2020-07-25 02:26:20,355 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-7a062a66
>> > job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to setup
>> > keystore and generate CSR for system vm: v-25027-VM
>> > Line 697113: 2020-07-25 02:30:09,669 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-40:ctx-3be9fddc
>> > job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to setup
>> > keystore and generate CSR for system vm: v-20771-VM
>> > Line 701161: 2020-07-25 02:30:48,127 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d
>> > job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to setup
>> > keystore and generate CSR for system vm: s-24142-VM
>> > Line 701838: 2020-07-25 02:30:56,759 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-42:ctx-04ac8e71
>> > job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to setup
>> > keystore and generate CSR for system vm: v-24763-VM
>> > Line 703865: 2020-07-25 02:31:25,123 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-24bdbe8f
>> > job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to setup
>> > keystore and generate CSR for system vm: v-20773-VM
>> > Line 712078: 2020-07-25 02:32:28,666 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d5db5c2
>> > job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to setup
>> > keystore and generate CSR for system vm: s-25033-VM
>> > Line 715204: 2020-07-25 02:33:02,875 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-51:ctx-594da6d2
>> > job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to setup
>> > keystore and generate CSR for system vm: s-25035-VM
>> > Line 743140: 2020-07-25 02:38:52,257 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-a3f032da
>> > job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to setup
>> > keystore and generate CSR for system vm: s-25037-VM
>> > Line 817646: 2020-07-25 02:47:07,127 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-ca073bd1
>> > job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to setup
>> > keystore and generate CSR for system vm: s-25038-VM
>> > Line 847150: 2020-07-25 02:51:46,293 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-9db8c548
>> > job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to setup
>> > keystore and generate CSR for system vm: v-25040-VM
>> > Line 903949: 2020-07-25 02:57:48,515 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f35fd59a
>> > job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-25041-VM
>> > Line 957628: 2020-07-25 03:04:13,286 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-dc18956a
>> > job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-20771-VM
>> > Line 961606: 2020-07-25 03:05:00,522 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-83e48421
>> > job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-24763-VM
>> > Line 966633: 2020-07-25 03:05:29,767 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-30:ctx-897e2a66
>> > job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-20773-VM
>> > Line 968892: 2020-07-25 03:05:53,648 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-36:ctx-68e45b80
>> > job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-25041-VM
>> > Line 1003036: 2020-07-25 03:13:33,582 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-45:ctx-0f552e0e
>> > job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-20771-VM
>> > Line 1006496: 2020-07-25 03:14:09,787 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-08e5db24
>> > job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-24763-VM
>> > Line 1008408: 2020-07-25 03:14:39,584 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-8b9288a9
>> > job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-20773-VM
>> > Line 1023914: 2020-07-25 03:15:58,139 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d54d717
>> > job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-25046-VM
>> > Line 1089568: 2020-07-25 03:24:24,534 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-62ff8844
>> > job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to setup
>> > keystore and generate CSR for system vm: v-25048-VM
>> > Line 1097605: 2020-07-25 03:25:23,179 ERROR
>> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-62:ctx-24407a9e
>> > job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to setup
>> > keystore and generate CSR for system vm: s-25050-VM
>> >
>> >
>> >
>> >
>> > On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <andrija.panic@gmail.com>
>> > wrote:
>> >
>> > > Ammad, no such log lines in the file you uploaded....?
>> > >
>> > > On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedammad83@gmail.com>
wrote:
>> > >
>> > > > Hi Andrija,
>> > > >
>> > > > The error occurred only one time. Can you please try to grep "Failed
to
>> > > > setup keystore and generate CSR" you will see all the systemVMs that
I
>> > > > tried to create failed because of certificate generation failed. OR
you
>> > > can
>> > > > grep job-1208321/job-1208452 this job as well.
>> > > >
>> > > > Also I mentioned above that same thing is happening in my test
>> > > environment
>> > > > that I created 4.11.1 then upgraded 4.11.3 > 4.13.1.
>> > > >
>> > > > Ammad Ali
>> > > >
>> > > > On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic <
>> > andrija.panic@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > 2020-07-25 03:23:51,093 WARN  [c.c.v.SystemVmLoadScanner]
>> > > > > (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception
>> > > Failed
>> > > > to
>> > > > > fetch any free public IP address
>> > > > > com.cloud.utils.exception.CloudRuntimeException: Failed to fetch
any
>> > > free
>> > > > > public IP address
>> > > > >
>> > > > > ?
>> > > > >
>> > > > > On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedammad83@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > I have checked further, the key is successfully injected
in
>> > > > systemvm.iso
>> > > > > on
>> > > > > > the management node and copied successfully to xenserver.
I have
>> > > > checked
>> > > > > on
>> > > > > > systemVM there is no public key found in authorized_keys
of root.
>> > > > > >
>> > > > > > How can I troubleshoot this further ? how can I enable trace
>> > logging
>> > > > for
>> > > > > > management server to see if there is something problematic
>> > happening.
>> > > > > >
>> > > > > > Ammad Ali
>> > > > > >
>> > > > > > On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedammad83@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > I have setup my test environment, here is what I did:
>> > > > > > >
>> > > > > > > - Installed ACS 4.11.1 and added xenserver 7.0 host
in it.
>> > > SystemVMs
>> > > > > are
>> > > > > > > up in the zone and agents are up.
>> > > > > > > - Then upgraded the system to 4.11.3. Recreated systemVMs,
agent
>> > is
>> > > > up
>> > > > > > and
>> > > > > > > systemVM are up with 4.11.3 systemVM version.
>> > > > > > > - Then upgraded the system to 4.13.1, recreated systemVMs
are
>> > > running
>> > > > > but
>> > > > > > > agents are not up.
>> > > > > > >
>> > > > > > > I have checked md5sum of systemvm.iso on xenserver
and management
>> > > > > server,
>> > > > > > > both are same.
>> > > > > > >
>> > > > > > > [root@xenserver iso]# md5sum
>> > > > /opt/xensource/packages/iso/systemvm.iso
>> > > > > > > baba18f156395da3a5d8208727d8f421
>> > > > > > /opt/xensource/packages/iso/systemvm.iso
>> > > > > > >
>> > > > > > > [root@cloudstack-upgrade vms]# md5sum
>> > > > > > > /usr/share/cloudstack-common/vms/systemvm.iso
>> > > > > > > baba18f156395da3a5d8208727d8f421
>> > > > > > >  /usr/share/cloudstack-common/vms/systemvm.iso
>> > > > > > >
>> > > > > > > Also the private key on xenserver root and management
server are
>> > > > same.
>> > > > > > >
>> > > > > > > management server
>> > > > > > > /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud
>> > > > > > >
>> > > > > > > xenserver path
>> > > > > > > /root/.ssh/id_rsa.cloud
>> > > > > > >
>> > > > > > > - Ammad Ali
>> > > > > > >
>> > > > > > > On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <
>> > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > >>
>> > > > > > >>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > > > > > >>
>> > > > > > >> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >>> I have reverted the version back to 4.11.3.
But I have saved
>> > logs
>> > > > > > >>> starting from upgrade.
>> > > > > > >>>
>> > > > > > >>> I think the key has been copied successfully
in system vm iso.
>> > > > > > >>>
>> > > > > > >>> 2020-07-25 00:34:17,214 INFO  [c.c.s.ConfigurationServerImpl]
>> > > > > > >>> (main:null) (logid:) Going to update systemvm
iso with
>> > generated
>> > > > > > keypairs
>> > > > > > >>> if needed
>> > > > > > >>> 2020-07-25 00:34:17,214 INFO  [c.c.s.ConfigurationServerImpl]
>> > > > > > >>> (main:null) (logid:) Trying to inject public
and private keys
>> > > into
>> > > > > > systemvm
>> > > > > > >>> iso
>> > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script]
(main:null)
>> > > (logid:)
>> > > > > > >>> Looking for vms/systemvm.iso in the classpath
>> > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script]
(main:null)
>> > > (logid:)
>> > > > > > >>> System resource:
>> > > file:/usr/share/cloudstack-common/vms/systemvm.iso
>> > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script]
(main:null)
>> > > (logid:)
>> > > > > > >>> Absolute path =  /usr/share/cloudstack-common/vms/systemvm.iso
>> > > > > > >>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl]
>> > > > > > >>> (main:null) (logid:) Executing: /bin/bash
>> > > > > > >>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
>> > > > > > >>> /var/cloudstack/management/.ssh/id_rsa.pub
>> > > > > > >>> /var/cloudstack/management/.ssh/id_rsa
>> > > > > > >>> /usr/share/cloudstack-common/vms/systemvm.iso
>> > > > > > >>> 2020-07-25 00:34:17,636 INFO  [c.c.s.ConfigurationServerImpl]
>> > > > > > >>> (main:null) (logid:) Injected public and private
keys into
>> > > systemvm
>> > > > > iso
>> > > > > > >>> with result : null
>> > > > > > >>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> > > > > > >>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d)
Copying
>> > > > > > >>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>> > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.22
with permission
>> > > 0644
>> > > > > > >>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> > > > > > >>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a)
Copying
>> > > > > > >>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>> > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.5
with permission
>> > 0644
>> > > > > > >>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> > > > > > >>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b)
Copying
>> > > > > > >>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>> > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.9
with permission
>> > 0644
>> > > > > > >>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> > > > > > >>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf)
Copying
>> > > > > > >>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>> > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.5
with permission
>> > 0644
>> > > > > > >>>
>> > > > > > >>> I have attached complete management logs. The
start of logs is
>> > > the
>> > > > > > start
>> > > > > > >>> of management server after upgrade of management
server from
>> > > 4.11.3
>> > > > > to
>> > > > > > >>> 4.13.1.
>> > > > > > >>>
>> > > > > > >>> Ammad Ali
>> > > > > > >>>
>> > > > > > >>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic
<
>> > > > > andrija.panic@gmail.com
>> > > > > > >
>> > > > > > >>> wrote:
>> > > > > > >>>
>> > > > > > >>>> Do you get an error while trying to inject
ssh key into the
>> > > > > > systemvm.iso
>> > > > > > >>>> (mgmt logs) , or can you confirm that the
systemvm.iso on XS
>> > > host
>> > > > > and
>> > > > > > >>>> the
>> > > > > > >>>> one on the mgmt server are identical, md5sum
them (i.e. has
>> > the
>> > > > iso
>> > > > > > been
>> > > > > > >>>> copied over to the XS successfully) - this
might explain not
>> > > being
>> > > > > > able
>> > > > > > >>>> to
>> > > > > > >>>> login via ssh with your private key.
>> > > > > > >>>>
>> > > > > > >>>> Best,
>> > > > > > >>>>
>> > > > > > >>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed,
>> > >
>> > > > > wrote:
>> > > > > > >>>>
>> > > > > > >>>> > Yes this is exactly the same issue
that i faced.
>> > > > > > >>>> >
>> > > > > > >>>> > Sent from my iPhone
>> > > > > > >>>> >
>> > > > > > >>>> > > On 12-Aug-2020, at 8:35 AM, Eric
Lee Green <
>> > > > > > >>>> eric.lee.green@gmail.com>
>> > > > > > >>>> > wrote:
>> > > > > > >>>> > >
>> > > > > > >>>> > > ﻿Correct, 4.11.3 template is
used for 4.11.3, 4.12, and
>> > > 4.13.
>> > > > > 4.14
>> > > > > > >>>> moves
>> > > > > > >>>> > to the 4.14.0 template.
>> > > > > > >>>> > >
>> > > > > > >>>> > > There seems to be something odd
happening key-wise
>> > sometimes
>> > > > > with
>> > > > > > >>>> > upgrades from 4.11.3 to 4.13.1 or
4.14.0.   I managed an
>> > > > > > from
>> > > > > > >>>> > 4.11.3 to 4.13.1 that *almost* worked,
but the secondary
>> > > storage
>> > > > > VM
>> > > > > > >>>> > wouldn't work and thus I couldn't
spawn new virtual
>> > machines.
>> > > > Same
>> > > > > > >>>> symptom
>> > > > > > >>>> > -- key error when the agent tried
to ssh into it. And
>> > deleting
>> > > > it
>> > > > > > and
>> > > > > > >>>> > making it respawn didn't help. Then
I tried 4.11.3 to 4.14.0
>> > > and
>> > > > > > >>>> *all* the
>> > > > > > >>>> > VM's failed at that point (of course,
that was with the new
>> > > > > > template).
>> > > > > > >>>> > >
>> > > > > > >>>> > > Right now I'm back at 4.11.3
until this can be figured
>> > out.
>> > > > > > >>>> > >
>> > > > > > >>>> > >> On 8/11/2020 5:53 AM, Ammad
Syed wrote:
>> > > > > > >>>> > >> Hi,
>> > > > > > >>>> > >>
>> > > > > > >>>> > >> I think 4.12 and 4.13 uses
same systemVM template i.e
>> > > 4.11.3
>> > > > > > >>>> version,
>> > > > > > >>>> > which
>> > > > > > >>>> > >> I already have registered.
Currently I am running 4.11.3
>> > > > > version
>> > > > > > >>>> of ACS.
>> > > > > > >>>> > >>
>> > > > > > >>>> > >> MariaDB [cloud]> SELECT
id,name,type,cross_zones,state
>> > FROM
>> > > > > > >>>> > >> cloud.vm_template WHERE name
like '%systemvm-xenserver%'
>> > > AND
>> > > > > > >>>> removed IS
>> > > > > > >>>> > >> NULL;
>> > > > > > >>>> > >>
>> > > > > > >>>>
>> > > > > >
>> > > >
>> > +------+-----------------------------+--------+-------------+----------+
>> > > > > > >>>> > >> | id   | name
| type   |
>> > > cross_zones |
>> > > > > > >>>> state    |
>> > > > > > >>>> > >>
>> > > > > > >>>>
>> > > > > >
>> > > >
>> > +------+-----------------------------+--------+-------------+----------+
>> > > > > > >>>> > >> |  337 | systemvm-xenserver-3.0.0
| SYSTEM |
>> > >  0 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  418 | systemvm-xenserver-4.2
| SYSTEM |
>> > >  0 |
>> > > > > > >>>> Active   |
>> > > > > > >>>> > >> |  472 | systemvm-xenserver-4.3
| USER   |
>> > >  1 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  473 | systemvm-xenserver-4.3
| USER   |
>> > >  1 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  474 | systemvm-xenserver-4.3
| USER   |
>> > >  1 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  475 | systemvm-xenserver-4.3
| USER   |
>> > >  1 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  476 | systemvm-xenserver-4.3
| USER   |
>> > >  0 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  479 | systemvm-xenserver-4.3-2
| USER   |
>> > >  1 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  480 | systemvm-xenserver-4.3
| SYSTEM |
>> > >  0 |
>> > > > > > >>>> Active   |
>> > > > > > >>>> > >> |  549 | systemvm-xenserver-4.5.1
| USER   |
>> > >  0 |
>> > > > > > >>>> Active   |
>> > > > > > >>>> > >> |  550 | systemvm-xenserver-4.5.1
| SYSTEM |
>> > >  0 |
>> > > > > > >>>> Active   |
>> > > > > > >>>> > >> |  651 | systemvm-xenserver-4.7.0
| USER   |
>> > >  0 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  652 | systemvm-xenserver-4.7.0
| USER   |
>> > >  0 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  653 | systemvm-xenserver-4.7.0
| SYSTEM |
>> > >  0 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  737 | systemvm-xenserver-4.9.2
| SYSTEM |
>> > >  1 |
>> > > > > > >>>> Inactive |
>> > > > > > >>>> > >> |  739 | systemvm-xenserver-4.9.2-sb
| SYSTEM |
>> > >  1 |
>> > > > > > >>>> Active   |
>> > > > > > >>>> > >> | 1245 | systemvm-xenserver-4.11.1
| SYSTEM |
>> > >  1 |
>> > > > > > >>>> Active   |
>> > > > > > >>>> > >> | 1584 | systemvm-xenserver-4.11.2
| SYSTEM |
>> > >  1 |
>> > > > > > >>>> Active   |
>> > > > > > >>>> > >> | 1677 | systemvm-xenserver-4.11.3
| SYSTEM |
>> > >  1 |
>> > > > > > >>>> Active   |
>> > > > > > >>>> > >>
>> > > > > > >>>>
>> > > > > >
>> > > >
>> > +------+-----------------------------+--------+-------------+----------+
>> > > > > > >>>> > >>
>> > > > > > >>>> > >> - Ammad
>> > > > > > >>>> > >>
>> > > > > > >>>> > >> On Tue, Aug 11, 2020 at 5:17
PM Pierre-Luc Dion <
>> > > > > > >>>> pdion891@apache.org>
>> > > > > > >>>> > wrote:
>> > > > > > >>>> > >> db.
>> > > > > > >>>> > >>> Hi Syed,
>> > > > > > >>>> > >>> From 4.12, the systemvm
>> > > because
>> > > > of
>> > > > > > OS
>> > > > > > >>>> > change in
>> > > > > > >>>> > >>> the template, moved to
>> > Because
>> > > > of
>> > > > > > >>>> that,
>> > > > > > >>>> > some VR
>> > > > > > >>>> > >>> scripts have changed
and make obsolete older version of
>> > > VRs,
>> > > > > so
>> > > > > > >>>> you
>> > > > > > >>>> > will
>> > > > > > >>>> > >>> most likely have to register
an updated systemvm
>> > templates
>> > > > and
>> > > > > > >>>> upgrade
>> > > > > > >>>> > your
>> > > > > > >>>> > >>> system VMs and VRs.
>> > > > > > >>>> > >>>
>> > > > > > >>>> > >>> Regards,
>> > > > > > >>>> > >>>
>> > > > > > >>>> > >>>> On Tue, Aug 11, 2020
at 6:24 AM Ammad Syed <
>> > > > > > >>>> syedammad83@gmail.com>
>> > > > > > >>>> > wrote:
>> > > > > > >>>> > >>>
>> > > > > > >>>> > >>>> Hi Guys,
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> I was previously
on 4.9.3 cloudstack and upgraded to
>> > > 4.11.1
>> > > > > > then
>> > > > > > >>>> > 4.11.3.
>> > > > > > >>>> > >>>> The version 4.11.3
is working fine since six months.
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> Now I have tried
to upgrade my system from 4.11.3 to
>> > > > 4.13.1.
>> > > > > > The
>> > > > > > >>>> > upgrade
>> > > > > > >>>> > >>>> goes successful.
I didn't uploaded any system VM
>> > > template.
>> > > > > > >>>> However the
>> > > > > > >>>> > >>>> problem occured when
I recreated my systemVM of POD,
>> > the
>> > > VM
>> > > > > > >>>> recreated
>> > > > > > >>>> > and
>> > > > > > >>>> > >>>> its state was running
but agent state was not getting
>> > up,
>> > > > its
>> > > > > > >>>> showing
>> > > > > > >>>> > >>> blank
>> > > > > > >>>> > >>>> in column.
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> Digging further via
job logs, the job is failed with
>> > > error
>> > > > > that
>> > > > > > >>>> > unable to
>> > > > > > >>>> > >>>> execute command via
ssh. Below are the logs.
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> 2020-07-25 02:30:48,126
ERROR [c.c.u.s.SshHelper]
>> > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31)
(logid:9fa7dece) SSH
>> > > > execution
>> > > > > > of
>> > > > > > >>>> > command
>> > > > > > >>>> > >>>> /opt/cloud/bin/router_proxy.sh
keystore-s
>> > > > > > >>>> > >>>> etup 169.254.2.199
>> > > > > > >>>> /usr/local/cloud/systemvm/conf/agent.properties
>> > > > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/cloud.jks
>> > TJaQYChYBwKh7Cx9
>> > > > 365
>> > > > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/clou
>> > > > > > >>>> > >>>> d.csr has an error
status code in return. Result
>> > output:
>> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127
DEBUG
>> > > [c.c.a.m.DirectAgentAttache]
>> > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31)
(logid:9fa7dece) Seq
>> > > > > > >>>> > >>>> 906-3195585410596077730:
>> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127
DEBUG [c.c.a.t.Request]
>> > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31)
(logid:9fa7dece) Seq
>> > > > > > >>>> > >>>> 906-3195585410596077730:
Processing:  { Ans: , MgmtId:
>> > > > > > 779271079
>> > > > > > >>>> > >>>> 43497, via: 906(xen-21-10-a3-khi02),
Ver: v1, Flags:
>> > 10,
>> > > > > > >>>> > >>>> [{"org.apache.cloudstack.ca
>> > > > > > >>>> > >>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}]
>> > > > > > >>>> > >>>> }
>> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127
DEBUG [c.c.a.t.Request]
>> > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d
>> > > job-1208155/job-1208258
>> > > > > > >>>> > ctx-df740f75)
>> > > > > > >>>> > >>>> (logid:9fa7dece)
Seq 906-319558541059607773
>> > > > > > >>>> > >>>> 0: Received:  { Ans:
, MgmtId: 77927107943497, via:
>> > > > > > >>>> > >>>> 906(xen-21-10-a3-khi02),
Ver: v1, Flags: 10, {
>> > > > > > >>>> SetupKeystoreAnswer } }
>> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127
ERROR
>> > > > > [c.c.v.VirtualMachineManagerImpl]
>> > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d
>> > > job-1208155/job-1208258
>> > > > > > >>>> > ctx-df740f75)
>> > > > > > >>>> > >>>> (logid:9fa7dece)
Failed to setup keystore and generate
>> > > CSR
>> > > > > for
>> > > > > > >>>> system
>> > > > > > >>>> > vm:
>> > > > > > >>>> > >>>> s-24142-VM
>> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127
DEBUG
>> > > [c.c.v.VmWorkJobHandlerProxy]
>> > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d
>> > > job-1208155/job-1208258
>> > > > > > >>>> > ctx-df740f75)
>> > > > > > >>>> > >>>> (logid:9fa7dece)
Done executing VM work job:
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>
>> > > > > > >>>> >
>> > > > > > >>>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"}
>> > > > > > >>>> > >>>> 2020-07-25 02:30:48,128
DEBUG
>> > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d
>> > > job-1208155/job-1208258
>> > > > > > >>>> > ctx-df740f75)
>> > > > > > >>>> > >>>> (logid:9fa7dece)
Complete async job-1208258, jobStatus:
>> > > > > > >>>> SUCCEEDED,
>> > > > > > >>>> > >>>> resultCode: 0, result:
null
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> I tried to dig it
further, I was unable to login
>> > systemVM
>> > > > via
>> > > > > > >>>> ssh from
>> > > > > > >>>> > >>>> xenserver host with
key /root/.ssh/id_rsa.cloud placed.
>> > > > Look
>> > > > > > like
>> > > > > > >>>> > private
>> > > > > > >>>> > >>>> key issue. However
I am able to login on my old
>> > > systemVMs (
>> > > > > i.e
>> > > > > > >>>> > created
>> > > > > > >>>> > >>> on
>> > > > > > >>>> > >>>> ACS 4.11.3)
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> Also I have SSL certificate
enabled for console proxy
>> > on
>> > > my
>> > > > > ACS
>> > > > > > >>>> 4.11.3
>> > > > > > >>>> > >>> and
>> > > > > > >>>> > >>>> I am using only xenserver
7.0 hosts.
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> I tried to disable
SSL on secstorage and console proxy
>> > > from
>> > > > > > >>>> global
>> > > > > > >>>> > >>>> settings, but still
didn't worked.
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> I had a fresh installation
of ACS 4.13.1 with xenserver
>> > > > 7.0,
>> > > > > > >>>> systemVMs
>> > > > > > >>>> > >>> are
>> > > > > > >>>> > >>>> working fine in it.
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> --
>> > > > > > >>>> > >>>> Regards,
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>>> Syed Ammad Ali
>> > > > > > >>>> > >>>>
>> > > > > > >>>> > >>
>> > > > > > >>>> >
>> > > > > > >>>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>> --
>> > > > > > >>> Regards,
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>> Syed Ammad Ali
>> > > > > > >>>
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> --
>> > > > > > >> Regards,
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> Syed Ammad Ali
>> > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Regards,
>> > > > > > >
>> > > > > > >
>> > > > > > > Syed Ammad Ali
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Regards,
>> > > > > >
>> > > > > >
>> > > > > > Syed Ammad Ali
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Andrija Panić
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Regards,
>> > > >
>> > > >
>> > > > Syed Ammad Ali
>> > > >
>> > >
>> > >
>> > > --
>> > >
>> > > Andrija Panić
>> > >
>> >
>> >
>> > --
>> > Regards,
>> >
>> >
>> >
>>
>>
>> --
>>
>> Andrija Panić
>
>
> --
> Regards,
>
>