cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ammad Syed <syedamma...@gmail.com>
Subject Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Date Mon, 17 Aug 2020 13:23:30 GMT
I have rechecked the logs are there. Please review below lines.


C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors
(27 hits)
Line 409969: 2020-07-25 01:45:52,320 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334
job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to setup
keystore and generate CSR for system vm: s-25023-VM
Line 437413: 2020-07-25 01:51:33,845 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-70:ctx-457ce4ee
job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to setup
keystore and generate CSR for system vm: s-25024-VM
Line 549538: 2020-07-25 02:06:44,833 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-4634f7e0
job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to setup
keystore and generate CSR for system vm: v-25027-VM
Line 663111: 2020-07-25 02:23:43,783 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-fe0b857b
job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to setup
keystore and generate CSR for system vm: v-20771-VM
Line 667060: 2020-07-25 02:24:30,221 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-662cfe5d
job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to setup
keystore and generate CSR for system vm: v-24763-VM
Line 674269: 2020-07-25 02:25:47,345 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-b6529613
job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to setup
keystore and generate CSR for system vm: v-20773-VM
Line 678169: 2020-07-25 02:26:20,355 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-7a062a66
job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to setup
keystore and generate CSR for system vm: v-25027-VM
Line 697113: 2020-07-25 02:30:09,669 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-40:ctx-3be9fddc
job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to setup
keystore and generate CSR for system vm: v-20771-VM
Line 701161: 2020-07-25 02:30:48,127 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d
job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to setup
keystore and generate CSR for system vm: s-24142-VM
Line 701838: 2020-07-25 02:30:56,759 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-42:ctx-04ac8e71
job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to setup
keystore and generate CSR for system vm: v-24763-VM
Line 703865: 2020-07-25 02:31:25,123 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-24bdbe8f
job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to setup
keystore and generate CSR for system vm: v-20773-VM
Line 712078: 2020-07-25 02:32:28,666 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d5db5c2
job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to setup
keystore and generate CSR for system vm: s-25033-VM
Line 715204: 2020-07-25 02:33:02,875 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-51:ctx-594da6d2
job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to setup
keystore and generate CSR for system vm: s-25035-VM
Line 743140: 2020-07-25 02:38:52,257 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-a3f032da
job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to setup
keystore and generate CSR for system vm: s-25037-VM
Line 817646: 2020-07-25 02:47:07,127 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-ca073bd1
job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to setup
keystore and generate CSR for system vm: s-25038-VM
Line 847150: 2020-07-25 02:51:46,293 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-9db8c548
job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to setup
keystore and generate CSR for system vm: v-25040-VM
Line 903949: 2020-07-25 02:57:48,515 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f35fd59a
job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-25041-VM
Line 957628: 2020-07-25 03:04:13,286 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-dc18956a
job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-20771-VM
Line 961606: 2020-07-25 03:05:00,522 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-83e48421
job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-24763-VM
Line 966633: 2020-07-25 03:05:29,767 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-30:ctx-897e2a66
job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-20773-VM
Line 968892: 2020-07-25 03:05:53,648 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-36:ctx-68e45b80
job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-25041-VM
Line 1003036: 2020-07-25 03:13:33,582 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-45:ctx-0f552e0e
job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-20771-VM
Line 1006496: 2020-07-25 03:14:09,787 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-08e5db24
job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-24763-VM
Line 1008408: 2020-07-25 03:14:39,584 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-8b9288a9
job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-20773-VM
Line 1023914: 2020-07-25 03:15:58,139 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d54d717
job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-25046-VM
Line 1089568: 2020-07-25 03:24:24,534 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-62ff8844
job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to setup
keystore and generate CSR for system vm: v-25048-VM
Line 1097605: 2020-07-25 03:25:23,179 ERROR
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-62:ctx-24407a9e
job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to setup
keystore and generate CSR for system vm: s-25050-VM

-Ammad Ali



On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <andrija.panic@gmail.com>
wrote:

> Ammad, no such log lines in the file you uploaded....?
>
> On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedammad83@gmail.com> wrote:
>
> > Hi Andrija,
> >
> > The error occurred only one time. Can you please try to grep "Failed to
> > setup keystore and generate CSR" you will see all the systemVMs that I
> > tried to create failed because of certificate generation failed. OR you
> can
> > grep job-1208321/job-1208452 this job as well.
> >
> > Also I mentioned above that same thing is happening in my test
> environment
> > that I created 4.11.1 then upgraded 4.11.3 > 4.13.1.
> >
> > Ammad Ali
> >
> > On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic <andrija.panic@gmail.com>
> > wrote:
> >
> > > 2020-07-25 03:23:51,093 WARN  [c.c.v.SystemVmLoadScanner]
> > > (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception
> Failed
> > to
> > > fetch any free public IP address
> > > com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any
> free
> > > public IP address
> > >
> > > ?
> > >
> > > On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedammad83@gmail.com>
> wrote:
> > >
> > > > I have checked further, the key is successfully injected in
> > systemvm.iso
> > > on
> > > > the management node and copied successfully to xenserver. I have
> > checked
> > > on
> > > > systemVM there is no public key found in authorized_keys of root.
> > > >
> > > > How can I troubleshoot this further ? how can I enable trace logging
> > for
> > > > management server to see if there is something problematic happening.
> > > >
> > > > Ammad Ali
> > > >
> > > > On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedammad83@gmail.com>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have setup my test environment, here is what I did:
> > > > >
> > > > > - Installed ACS 4.11.1 and added xenserver 7.0 host in it.
> SystemVMs
> > > are
> > > > > up in the zone and agents are up.
> > > > > - Then upgraded the system to 4.11.3. Recreated systemVMs, agent
is
> > up
> > > > and
> > > > > systemVM are up with 4.11.3 systemVM version.
> > > > > - Then upgraded the system to 4.13.1, recreated systemVMs are
> running
> > > but
> > > > > agents are not up.
> > > > >
> > > > > I have checked md5sum of systemvm.iso on xenserver and management
> > > server,
> > > > > both are same.
> > > > >
> > > > > [root@xenserver iso]# md5sum
> > /opt/xensource/packages/iso/systemvm.iso
> > > > > baba18f156395da3a5d8208727d8f421
> > > > /opt/xensource/packages/iso/systemvm.iso
> > > > >
> > > > > [root@cloudstack-upgrade vms]# md5sum
> > > > > /usr/share/cloudstack-common/vms/systemvm.iso
> > > > > baba18f156395da3a5d8208727d8f421
> > > > >  /usr/share/cloudstack-common/vms/systemvm.iso
> > > > >
> > > > > Also the private key on xenserver root and management server are
> > same.
> > > > >
> > > > > management server
> > > > > /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud
> > > > >
> > > > > xenserver path
> > > > > /root/.ssh/id_rsa.cloud
> > > > >
> > > > > - Ammad Ali
> > > > >
> > > > > On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <syedammad83@gmail.com
> >
> > > > wrote:
> > > > >
> > > > >> Here is the link for download management logs.
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing
> > > > >>
> > > > >> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <
> syedammad83@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> I have reverted the version back to 4.11.3. But I have saved
logs
> > > > >>> starting from upgrade.
> > > > >>>
> > > > >>> I think the key has been copied successfully in system vm
iso.
> > > > >>>
> > > > >>> 2020-07-25 00:34:17,214 INFO  [c.c.s.ConfigurationServerImpl]
> > > > >>> (main:null) (logid:) Going to update systemvm iso with generated
> > > > keypairs
> > > > >>> if needed
> > > > >>> 2020-07-25 00:34:17,214 INFO  [c.c.s.ConfigurationServerImpl]
> > > > >>> (main:null) (logid:) Trying to inject public and private
keys
> into
> > > > systemvm
> > > > >>> iso
> > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> (logid:)
> > > > >>> Looking for vms/systemvm.iso in the classpath
> > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> (logid:)
> > > > >>> System resource:
> file:/usr/share/cloudstack-common/vms/systemvm.iso
> > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> (logid:)
> > > > >>> Absolute path =  /usr/share/cloudstack-common/vms/systemvm.iso
> > > > >>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl]
> > > > >>> (main:null) (logid:) Executing: /bin/bash
> > > > >>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> > > > >>> /var/cloudstack/management/.ssh/id_rsa.pub
> > > > >>> /var/cloudstack/management/.ssh/id_rsa
> > > > >>> /usr/share/cloudstack-common/vms/systemvm.iso
> > > > >>> 2020-07-25 00:34:17,636 INFO  [c.c.s.ConfigurationServerImpl]
> > > > >>> (main:null) (logid:) Injected public and private keys into
> systemvm
> > > iso
> > > > >>> with result : null
> > > > >>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > > > >>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying
> > > > >>>
> > > >
> > >
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > > > >>> to /opt/xensource/packages/iso on 172.16.2.22 with permission
> 0644
> > > > >>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > > > >>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying
> > > > >>>
> > > >
> > >
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > > > >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission
0644
> > > > >>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > > > >>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying
> > > > >>>
> > > >
> > >
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > > > >>> to /opt/xensource/packages/iso on 172.16.2.9 with permission
0644
> > > > >>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > > > >>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying
> > > > >>>
> > > >
> > >
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > > > >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission
0644
> > > > >>>
> > > > >>> I have attached complete management logs. The start of logs
is
> the
> > > > start
> > > > >>> of management server after upgrade of management server from
> 4.11.3
> > > to
> > > > >>> 4.13.1.
> > > > >>>
> > > > >>> Ammad Ali
> > > > >>>
> > > > >>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic <
> > > andrija.panic@gmail.com
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Do you get an error while trying to inject ssh key into
the
> > > > systemvm.iso
> > > > >>>> (mgmt logs) , or can you confirm that the systemvm.iso
on XS
> host
> > > and
> > > > >>>> the
> > > > >>>> one on the mgmt server are identical, md5sum them (i.e.
has the
> > iso
> > > > been
> > > > >>>> copied over to the XS successfully) - this might explain
not
> being
> > > > able
> > > > >>>> to
> > > > >>>> login via ssh with your private key.
> > > > >>>>
> > > > >>>> Best,
> > > > >>>>
> > > > >>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <syedammad83@gmail.com>
> > > wrote:
> > > > >>>>
> > > > >>>> > Yes this is exactly the same issue that i faced.
> > > > >>>> >
> > > > >>>> > Sent from my iPhone
> > > > >>>> >
> > > > >>>> > > On 12-Aug-2020, at 8:35 AM, Eric Lee Green
<
> > > > >>>> eric.lee.green@gmail.com>
> > > > >>>> > wrote:
> > > > >>>> > >
> > > > >>>> > > Correct, 4.11.3 template is used for 4.11.3,
4.12, and
> 4.13.
> > > 4.14
> > > > >>>> moves
> > > > >>>> > to the 4.14.0 template.
> > > > >>>> > >
> > > > >>>> > > There seems to be something odd happening key-wise
sometimes
> > > with
> > > > >>>> > upgrades from 4.11.3 to 4.13.1 or 4.14.0.   I managed
an
> upgrade
> > > > from
> > > > >>>> > 4.11.3 to 4.13.1 that *almost* worked, but the secondary
> storage
> > > VM
> > > > >>>> > wouldn't work and thus I couldn't spawn new virtual
machines.
> > Same
> > > > >>>> symptom
> > > > >>>> > -- key error when the agent tried to ssh into it.
And deleting
> > it
> > > > and
> > > > >>>> > making it respawn didn't help. Then I tried 4.11.3
to 4.14.0
> and
> > > > >>>> *all* the
> > > > >>>> > VM's failed at that point (of course, that was with
the new
> > > > template).
> > > > >>>> > >
> > > > >>>> > > Right now I'm back at 4.11.3 until this can
be figured out.
> > > > >>>> > >
> > > > >>>> > >> On 8/11/2020 5:53 AM, Ammad Syed wrote:
> > > > >>>> > >> Hi,
> > > > >>>> > >>
> > > > >>>> > >> I think 4.12 and 4.13 uses same systemVM
template i.e
> 4.11.3
> > > > >>>> version,
> > > > >>>> > which
> > > > >>>> > >> I already have registered. Currently I
am running 4.11.3
> > > version
> > > > >>>> of ACS.
> > > > >>>> > >>
> > > > >>>> > >> MariaDB [cloud]> SELECT id,name,type,cross_zones,state
FROM
> > > > >>>> > >> cloud.vm_template WHERE name like '%systemvm-xenserver%'
> AND
> > > > >>>> removed IS
> > > > >>>> > >> NULL;
> > > > >>>> > >>
> > > > >>>>
> > > >
> > +------+-----------------------------+--------+-------------+----------+
> > > > >>>> > >> | id   | name                        |
type   |
> cross_zones |
> > > > >>>> state    |
> > > > >>>> > >>
> > > > >>>>
> > > >
> > +------+-----------------------------+--------+-------------+----------+
> > > > >>>> > >> |  337 | systemvm-xenserver-3.0.0    |
SYSTEM |
>  0 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  418 | systemvm-xenserver-4.2      |
SYSTEM |
>  0 |
> > > > >>>> Active   |
> > > > >>>> > >> |  472 | systemvm-xenserver-4.3      |
USER   |
>  1 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  473 | systemvm-xenserver-4.3      |
USER   |
>  1 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  474 | systemvm-xenserver-4.3      |
USER   |
>  1 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  475 | systemvm-xenserver-4.3      |
USER   |
>  1 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  476 | systemvm-xenserver-4.3      |
USER   |
>  0 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  479 | systemvm-xenserver-4.3-2    |
USER   |
>  1 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  480 | systemvm-xenserver-4.3      |
SYSTEM |
>  0 |
> > > > >>>> Active   |
> > > > >>>> > >> |  549 | systemvm-xenserver-4.5.1    |
USER   |
>  0 |
> > > > >>>> Active   |
> > > > >>>> > >> |  550 | systemvm-xenserver-4.5.1    |
SYSTEM |
>  0 |
> > > > >>>> Active   |
> > > > >>>> > >> |  651 | systemvm-xenserver-4.7.0    |
USER   |
>  0 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  652 | systemvm-xenserver-4.7.0    |
USER   |
>  0 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  653 | systemvm-xenserver-4.7.0    |
SYSTEM |
>  0 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  737 | systemvm-xenserver-4.9.2    |
SYSTEM |
>  1 |
> > > > >>>> Inactive |
> > > > >>>> > >> |  739 | systemvm-xenserver-4.9.2-sb |
SYSTEM |
>  1 |
> > > > >>>> Active   |
> > > > >>>> > >> | 1245 | systemvm-xenserver-4.11.1   |
SYSTEM |
>  1 |
> > > > >>>> Active   |
> > > > >>>> > >> | 1584 | systemvm-xenserver-4.11.2   |
SYSTEM |
>  1 |
> > > > >>>> Active   |
> > > > >>>> > >> | 1677 | systemvm-xenserver-4.11.3   |
SYSTEM |
>  1 |
> > > > >>>> Active   |
> > > > >>>> > >>
> > > > >>>>
> > > >
> > +------+-----------------------------+--------+-------------+----------+
> > > > >>>> > >>
> > > > >>>> > >> - Ammad
> > > > >>>> > >>
> > > > >>>> > >> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc
Dion <
> > > > >>>> pdion891@apache.org>
> > > > >>>> > wrote:
> > > > >>>> > >> db.
> > > > >>>> > >>> Hi Syed,
> > > > >>>> > >>> From 4.12, the systemvm template had
to be upgraded
> because
> > of
> > > > OS
> > > > >>>> > change in
> > > > >>>> > >>> the template, moved to a latest version
of Debian. Because
> > of
> > > > >>>> that,
> > > > >>>> > some VR
> > > > >>>> > >>> scripts have changed and make obsolete
older version of
> VRs,
> > > so
> > > > >>>> you
> > > > >>>> > will
> > > > >>>> > >>> most likely have to register an updated
systemvm templates
> > and
> > > > >>>> upgrade
> > > > >>>> > your
> > > > >>>> > >>> system VMs and VRs.
> > > > >>>> > >>>
> > > > >>>> > >>> Regards,
> > > > >>>> > >>>
> > > > >>>> > >>>> On Tue, Aug 11, 2020 at 6:24 AM
Ammad Syed <
> > > > >>>> syedammad83@gmail.com>
> > > > >>>> > wrote:
> > > > >>>> > >>>
> > > > >>>> > >>>> Hi Guys,
> > > > >>>> > >>>>
> > > > >>>> > >>>> I was previously on 4.9.3 cloudstack
and upgraded to
> 4.11.1
> > > > then
> > > > >>>> > 4.11.3.
> > > > >>>> > >>>> The version 4.11.3 is working fine
since six months.
> > > > >>>> > >>>>
> > > > >>>> > >>>> Now I have tried to upgrade my
system from 4.11.3 to
> > 4.13.1.
> > > > The
> > > > >>>> > upgrade
> > > > >>>> > >>>> goes successful. I didn't uploaded
any system VM
> template.
> > > > >>>> However the
> > > > >>>> > >>>> problem occured when I recreated
my systemVM of POD, the
> VM
> > > > >>>> recreated
> > > > >>>> > and
> > > > >>>> > >>>> its state was running but agent
state was not getting up,
> > its
> > > > >>>> showing
> > > > >>>> > >>> blank
> > > > >>>> > >>>> in column.
> > > > >>>> > >>>>
> > > > >>>> > >>>> Digging further via job logs, the
job is failed with
> error
> > > that
> > > > >>>> > unable to
> > > > >>>> > >>>> execute command via ssh. Below
are the logs.
> > > > >>>> > >>>>
> > > > >>>> > >>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper]
> > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31)
(logid:9fa7dece) SSH
> > execution
> > > > of
> > > > >>>> > command
> > > > >>>> > >>>> /opt/cloud/bin/router_proxy.sh
keystore-s
> > > > >>>> > >>>> etup 169.254.2.199
> > > > >>>> /usr/local/cloud/systemvm/conf/agent.properties
> > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/cloud.jks
TJaQYChYBwKh7Cx9
> > 365
> > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/clou
> > > > >>>> > >>>> d.csr has an error status code
in return. Result output:
> > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG
> [c.c.a.m.DirectAgentAttache]
> > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31)
(logid:9fa7dece) Seq
> > > > >>>> > >>>> 906-3195585410596077730: Response
Received:
> > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31)
(logid:9fa7dece) Seq
> > > > >>>> > >>>> 906-3195585410596077730: Processing:
 { Ans: , MgmtId:
> > > > 779271079
> > > > >>>> > >>>> 43497, via: 906(xen-21-10-a3-khi02),
Ver: v1, Flags: 10,
> > > > >>>> > >>>> [{"org.apache.cloudstack.ca
> > > > >>>> > >>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}]
> > > > >>>> > >>>> }
> > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d
> job-1208155/job-1208258
> > > > >>>> > ctx-df740f75)
> > > > >>>> > >>>> (logid:9fa7dece) Seq 906-319558541059607773
> > > > >>>> > >>>> 0: Received:  { Ans: , MgmtId:
77927107943497, via:
> > > > >>>> > >>>> 906(xen-21-10-a3-khi02), Ver: v1,
Flags: 10, {
> > > > >>>> SetupKeystoreAnswer } }
> > > > >>>> > >>>> 2020-07-25 02:30:48,127 ERROR
> > > [c.c.v.VirtualMachineManagerImpl]
> > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d
> job-1208155/job-1208258
> > > > >>>> > ctx-df740f75)
> > > > >>>> > >>>> (logid:9fa7dece) Failed to setup
keystore and generate
> CSR
> > > for
> > > > >>>> system
> > > > >>>> > vm:
> > > > >>>> > >>>> s-24142-VM
> > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG
> [c.c.v.VmWorkJobHandlerProxy]
> > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d
> job-1208155/job-1208258
> > > > >>>> > ctx-df740f75)
> > > > >>>> > >>>> (logid:9fa7dece) Done executing
VM work job:
> > > > >>>> > >>>>
> > > > >>>> > >>>>
> > > > >>>> > >>>
> > > > >>>> >
> > > > >>>>
> > > >
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"}
> > > > >>>> > >>>> 2020-07-25 02:30:48,128 DEBUG
> > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d
> job-1208155/job-1208258
> > > > >>>> > ctx-df740f75)
> > > > >>>> > >>>> (logid:9fa7dece) Complete async
job-1208258, jobStatus:
> > > > >>>> SUCCEEDED,
> > > > >>>> > >>>> resultCode: 0, result: null
> > > > >>>> > >>>>
> > > > >>>> > >>>> I tried to dig it further, I was
unable to login systemVM
> > via
> > > > >>>> ssh from
> > > > >>>> > >>>> xenserver host with key /root/.ssh/id_rsa.cloud
placed.
> > Look
> > > > like
> > > > >>>> > private
> > > > >>>> > >>>> key issue. However I am able to
login on my old
> systemVMs (
> > > i.e
> > > > >>>> > created
> > > > >>>> > >>> on
> > > > >>>> > >>>> ACS 4.11.3)
> > > > >>>> > >>>>
> > > > >>>> > >>>> Also I have SSL certificate enabled
for console proxy on
> my
> > > ACS
> > > > >>>> 4.11.3
> > > > >>>> > >>> and
> > > > >>>> > >>>> I am using only xenserver 7.0 hosts.
> > > > >>>> > >>>>
> > > > >>>> > >>>> I tried to disable SSL on secstorage
and console proxy
> from
> > > > >>>> global
> > > > >>>> > >>>> settings, but still didn't worked.
> > > > >>>> > >>>>
> > > > >>>> > >>>> I had a fresh installation of ACS
4.13.1 with xenserver
> > 7.0,
> > > > >>>> systemVMs
> > > > >>>> > >>> are
> > > > >>>> > >>>> working fine in it.
> > > > >>>> > >>>>
> > > > >>>> > >>>> Please advise.
> > > > >>>> > >>>> --
> > > > >>>> > >>>> Regards,
> > > > >>>> > >>>>
> > > > >>>> > >>>>
> > > > >>>> > >>>> Syed Ammad Ali
> > > > >>>> > >>>>
> > > > >>>> > >>
> > > > >>>> >
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Regards,
> > > > >>>
> > > > >>>
> > > > >>> Syed Ammad Ali
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Regards,
> > > > >>
> > > > >>
> > > > >> Syed Ammad Ali
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > >
> > > > > Syed Ammad Ali
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > >
> > > > Syed Ammad Ali
> > > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> > --
> > Regards,
> >
> >
> > Syed Ammad Ali
> >
>
>
> --
>
> Andrija Panić
>


-- 
Regards,


Syed Ammad Ali

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message