cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cristian Ciobanu <cristia...@istream.today>
Subject Re: cloud: Password server at 192.xx.1xx.79 did not have any password for the VM - After upgrade to ACS 4.14
Date Fri, 19 Jun 2020 20:53:36 GMT
I found out that there is a firewall issue and sshd config issue on VR on
this ACS version (4.14) when it is configured with basic networking.

By default management server is able to establish ssh connection only via
local IP with VR: eth1 172.11.0.167/24, but in order to run health check si
trying to connect via public IPs of the VR, this is not possible because of
this :

sshd config :
Port 3922
#AddressFamily any
ListenAddress 172.11.0.167, here i changed to 0.0.0.0

iptables :
-A INPUT -i eth1 -p tcp -m tcp --dport 3922 -m state --state
NEW,ESTABLISHED -j ACCEPT  ( rule for eth0 is missing ) in basic network it
will not work without this. I have added a rule to allow also for eth0

Regarding password issue:
in VR iptables there is only this rule :
-A INPUT -s 158.xx.xx.224/28 -i eth0 -p tcp -m tcp --dport 8080 -m state
--state NEW -j ACCEPT, only for the first, main public IP, not for all the
IPs, so i have added a rule to allow 8080 on each public IP from this
router.

Everything works now, till i destroy the router and i have to reconfigure
again.


root@r-3480-VM:~#
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP group default qlen 1000
    link/ether 1e:00:91:00:00:33 brd ff:ff:ff:ff:ff:ff
    inet 158.xx.xx.226/28 brd 158.xx.xx.239 scope global eth0
       valid_lft forever preferred_lft forever
    inet 167.xxx.xx.246/28 brd 167.xxx.xx.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 149.xx.xxx.80/27 brd 149.xx.xxx.95 scope global eth0
       valid_lft forever preferred_lft forever
    inet 192.xx.xxx.79/26 brd 192.xx.xxx.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet 198.xx.xxx.162/27 brd 198.xx.xxx.191 scope global eth0
       valid_lft forever preferred_lft forever
    inet 149.xx.xxx.99/27 brd 149.xx.xxx.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet 144.xxx.xx.199/27 brd 144.xxx.xx.223 scope global eth0
       valid_lft forever preferred_lft forever
    inet 144.xxx.xxx.177/27 brd 144.xxx.xxx.191 scope global eth0
       valid_lft forever preferred_lft forever
    inet 66.xxx.xxx.133/27 brd 66.xx.xxx.159 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP group default qlen 1000
    link/ether 02:00:57:d0:02:14 brd ff:ff:ff:ff:ff:ff
    inet 172.11.0.167/24 brd 172.11.0.255 scope global eth1
       valid_lft forever preferred_lft forever
root@r-3480-VM:~#


Regards,
Cristian

On Fri, 19 Jun 2020 at 21:40, Cristian Ciobanu <cristian.c@istream.today>
wrote:

> Hello,
>
>    This is what I tried first, restart the VM before trying to reset the
> pass.
>    The line you ask about was from messages log file. (vR)
>  BTW, I saw that now there is a local IP assigned to systems VM router.
> Till now only public IP was assigned, not sure if this have something to do
> with.
>
> Regards,
> Cristian
>
> On Fri, 19 Jun 2020, 19:04 Andrija Panic, <andrija.panic@gmail.com> wrote:
>
>> After the the upgrade to 4.14, I assume you have restarted all existing
>> VRs
>> or networks (new VRs are created from the new systemVM template for 4.14)?
>>
>> If you see the password inside this file (and you do, as it seems) - that
>> means that the VM (script) did not fetch the password. When the VM fetches
>> the password, it will say "saved" instead of the actual password in that
>> file.
>> Can you just reboot once more the VM - do NOT reset the password in the
>> meantime? Check logs after that and see if the password was changed.
>>
>> What about these 2 lines?
>>
>> Jun 19 14:08:04 systemvm passwd_server_ip.py: serve_password: password
>> saved
>> for VM IP 192.xx.xxx.80
>>
>> This indicates that the password was sent to the VM,
>>
>> Regards,
>> Andrija
>>
>>
>> On Fri, 19 Jun 2020 at 17:11, <cristian.c@istream.today> wrote:
>>
>> > Hi Andrija,
>> >
>> >    Please see :
>> >
>> >
>> > root@r-2705-VM:~# cat /var/cache/cloud/passwords-192.xx.xxx.79
>> > 192.xx.xxx.108=Yj4AZj
>> > 192.xx.xxx.101=pnj6dD
>> > 192.xx.xxx.115=Q7wyGw
>> > 192.xx.xxx.80=y2sS7E
>> >
>> >
>> > y2sS7E
>> >
>> > VM IP : 192.xx.xxx.80
>> >
>> >
>> > Regards,
>> > Cristian
>> >
>> > -----Original Message-----
>> > From: Andrija Panic <andrija.panic@gmail.com>
>> > Sent: Friday, June 19, 2020 5:56 PM
>> > To: users <users@cloudstack.apache.org>
>> > Subject: Re: cloud: Password server at 192.xx.1xx.79 did not have any
>> > password for the VM - After upgrade to ACS 4.14
>> >
>> > Can you reset a password for a VM, boot the VM and then provide the
>> > content of the file /var/cache/cloud/password* from the VR?
>> >
>> > Regards,
>> > Andrija
>> >
>> > On Fri, 19 Jun 2020 at 16:20, <cristian.c@istream.today> wrote:
>> >
>> > > Hello folks,
>> > >
>> > >
>> > >
>> > >      I have successfully upgraded my Cloudstack 4.11 to 4.14 ( VMware
>> > with
>> > > Basic Networking ) everything works except password for VMs.   I did
>> > > multiple testes, different OS, looks like it is not working anymore,
>> > > any idea why?
>> > >
>> > >
>> > >
>> > > VM log:
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Jun 19 10:57:14 localhost cloud-set-guest-password: Starting
>> > > cloud-set-guest-password:  [  OK  ]
>> > >
>> > > Jun 19 10:57:14 localhost cloud-set-guest-sshkey: Starting
>> > > cloud-set-guest-sshkey:  [  OK  ]
>> > >
>> > > Jun 19 10:57:14 localhost cloud: Sending request to ssh key server at
>> > > 192.xx.xxx.79
>> > >
>> > > Jun 19 10:57:14 localhost cloud: Found password server IP
>> > > 192.xx.xxx.79 in
>> > >
>> > > /var/lib/NetworkManager/dhclient-6395a6b2-9b5d-4daa-86bd-343e5b823d5e-
>> > > eno167
>> > > 77752.lease
>> > >
>> > > Jun 19 10:57:14 localhost cloud: Sending request to password server at
>> > > 192.xx.xxx79
>> > >
>> > > Jun 19 10:57:15 localhost systemd: Started Dynamic System Tuning
>> Daemon.
>> > >
>> > > Jun 19 10:57:15 localhost systemd: Started Postfix Mail Transport
>> Agent.
>> > >
>> > > Jun 19 10:57:15 localhost kdumpctl: No memory reserved for crash
>> kernel.
>> > >
>> > > Jun 19 10:57:15 localhost kdumpctl: Starting kdump: [FAILED]
>> > >
>> > > Jun 19 10:57:15 localhost systemd: kdump.service: main process exited,
>> > > code=exited, status=1/FAILURE
>> > >
>> > > Jun 19 10:57:15 localhost systemd: Failed to start Crash recovery
>> > > kernel arming.
>> > >
>> > > Jun 19 10:57:15 localhost systemd: Unit kdump.service entered failed
>> > state.
>> > >
>> > > Jun 19 10:57:15 localhost systemd: kdump.service failed.
>> > >
>> > > Jun 19 10:57:42 localhost systemd: Created slice user-0.slice.
>> > >
>> > > Jun 19 10:57:42 localhost systemd: Starting user-0.slice.
>> > >
>> > > Jun 19 10:57:42 localhost systemd: Started Session 1 of user root.
>> > >
>> > > Jun 19 10:57:42 localhost systemd-logind: New session 1 of user root.
>> > >
>> > > Jun 19 10:57:42 localhost systemd: Starting Session 1 of user root.
>> > >
>> > > Jun 19 10:58:17 localhost cloud: Failed to get ssh keys from any
>> > > server
>> > >
>> > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service:
>> > > main process exited, code=exited, status=1/FAILURE
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Failed to start Cloud Set Guest
>> > > SSHKey Service.
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Unit cloud-set-guest-sshkey.service
>> > > entered failed state.
>> > >
>> > > Jun 19 10:58:17 localhost systemd: cloud-set-guest-sshkey.service
>> failed.
>> > >
>> > > Jun 19 10:58:17 localhost cloud: Got response from server at
>> > > 192.xx.xxx.79
>> > >
>> > > Jun 19 10:58:17 localhost cloud: Password server at 192.xx.xxx.79did
>> > > not have any password for the VM
>> > >
>> > > Jun 19 10:58:17 localhost cloud: Did not need to change password.
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Started Cloud Set Guest Password
>> > > Service.
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Reached target Multi-User System.
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Starting Multi-User System.
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Started Stop Read-Ahead Data
>> > > Collection 10s After Completed Startup.
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Starting Update UTMP about System
>> > > Runlevel Changes...
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Started Update UTMP about System
>> > > Runlevel Changes.
>> > >
>> > > Jun 19 10:58:17 localhost systemd: Startup finished in 521ms (kernel)
>> > > + 1.563s (initrd) + 1min 10.596s (userspace) = 1min 12.681s.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Router Log :
>> > >
>> > > Jun 19 12:15:24 systemvm cloud: VR config: create file success
>> > >
>> > > Jun 19 12:15:24 systemvm cloud: VR config: executing:
>> > > /opt/cloud/bin/update_config.py
>> > > vm_metadata.json.a997727c-51e2-4730-b1e4-5a033cf8672f
>> > >
>> > > Jun 19 12:15:24 systemvm cloud: VR config: execution success
>> > >
>> > > Jun 19 12:15:24 systemvm cloud: VR config: creating file:
>> > > /var/cache/cloud/vm_metadata.json.079f12c9-45d1-48e0-986c-0ed68f764128
>> > >
>> > > Jun 19 12:15:24 systemvm cloud: VR config: create file success
>> > >
>> > > Jun 19 12:15:24 systemvm cloud: VR config: executing:
>> > > /opt/cloud/bin/update_config.py
>> > > vm_metadata.json.079f12c9-45d1-48e0-986c-0ed68f764128
>> > >
>> > > Jun 19 12:08:48 systemvm cloud: VR config: execution success
>> > >
>> > > Jun 19 12:08:50 systemvm cloud: VR config: Flushing conntrack table
>> > >
>> > > Jun 19 12:08:50 systemvm cloud: VR config: Flushing conntrack table
>> > > completed
>> > >
>> > > Jun 19 12:13:46 systemvm kernel: [  320.771392] nf_conntrack: default
>> > > automatic helper assignment has been turned off for security reasons
>> > > and CT-based  firewall rule not found. Use the iptables CT target to
>> > > attach helpers instead.
>> > >
>> > > Jun 19 13:38:36 systemvm passwd_server_ip.py: serve_password: password
>> > > saved for VM IP 192.xx.xxx.101
>> > >
>> > > Jun 19 13:47:24 systemvm passwd_server_ip.py: serve_password: password
>> > > saved for VM IP 192.xx.xxx.101
>> > >
>> > > Jun 19 13:53:00 systemvm passwd_server_ip.py: serve_password: password
>> > > saved for VM IP 192.xx.xxx.108
>> > >
>> > > Jun 19 14:05:22 systemvm passwd_server_ip.py: serve_password: password
>> > > saved for VM IP 192.xx.xxx.108
>> > >
>> > > Jun 19 14:08:04 systemvm passwd_server_ip.py: serve_password: password
>> > > saved for VM IP 192.xx.xxx.80
>> > >
>> > >
>> > >
>> > > 2020-06-19 14:08:19,737 INFO     Executing: systemctl start
>> > > cloud-password-server@192.xx.xxx.79
>> > >
>> > > 2020-06-19 14:08:19,742 INFO     Service
>> > > cloud-password-server@192.xx.xxx.79
>> > > start
>> > >
>> > > 2020-06-19 14:08:19,742 INFO     Checking if default ipv4 route is
>> > present
>> > >
>> > > 2020-06-19 14:08:19,742 INFO     Executing: ip -4 route list 0/0
>> > >
>> > > 2020-06-19 14:08:19,744 INFO     Default route found: default via
>> > > 158.xx.xx.238 dev eth0
>> > >
>> > > 2020-06-19 14:08:19,744 INFO     Address found in DataBag ==>
>> > > {u'public_ip':
>> > > u'198.xx.xxx.162', u'nic_dev_id': u'0', u'network':
>> > > u'198.xxx.xxx.160/27',
>> > > u'netmask': u'255.255.255.224', u'broadcast': u'198.xxx.xxx.191',
>> u'add':
>> > > True, u'nw_type': u'guest', u'device': u'eth0', u'cidr':
>> > > u'198.xxx.xxx.162/27', u'size': u'27'}
>> > >
>> > > 2020-06-19 14:08:19,744 INFO     Address 198.xx.xxx.162/27 on device
>> eth0
>> > > already configured
>> > >
>> > > 2020-06-19 14:08:19,744 INFO     Adding route table: 100 Table_eth0 to
>> > > /etc/iproute2/rt_tables if not present
>> > >
>> > > 2020-06-19 14:08:19,744 INFO     Executing: ip rule show
>> > >
>> > > 2020-06-19 14:08:19,746 INFO     Executing: ip rule show
>> > >
>> > > 2020-06-19 14:08:19,748 INFO     Executing: ip link show eth0 | grep
>> > 'state
>> > > DOWN'
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Best regards,
>> > >
>> > > Cristian
>> > >
>> > >
>> >
>> > --
>> >
>> > Andrija Panić
>> >
>> >
>>
>> --
>>
>> Andrija Panić
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message