cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Lee Green <eric.lee.gr...@gmail.com>
Subject Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Date Wed, 22 May 2019 19:49:39 GMT
EXCELLENT!

That did it. I downgraded to the libvirt and openssh rpms from Centos 
7.5, which was the latest DVD I had hanging around (likely the one I 
used to install the systems with in the first place). Turns out it was 
the Red Hat updates that came in when I did 'rpm update' that broke 
things, not the Cloudstack upgrade. I am now happily running on 
Cloudstack 4.11.2. Got rid of the qemu-kvm-ev stuff, btw, I had 
installed that at one point trying to resolve the problems, but have now 
downgraded it back to the release stuff.

In case it is not obvious, I am the person who is serious about security 
policies. The company itself doesn't care as long as we have one to show 
to investors and customers.

Regarding hardware, security policies don't cost a lot of money given 
all the open source tools we have available now. New hardware does cost 
money. Old commercial grade hardware is ridiculously cheap from Fleabay 
right now as big corporations dump "obsolete" gear.  I bought three used 
commercial-grade 24-port 10 gigabit switches for the price of a single 
new 1 gigabit switch, for example. I won't have cheap white box junk in 
the back room, it's three racks of commercial-quality gear with dual 
processors, ECC, 10gbit Ethernet, IPMI for remote management, etc., 
reliable as a brick -- just old and a bit power hungry. Power is the 
biggest issue that will lead to an upgrade, not reliability or 
performance -- I only have power budget for three racks, and new 
hardware is much more dense for a specific power consumption (modern 16 
core processors use about the same power as the 6 core processors I have 
in my nodes). When I have the budget to upgrade the hardware, I will, 
but what's there works, is reliable as a brick and not an issue, unlike 
Red Hat breaking my infrastructure with incompatible updates GRR!

On 5/22/19 9:20 AM, Andrija Panic wrote:
> +1 on what Nux said - simply downgrade the packages and check if that works
> - no need to reinstall everything !
> As for the $3333 from github (CentOS 7.6 issue) - simple new line at the
> end of id_rsa file and/or downgrading the ssh-keygen will work.
>
> Don't want to annoy you, but if the company is so serious about security
> policies ask them to be serious about HW as well (regular white trash PC
> will do...so you can test software itself) - otherwise, you personally
> suffer the consequences and look bad  (had the same pain many years ago...).
>
> On Wed, 22 May 2019 at 17:44, Nux! <nux@li.nux.ro> wrote:
>
>> Eric,
>>
>> I think you can get away with just downgrading to stock qemu-kvm packages
>> (or patch your cloudstack installation).
>> BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages
>> don't get the same level of testing and QA.
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
>> ----- Original Message -----
>>> From: "Eric Lee Green" <eric.lee.green@gmail.com>
>>> To: "users" <users@cloudstack.apache.org>
>>> Sent: Wednesday, 22 May, 2019 15:33:15
>>> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>>> Okay. This makes sense.
>>>
>>> And people wonder why Amazon decided to make their own Linux rather than
>>> use Centos and why Ubuntu has seized huge market share from Red Hat in
>>> the past few years. SIGH.
>>>
>>> Downgrading my CentOS is not going to be easy. There were security
>>> patches in the latest CentOS that I am required to have. It would
>>> actually be easier to just switch to Ubuntu at this point since we're
>>> talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
>>> I hope IBM fixes them, but I have my doubts.
>>>
>>> On 5/22/2019 1:05 AM, Andrija Panic wrote:
>>>> Hi Eric, all,
>>>>
>>>> I believe you might be hitting this one - issues on latest CentOS 7.6:
>>>> https://github.com/apache/cloudstack/pull/3333 due to changes in the OS
>>>> itself...
>>>>
>>>> If you believe that is the case, please try with CentOS 7.4 (can confirm
>>>> works fine) and/or CentOS 7.5.
>>>>
>>>> Best,
>>>> Andrija
>>>>
>>>>
>>>>
>>>> On Wed, 22 May 2019 at 09:04, Eric Lee Green <eric.lee.green@gmail.com>
>>>> wrote:
>>>>
>>>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
>>>>>> Hello Eric,
>>>>>>
>>>>>>       If the router version is displayed as UNKNOWN in the portal,
it
>>>>> would be
>>>>>> a connectivity issue. Check your connections related to firewall
rules
>>>>>> between the ACP management hosts, hypervisor and VR.  Is your VR
>>>>> management
>>>>>> network setup correctly?
>>>>> Communications between the hosts is working fine and I have not changed
>>>>> the VR management network between the running 4.9 and the not-running
>>>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
>>>>> should properly forward VR traffic.
>>>>>
>>>>> More tomorrow when I have some sleep since it is now midnight here in
>>>>> the SF Bay area.
>>>>>
>>>>>
>>>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <
>> eric.lee.green@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the response, sorry if I sound frustrated, but this
is
>>>>>>> supposed to be a simple easy process and it's been horrible all
the
>> way
>>>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and
now
>> 4.11.2
>>>>>>> has failed to upgrade too. I've thus far spent around 16 hours
of my
>>>>>>> time on what should have taken an hour at most. I'm frustrated
and
>>>>> bummed.
>>>>>>> [root@cloud2 ~]# rpm -q centos-release
>>>>>>> centos-release-7-6.1810.2.el7.centos.x86_64
>>>>>>> [root@cloud2 ~]# rpm -q libvirt
>>>>>>> libvirt-4.5.0-10.el7_6.9.x86_64
>>>>>>> [root@cloud1 ~]# rpm -qa | grep kvm
>>>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
>>>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
>>>>>>> [root@cloud2 ~]# cat /proc/version
>>>>>>> Linux version 3.10.0-862.6.3.el7.x86_64
>>>>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623
(Red
>> Hat
>>>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
>>>>>>> [root@cloud2 ~]# cat /proc/cpuinfo
>>>>>>> ...
>>>>>>> processor       : 23
>>>>>>> vendor_id       : GenuineIntel
>>>>>>> cpu family      : 6
>>>>>>> model           : 44
>>>>>>> model name      : Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz
>>>>>>> stepping        : 2
>>>>>>> microcode       : 0x1f
>>>>>>> cpu MHz         : 3068.000
>>>>>>> cache size      : 12288 KB
>>>>>>> physical id     : 1
>>>>>>> siblings        : 12
>>>>>>> core id         : 10
>>>>>>> cpu cores       : 6
>>>>>>> apicid          : 53
>>>>>>> initial apicid  : 53
>>>>>>> fpu             : yes
>>>>>>> fpu_exception   : yes
>>>>>>> cpuid level     : 11
>>>>>>> wp              : yes
>>>>>>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep
mtrr
>> pge
>>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
pbe
>>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
>> rep_good
>>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor
>> ds_cpl
>>>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
>>>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid
>> dtherm
>>>>>>> ida arat
>>>>>>> bogomips        : 6133.21
>>>>>>> clflush size    : 64
>>>>>>> cache_alignment : 64
>>>>>>> address sizes   : 40 bits physical, 48 bits virtual
>>>>>>> [root@cloud1 ~]# free
>>>>>>>                   total        used        free      shared buff/cache
>>>>>>> available
>>>>>>> Mem:      123596388    79795792    34047696      632116 9752900
>>>>> 39999332
>>>>>>> Swap:       4194300     1956824     2237476
>>>>>>> [root@cloud1 ~]# yum update
>>>>>>>
>>>>>>>
>> =========================================================================================================================================================================
>>>>>>>      Package Arch Version Repository                        
    Size
>>>>>>>
>>>>>>>
>> =========================================================================================================================================================================
>>>>>>> Updating:
>>>>>>>      cloudstack-agent x86_64 4.11.2.0-1.el7.centos
>>>>>>> cloudstack411                          47 M
>>>>>>>      cloudstack-common x86_64 4.11.2.0-1.el7.centos
>>>>>>> cloudstack411                          58 M
>>>>>>>      cloudstack-management x86_64 4.11.2.0-1.el7.centos
>>>>>>> cloudstack411                          82 M
>>>>>>>
>>>>>>> Three compute servers running the above processor averaging 128gb
of
>>>>>>> memory apiece, two data servers running NFS for primary and secondary
>>>>>>> storage. Running the 4.11.2 community RPM's above.
>>>>>>>
>>>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating.
The
>>>>>>> routers in fact started with the template, it said 4.11.2 when
I
>> opened
>>>>>>> up the console window and looked at the login prompt. But they
never
>> got
>>>>>>> configured, when I logged in with root/password at the console
prompt
>>>>>>> they had no networking set up and no configuration provided,
the
>>>>>>> cloud-init output in /var/log was essentially blank.
>>>>>>>
>>>>>>> Do I recall correctly that there is an issue with a particular
>> version
>>>>>>> of the hypervisor on the latest Centos 7? Any other information
that
>> you
>>>>>>> need? I think I provided the complete log file for the cloud-init
of
>> the
>>>>>>> router in another email...
>>>>>>>
>>>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> Can you describe your environment in detail such as management
>> server
>>>>>>> host distro, hypervisor version, current CloudStack version,
did you
>>>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Rohit Yadav
>>>>>>>>
>>>>>>>> ________________________________
>>>>>>>> From: Eric Lee Green <eric.lee.green@gmail.com>
>>>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>>>>>>>>
>>>>>>>> You may remember me as the person who had to roll back to
Cloudstack
>>>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual
machines
>>>>> once
>>>>>>>> I upgraded to it, claiming that there were inadequate resources
even
>>>>>>>> though I had over 150 gigabytes of memory free in my cluster
and
>> oodles
>>>>>>>> of CPU free (and a minimum of 40gb on each node, plenty to
start a
>>>>> 512mb
>>>>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2
and
>>>>>>>> *again* it's misbehaving.
>>>>>>>>
>>>>>>>> The symptom is that my virtual routers when I log into their
console
>>>>>>>> show 4.11.2 but when I look at them in the console they say
>> 'Version:
>>>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address
or link
>>>>>>>> local IP address it fails. And when I try to start up a virtual
>> machine
>>>>>>>> that uses that virtual network, it says "Network unavailable',
even
>>>>>>>> though the router for that network is showing up and running.
>>>>>>>>
>>>>>>>> Clearly something's broken in the virtual routers but I don't
know
>> what
>>>>>>>> because I can't get into the router virtual machines. How
do I get
>> the
>>>>>>>> console password to get into the router virtual machines?
It's
>>>>> encrypted
>>>>>>>> in the database (duh), how do I decrypt it?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> rohit.yadav@shapeblue.com
>>>>>>>> www.shapeblue.com
>>>>>>>> Amadeus House, Floral Street, London  WC2E 9DPUK
>>>>>>>> @shapeblue
>>>>>>>>
>>>>>>>>
>>>>>>>>
>


Mime
View raw message