cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Date Wed, 22 May 2019 19:52:19 GMT
Important to see you happy with 4.11.2 - the rest (HW) will follow ;)

On Wed, 22 May 2019 at 21:49, Eric Lee Green <eric.lee.green@gmail.com>
wrote:

> EXCELLENT!
>
> That did it. I downgraded to the libvirt and openssh rpms from Centos
> 7.5, which was the latest DVD I had hanging around (likely the one I
> used to install the systems with in the first place). Turns out it was
> the Red Hat updates that came in when I did 'rpm update' that broke
> things, not the Cloudstack upgrade. I am now happily running on
> Cloudstack 4.11.2. Got rid of the qemu-kvm-ev stuff, btw, I had
> installed that at one point trying to resolve the problems, but have now
> downgraded it back to the release stuff.
>
> In case it is not obvious, I am the person who is serious about security
> policies. The company itself doesn't care as long as we have one to show
> to investors and customers.
>
> Regarding hardware, security policies don't cost a lot of money given
> all the open source tools we have available now. New hardware does cost
> money. Old commercial grade hardware is ridiculously cheap from Fleabay
> right now as big corporations dump "obsolete" gear.  I bought three used
> commercial-grade 24-port 10 gigabit switches for the price of a single
> new 1 gigabit switch, for example. I won't have cheap white box junk in
> the back room, it's three racks of commercial-quality gear with dual
> processors, ECC, 10gbit Ethernet, IPMI for remote management, etc.,
> reliable as a brick -- just old and a bit power hungry. Power is the
> biggest issue that will lead to an upgrade, not reliability or
> performance -- I only have power budget for three racks, and new
> hardware is much more dense for a specific power consumption (modern 16
> core processors use about the same power as the 6 core processors I have
> in my nodes). When I have the budget to upgrade the hardware, I will,
> but what's there works, is reliable as a brick and not an issue, unlike
> Red Hat breaking my infrastructure with incompatible updates GRR!
>
> On 5/22/19 9:20 AM, Andrija Panic wrote:
> > +1 on what Nux said - simply downgrade the packages and check if that
> works
> > - no need to reinstall everything !
> > As for the $3333 from github (CentOS 7.6 issue) - simple new line at the
> > end of id_rsa file and/or downgrading the ssh-keygen will work.
> >
> > Don't want to annoy you, but if the company is so serious about security
> > policies ask them to be serious about HW as well (regular white trash PC
> > will do...so you can test software itself) - otherwise, you personally
> > suffer the consequences and look bad  (had the same pain many years
> ago...).
> >
> > On Wed, 22 May 2019 at 17:44, Nux! <nux@li.nux.ro> wrote:
> >
> >> Eric,
> >>
> >> I think you can get away with just downgrading to stock qemu-kvm
> packages
> >> (or patch your cloudstack installation).
> >> BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages
> >> don't get the same level of testing and QA.
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >> ----- Original Message -----
> >>> From: "Eric Lee Green" <eric.lee.green@gmail.com>
> >>> To: "users" <users@cloudstack.apache.org>
> >>> Sent: Wednesday, 22 May, 2019 15:33:15
> >>> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> >>> Okay. This makes sense.
> >>>
> >>> And people wonder why Amazon decided to make their own Linux rather
> than
> >>> use Centos and why Ubuntu has seized huge market share from Red Hat in
> >>> the past few years. SIGH.
> >>>
> >>> Downgrading my CentOS is not going to be easy. There were security
> >>> patches in the latest CentOS that I am required to have. It would
> >>> actually be easier to just switch to Ubuntu at this point since we're
> >>> talking a complete re-install anyhow. SIGH. Thank you, Red Hat
> Software.
> >>> I hope IBM fixes them, but I have my doubts.
> >>>
> >>> On 5/22/2019 1:05 AM, Andrija Panic wrote:
> >>>> Hi Eric, all,
> >>>>
> >>>> I believe you might be hitting this one - issues on latest CentOS 7.6:
> >>>> https://github.com/apache/cloudstack/pull/3333 due to changes in the
> OS
> >>>> itself...
> >>>>
> >>>> If you believe that is the case, please try with CentOS 7.4 (can
> confirm
> >>>> works fine) and/or CentOS 7.5.
> >>>>
> >>>> Best,
> >>>> Andrija
> >>>>
> >>>>
> >>>>
> >>>> On Wed, 22 May 2019 at 09:04, Eric Lee Green <
> eric.lee.green@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
> >>>>>> Hello Eric,
> >>>>>>
> >>>>>>       If the router version is displayed as UNKNOWN in the portal,
> it
> >>>>> would be
> >>>>>> a connectivity issue. Check your connections related to firewall
> rules
> >>>>>> between the ACP management hosts, hypervisor and VR.  Is your
VR
> >>>>> management
> >>>>>> network setup correctly?
> >>>>> Communications between the hosts is working fine and I have not
> changed
> >>>>> the VR management network between the running 4.9 and the not-running
> >>>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack
> and
> >>>>> should properly forward VR traffic.
> >>>>>
> >>>>> More tomorrow when I have some sleep since it is now midnight here
in
> >>>>> the SF Bay area.
> >>>>>
> >>>>>
> >>>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <
> >> eric.lee.green@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for the response, sorry if I sound frustrated, but
this is
> >>>>>>> supposed to be a simple easy process and it's been horrible
all the
> >> way
> >>>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9,
and now
> >> 4.11.2
> >>>>>>> has failed to upgrade too. I've thus far spent around 16
hours of
> my
> >>>>>>> time on what should have taken an hour at most. I'm frustrated
and
> >>>>> bummed.
> >>>>>>> [root@cloud2 ~]# rpm -q centos-release
> >>>>>>> centos-release-7-6.1810.2.el7.centos.x86_64
> >>>>>>> [root@cloud2 ~]# rpm -q libvirt
> >>>>>>> libvirt-4.5.0-10.el7_6.9.x86_64
> >>>>>>> [root@cloud1 ~]# rpm -qa | grep kvm
> >>>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
> >>>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
> >>>>>>> [root@cloud2 ~]# cat /proc/version
> >>>>>>> Linux version 3.10.0-862.6.3.el7.x86_64
> >>>>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623
(Red
> >> Hat
> >>>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
> >>>>>>> [root@cloud2 ~]# cat /proc/cpuinfo
> >>>>>>> ...
> >>>>>>> processor       : 23
> >>>>>>> vendor_id       : GenuineIntel
> >>>>>>> cpu family      : 6
> >>>>>>> model           : 44
> >>>>>>> model name      : Intel(R) Xeon(R) CPU           X5675 
@ 3.07GHz
> >>>>>>> stepping        : 2
> >>>>>>> microcode       : 0x1f
> >>>>>>> cpu MHz         : 3068.000
> >>>>>>> cache size      : 12288 KB
> >>>>>>> physical id     : 1
> >>>>>>> siblings        : 12
> >>>>>>> core id         : 10
> >>>>>>> cpu cores       : 6
> >>>>>>> apicid          : 53
> >>>>>>> initial apicid  : 53
> >>>>>>> fpu             : yes
> >>>>>>> fpu_exception   : yes
> >>>>>>> cpuid level     : 11
> >>>>>>> wp              : yes
> >>>>>>> flags           : fpu vme de pse tsc msr pae mce cx8 apic
sep mtrr
> >> pge
> >>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss
ht tm pbe
> >>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs
bts
> >> rep_good
> >>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64
monitor
> >> ds_cpl
> >>>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2
popcnt
> >>>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority
ept vpid
> >> dtherm
> >>>>>>> ida arat
> >>>>>>> bogomips        : 6133.21
> >>>>>>> clflush size    : 64
> >>>>>>> cache_alignment : 64
> >>>>>>> address sizes   : 40 bits physical, 48 bits virtual
> >>>>>>> [root@cloud1 ~]# free
> >>>>>>>                   total        used        free      shared
> buff/cache
> >>>>>>> available
> >>>>>>> Mem:      123596388    79795792    34047696      632116
9752900
> >>>>> 39999332
> >>>>>>> Swap:       4194300     1956824     2237476
> >>>>>>> [root@cloud1 ~]# yum update
> >>>>>>>
> >>>>>>>
> >>
> =========================================================================================================================================================================
> >>>>>>>      Package Arch Version Repository
>  Size
> >>>>>>>
> >>>>>>>
> >>
> =========================================================================================================================================================================
> >>>>>>> Updating:
> >>>>>>>      cloudstack-agent x86_64 4.11.2.0-1.el7.centos
> >>>>>>> cloudstack411                          47 M
> >>>>>>>      cloudstack-common x86_64 4.11.2.0-1.el7.centos
> >>>>>>> cloudstack411                          58 M
> >>>>>>>      cloudstack-management x86_64 4.11.2.0-1.el7.centos
> >>>>>>> cloudstack411                          82 M
> >>>>>>>
> >>>>>>> Three compute servers running the above processor averaging
128gb
> of
> >>>>>>> memory apiece, two data servers running NFS for primary
and
> secondary
> >>>>>>> storage. Running the 4.11.2 community RPM's above.
> >>>>>>>
> >>>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating.
> The
> >>>>>>> routers in fact started with the template, it said 4.11.2
when I
> >> opened
> >>>>>>> up the console window and looked at the login prompt. But
they
> never
> >> got
> >>>>>>> configured, when I logged in with root/password at the console
> prompt
> >>>>>>> they had no networking set up and no configuration provided,
the
> >>>>>>> cloud-init output in /var/log was essentially blank.
> >>>>>>>
> >>>>>>> Do I recall correctly that there is an issue with a particular
> >> version
> >>>>>>> of the hypervisor on the latest Centos 7? Any other information
> that
> >> you
> >>>>>>> need? I think I provided the complete log file for the cloud-init
> of
> >> the
> >>>>>>> router in another email...
> >>>>>>>
> >>>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> >>>>>>>> Hi Eric,
> >>>>>>>>
> >>>>>>>> Can you describe your environment in detail such as
management
> >> server
> >>>>>>> host distro, hypervisor version, current CloudStack version,
did
> you
> >>>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Rohit Yadav
> >>>>>>>>
> >>>>>>>> ________________________________
> >>>>>>>> From: Eric Lee Green <eric.lee.green@gmail.com>
> >>>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> >>>>>>>>
> >>>>>>>> You may remember me as the person who had to roll back
to
> Cloudstack
> >>>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual
> machines
> >>>>> once
> >>>>>>>> I upgraded to it, claiming that there were inadequate
resources
> even
> >>>>>>>> though I had over 150 gigabytes of memory free in my
cluster and
> >> oodles
> >>>>>>>> of CPU free (and a minimum of 40gb on each node, plenty
to start a
> >>>>> 512mb
> >>>>>>>> router VM). So now I'm trying to upgrade to Cloudstack
4.11.2 and
> >>>>>>>> *again* it's misbehaving.
> >>>>>>>>
> >>>>>>>> The symptom is that my virtual routers when I log into
their
> console
> >>>>>>>> show 4.11.2 but when I look at them in the console they
say
> >> 'Version:
> >>>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP
address or
> link
> >>>>>>>> local IP address it fails. And when I try to start up
a virtual
> >> machine
> >>>>>>>> that uses that virtual network, it says "Network unavailable',
> even
> >>>>>>>> though the router for that network is showing up and
running.
> >>>>>>>>
> >>>>>>>> Clearly something's broken in the virtual routers but
I don't know
> >> what
> >>>>>>>> because I can't get into the router virtual machines.
How do I get
> >> the
> >>>>>>>> console password to get into the router virtual machines?
It's
> >>>>> encrypted
> >>>>>>>> in the database (duh), how do I decrypt it?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> rohit.yadav@shapeblue.com
> >>>>>>>> www.shapeblue.com
> >>>>>>>> Amadeus House, Floral Street, London  WC2E 9DPUK
> >>>>>>>> @shapeblue
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >
>
>

-- 

Andrija Panić

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message