cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Weingärtner <rafaelweingart...@gmail.com>
Subject Re: Cloudstack 4.7 password reset issue - resolved.
Date Fri, 15 Apr 2016 18:24:29 GMT
Those are great improvements, please create the PRs that we will review and
help you prepare the code to be merged. Just be aware that we are going to
have a feature freeze starting next Monday (if I am not mistaken) until the
4.9 version is closed.

On Fri, Apr 15, 2016 at 3:12 PM, Richard Klein (RSI) <rklein@rsitex.com>
wrote:

> Thanks for the replay and I like the 3 PR as well.  The changes are
> related to the environment we are using.  We are using CS 4.7.0 on CentOS7
> and VXLAN with a two 10Mb NICs per host that is bonded via multi-stack LACP
> switch for performance and redundancy.  We ran into the following 3 issues
> with this configuration.
>
> 1) The bond0 interface had 2 bridge interfaces.  The "cloudbr0" is bridged
> to bond0.  It's used for management and VLAN related guest networks. We
> were limited to 1500 MTU in order to be compatible with our infrastructure
> switches.   The "cloudbr1" is bridge to "bond0.xxxx" where "xxxx" is an
> internal VLAN ID that is restricted to Gluster and VXLAN traffic as well as
> using jumbo frames.  The problem was with the logic in the "modifyvxlan.sh"
> script.  When creating a VXLAN isolated network it would always use the
> "cloudbr0" interface even though the traffic label was defined as
> "cloudbr1".  This was due to how the physical interface search was being
> performed.  There was even a "TODO" in the script that suggesting passing
> the traffic label instead of doing a search.  So I modified the Java back
> end code and the "modifyvslan.sh" script to pass the traffic label and this
> corrected the problem and used the "cloudbr1" interface.
>
> 2) We discovered a very odd problem when migrating instances that use
> VXLAN.  For some reason we always got an error like "brbond0" not found.  I
> can look up the exact verbiage but basically it was not able to create the
> proper VXLAN bridges and such on the target host because it couldn't find
> the physical interface of "brbond0" which is true.  The bridge interface
> should have been "brvx-xxxxx".  This turned out to be a problem with
> libvirt and not cloudstack.  You could replicate the problem completely
> isolated from the Cloudstack environment.  For some reason I still can't
> explain, libvirt would always fail migrating a VXLAN.  Oddly enough this
> only happened if the VXLAN bridge interface created by CS started with
> "brvx-xxxx" (-xxxx is the VXLAN ID). Even if manually created and migrated
> (outside of CS control) we found that this would happen so it behaved as
> bug in libvirt not CS.  The solution was quite simple.  We change the code
> that builds the VXLAN bridges to use "vxbr-xxxx".  I really can't explain
> why but the name changed resolved the issue.
>
> 3) This last one is the password change issue we've been discussing which
> as you know was resolved by changing the "configure.py" script.
>
> I will try a pull request on number 3 first.  This is the simplest way for
> me to get use to the process.  I will then follow up on the others as time
> permits.  We are about to implement CS in a production environment so
> hopefully I can get to the PR soon.
>
>
> Richard Klein  <rklein@rsitex.com>
> RSI
> 5426 Guadalupe, Suite 100
> Austin TX 78751
>
>
>
>
>
> > -----Original Message-----
> > From: Rafael Weingärtner [mailto:rafaelweingartner@gmail.com]
> > Sent: Friday, April 15, 2016 12:02 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> >
> > Would it be possible for you to explain a little bit these changes?
> >
> > I believe a PR per change would be the best way to go.
> >
> > On Fri, Apr 15, 2016 at 1:31 PM, Richard Klein (RSI) <rklein@rsitex.com>
> > wrote:
> >
> > > I would be happy to submit a pull request but I am relatively new to
> > > using Git and GitHub.  I have a lot of experience with SVN and CVS.  I
> > > have read the following link about the process:
> > >
> > >    * https://cloudstack.apache.org/developers.html
> > >    * https://help.github.com/articles/creating-a-pull-request/
> > >
> > > I have forked the apache/cloudstack on GitHub and have been making
> > > changes to the 4.7.0 version on a separate branch.  This branch
> > > contains several code changes we had to make in order to get CS to run
> in
> > our environment.
> > > Since I am not familiar with Mavin I have created some non-standard
> > > version numbers in order to distinguish the RPMs and use a private
> > > repository so we can control the upgrade process.
> > >
> > > I see 2 options on submitting a pull request.  First is to submit it
> > > from the existing branch that contains all the modified code we've
> > > made to 4.7.0.  The only downside is it contains a lot of "pom.xml"
> > > version number changes as well.  The Second option is to create a
> > > branch for each of the 3 types of fixes we have made and do a pull
> request
> > for each one.
> > >
> > > Let me know if there are any additional resources I need to read up on
> > > and the proper method of submitting a pull request.
> > >
> > > Thanks!
> > >
> > > Richard Klein  <rklein@rsitex.com>
> > > RSI
> > > 5426 Guadalupe, Suite 100
> > > Austin TX 78751
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Remi Bergsma [mailto:RBergsma@schubergphilis.com]
> > > > Sent: Thursday, April 14, 2016 3:35 PM
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> > > >
> > > > Hi Richard,
> > > >
> > > > Great you fixed it! Can you send the patch of your fix as a spul
> > > > request
> > > on
> > > > github?
> > > >
> > > > Required upgrade is yes when the router reports a version older than
> > > > the minreq.sysvm.version (or similar) global setting. It's used to
> > > > upgrade
> > > systemvm
> > > > templates.
> > > >
> > > > Regards, Remi
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 13 Apr 2016, at 22:23, Richard Klein (RSI) <rklein@rsitex.com>
> > > wrote:
> > > > >
> > > > > I finally found the problem and resolved the issue.  The problem
> > > > > was
> > > in the
> > > > Python code change I made.  I had a flag variable that indicated to
> > > > save
> > > data
> > > > when it was changed while processing a list.  This worked fine as
> > > > long
> > > as it
> > > > executed the logic and defined the flag variable.  The problem was
> > > > during startup when it doesn't go through the loop and the flag
> > > > variable was undefined.  This cause the "update_config.py" to fail
> > > > which then bubbled
> > > back
> > > > up as an error and prevent the router from starting.
> > > > >
> > > > > Once I changed the code and rebuilt the project all worked well
> > > > > and
> > > the bug
> > > > is fixed.  Thanks so much for everyone's help.  This process was
> > > > very educational and looking forward to learning more.
> > > > >
> > > > > I do have one question just out of curiosity.  What makes the
> > > > > "Requires
> > > > Upgrade" column on the Home->Infrastructure->Virtual Router page
> > > indicated
> > > > "Yes"?
> > > > >
> > > > > Thanks again,
> > > > >
> > > > >
> > > > > Richard Klein  <rklein@rsitex.com> RSI
> > > > > 5426 Guadalupe, Suite 100
> > > > > Austin TX 78751
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >> -----Original Message-----
> > > > >> From: Rajani Karuturi [mailto:rajani@apache.org]
> > > > >> Sent: Tuesday, April 12, 2016 6:15 AM
> > > > >> To: users@cloudstack.apache.org
> > > > >> Subject: Re: Cloudstack 4.7 password reset issue.
> > > > >>
> > > > >> Can you check the vm_instance table for the VR entry and update
> > > > >> vm_template_id?
> > > > >>
> > > > >> This might be helpful
> > > > >> https://gist.github.com/terbolous/102ae8edd1cda192561c
> > > > >>
> > > > >> ~Rajani
> > > > >>
> > > > >> On Sat, Apr 9, 2016 at 5:45 AM, Richard Klein (RSI)
> > > > >> <rklein@rsitex.com>
> > > > >> wrote:
> > > > >>
> > > > >>> I found the password reset issue and it ended up being a
Python
> > > > >>> script on the VR.  I ended up modifying the
> > > > >>> "/opt/cloud/bin/configure.py" to resolve the issue.  Basically
> there
> > > is a
> > > > "/etc/cloud/vmpassword.json"
> > > > >>> file that is updated with the IP/password pair when the GUI
> password
> > > > >>> change is performed.  During the power on process the VM
> > > > >>> configuration info is sent to the router which reads the
> > > > >>> vmpassword.json file and sends the password changes to the
> password
> > > > >>> server cache file.  When the client retrieved the password
it was
> > > > >>> cleared from the password cache file but not the vmpassword.json
> > > > >>> file.  So every time a VM started the last password reset
was
> sent
> > > to the
> > > > password server again.
> > > > >>>
> > > > >>> The question I have now is how do I get the system VM template
> > > > >>> updated with the change?  Since we are using CS v4.7 we used
the
> > > > >>> system template for v4.6 per the installation instructions
for
> > > > >>> CentOS7 and KVM.  I performed the following steps to use
a new
> > system
> > > > VM template:
> > > > >>>
> > > > >>> * I copied the system vm template QCOW2 file from secondary
> storage
> > > > >>> to a work server and made a backup of it.
> > > > >>> * On the work server I mounted the QCOW2 template file using
> > > > >> "guestmount"
> > > > >>> tools and made the code changes to the template.
> > > > >>> * I then copied this modified template file to a web server
and
> > > > >>> registered the template in cloudstack with all checkboxes
off
> except
> > > > >>> for
> > > > >> "routing".
> > > > >>> * Then we set the cloudstack global value of
> "router.template.kvm"
> > > > >>> to the name of the new template.
> > > > >>> * The management services were restarted.
> > > > >>> * I picked a test VR, powered it off, destroyed it then let
the
> > > > >>> system recreate it.
> > > > >>> * When I look at the code I changed on the new VR it does
not
> appear.
> > > > >>>
> > > > >>> I even doubled checked the database and the vm_instance table
for
> > > > >>> the test VR showed the new template ID.  I must be missing
> something
> > > > >>> or I don't really understand how the system templates are
> created.
> > > > >>> Any help/suggestions would be appreciated.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> Richard Klein  <rklein@rsitex.com>
> > > > >>> RSI
> > > > >>> 5426 Guadalupe, Suite 100
> > > > >>> Austin TX 78751
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>> -----Original Message-----
> > > > >>>> From: Richard Klein (RSI)
> > > > >>>> Sent: Tuesday, April 05, 2016 2:32 PM
> > > > >>>> To: users@cloudstack.apache.org
> > > > >>>> Subject: RE: Cloudstack 4.7 password reset issue.
> > > > >>>>
> > > > >>>> The snippets for before and after the reboot via console
look
> the
> > > > >>>> same
> > > > >>> so I
> > > > >>>> pasted the 2nd set of message instead of the first. 
Sorry about
> > > that.
> > > > >>> I did
> > > > >>>> discover that the /var/lib/dhclient/dhclient.leases existed
but
> was
> > > > >>> empty.  I've
> > > > >>>> run across an issue with CentOS 7 where the lease file
is
> missing
> > > > >>>> so I
> > > > >>> wrote a
> > > > >>>> "cloud-dhcp-check" service that makes sure it exists
but now I
> need
> > > > >>>> to
> > > > >>> validate
> > > > >>>> its content.  That being said, I have insured that the
> > > > >>>> dhclient_leases
> > > > >>> was valid
> > > > >>>> and replicated the problem.
> > > > >>>>
> > > > >>>> The cloud-set-guest-xxxx scripts are from the master
branch
> GitHub
> > > > >>> repository
> > > > >>>> for apaches/cloudstack using the
> > > > >>>> "
> > > > >>>
> https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-
> > > > >>> se
> > > > >>> t-
> > > > >>>> guest-password.in" and the
> > > > >>>> "
> > > > >>>
> https://github.com/apache/cloudstack/blob/master/setup/bindir/cloud-
> > > > >>> se
> > > > >>> t-
> > > > >>>> guest-sshkey.in" links.
> > > > >>>>
> > > > >>>> I have attached the entire log from the VR but have some
> snippets
> > > > >>>> below
> > > > >>> along
> > > > >>>> with the VM client logs and the issue still occurs after
fixing
> the
> > > > >>>> dhcp
> > > > >>> lease file.
> > > > >>>> I did not perform any password resets via the GUI during
this
> > > process.
> > > > >>>
> > > > >>>
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
>



-- 
Rafael Weingärtner

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message