Hi Rohit,
I tryed renew certificate but it failed!
Now libvirt does not restart and agent is disconnected:
agent log:
2019-01-31 11:17:07,530 INFO
[resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper]
(Certificate Renewal Timer:null) (logid:fe1554cc) Restarting libvirt
after certificate provisioning/renewal
2019-01-31 11:17:07,567 INFO [cloud.agent.Agent]
(AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2019-01-31 11:17:07,568 WARN [cloud.agent.Agent] (Certificate Renewal
Timer:null) (logid:fe1554cc) Failed to execute post certificate renewal
command:
java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
at
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157)
at
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
2019-01-31 11:17:09,797 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Agent started
2019-01-31 11:17:09,800 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Implementation Version is 4.11.2.0
2019-01-31 11:17:09,802 INFO [cloud.agent.AgentShell] (main:null)
(logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2019-01-31 11:17:09,815 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Defaulting to using properties file for storage
2019-01-31 11:17:09,816 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Defaulting to the constant time backoff algorithm
2019-01-31 11:17:09,828 INFO [cloud.utils.LogUtils] (main:null)
(logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2019-01-31 11:17:09,850 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Using default Java settings for IPv6 preference for agent
connection
2019-01-31 11:17:09,998 INFO [cloud.agent.Agent] (main:null) (logid:)
id is 5
2019-01-31 11:17:10,030 INFO [kvm.resource.LibvirtConnection]
(main:null) (logid:) No existing libvirtd connection found. Opening a
new one
2019-01-31 11:17:10,175 ERROR [cloud.agent.AgentShell] (main:null)
(logid:) Unable to start agent:
com.cloud.utils.exception.CloudRuntimeException: Failed to connect
socket to '/var/run/libvirt/libvirt-sock': No such file or directory
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914)
at com.cloud.agent.Agent.<init>(Agent.java:190)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
(logs repeat)
syslog:
Jan 31 11:17:07 cshp214 sh[5065]: INFO
[resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper]
(Certificate Renewal Timer:) (logid:fe1554cc) Restarting libvirt after
certificate provisioning/renewal
Jan 31 11:17:07 cshp214 systemd[1]: Stopping CloudStack Agent...
Jan 31 11:17:07 cshp214 sh[5065]: INFO [cloud.agent.Agent]
(AgentShutdownThread:) (logid:) Stopping the agent: Reason = sig.kill
Jan 31 11:17:07 cshp214 sh[5065]: WARN [cloud.agent.Agent] (Certificate
Renewal Timer:) (logid:fe1554cc) Failed to execute post certificate
renewal command:
Jan 31 11:17:07 cshp214 sh[5065]: java.lang.IllegalStateException:
Shutdown in progress
Jan 31 11:17:07 cshp214 sh[5065]: #011at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
java.lang.Runtime.removeShutdownHook(Runtime.java:239)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
java.util.TimerThread.mainLoop(Timer.java:555)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
java.util.TimerThread.run(Timer.java:505)
Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading data:
Input/output error
Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading data:
Input/output error
Jan 31 11:17:08 cshp214 systemd[1]: Stopped CloudStack Agent.
Jan 31 11:17:08 cshp214 systemd[1]: Stopping Virtualization daemon...
Jan 31 11:17:08 cshp214 systemd[1]: Stopped Virtualization daemon.
Jan 31 11:17:08 cshp214 systemd[1]: Starting Virtualization daemon...
Jan 31 11:17:08 cshp214 systemd[1]: Started Virtualization daemon.
Jan 31 11:17:08 cshp214 systemd[1]: Started CloudStack Agent.
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN No appenders could be
found for logger (com.cloud.agent.AgentShell).
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN Please initialize the
log4j system properly.
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Agent started
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Implementation Version is 4.11.2.0
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) agent.properties found at
/etc/cloudstack/agent/agent.properties
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Defaulting to using properties file for storage
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Defaulting to the constant time backoff algorithm
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.utils.LogUtils] (main:)
(logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Using default Java settings for IPv6 preference for
agent connection
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.Agent] (main:)
(logid:) id is 5
Jan 31 11:17:10 cshp214 sh[25387]: INFO [kvm.resource.LibvirtConnection]
(main:) (logid:) No existing libvirtd connection found. Opening a new one
--
Jan 31 11:17:16 cshp214 snmpd[2460]: error on subcontainer 'ia_addr'
insert (-1)
Jan 31 11:17:16 cshp214 snmpd[2460]: message repeated 3 times: [ error
on subcontainer 'ia_addr' insert (-1)]
Jan 31 11:17:20 cshp214 systemd[1]: cloudstack-agent.service: Service
hold-off time over, scheduling restart.
Jan 31 11:17:20 cshp214 systemd[1]: Stopped CloudStack Agent.
Jan 31 11:17:20 cshp214 systemd[1]: Started CloudStack Agent.
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN No appenders could be
found for logger (com.cloud.agent.AgentShell).
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN Please initialize the
log4j system properly.
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Agent started
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Implementation Version is 4.11.2.0
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) agent.properties found at
/etc/cloudstack/agent/agent.properties
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Defaulting to using properties file for storage
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Defaulting to the constant time backoff algorithm
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.utils.LogUtils] (main:)
(logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Using default Java settings for IPv6 preference for
agent connection
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.Agent] (main:)
(logid:) id is 5
Jan 31 11:17:21 cshp214 sh[25457]: INFO [kvm.resource.LibvirtConnection]
(main:) (logid:) No existing libvirtd connection found. Opening a new one
Jan 31 11:17:21 cshp214 sh[25457]: libvirt: XML-RPC error : Failed to
connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Jan 31 11:17:21 cshp214 sh[25457]: ERROR [cloud.agent.AgentShell]
(main:) (logid:) Unable to start agent:
Jan 31 11:17:21 cshp214 sh[25457]:
com.cloud.utils.exception.CloudRuntimeException: Failed to connect
socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.Agent.<init>(Agent.java:190)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.start(AgentShell.java:512)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.main(AgentShell.java:547)
Jan 31 11:17:21 cshp214 sh[25457]: Unable to start agent: Failed to
connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Main
process exited, code=exited, status=67/n/a
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Unit
entered failed state.
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Failed
with result 'exit-code'.
Jan 31 11:17:21 cshp214 dnsmasq[4000]: read /etc/hosts - 13 addresses
Jan 31 11:17:21 cshp214 dnsmasq[4000]: read
/var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jan 31 11:17:21 cshp214 dnsmasq-dhcp[4000]: read
/var/lib/libvirt/dnsmasq/default.hostsfile
Jan 31 11:17:22 cshp214 snmpd[2460]: Connection from UDP:
[127.0.0.1]:37699->[127.0.0.1]:161
Jan 31 11:17:24 cshp214 snmpd[2460]: message repeated 2 times: [
Connection from UDP: [127.0.0.1]:37699->[127.0.0.1]:161]
Jan 31 11:17:24 cshp214 libvirtd[25368]: libvirt version: 1.3.1,
package: 1ubuntu10.24 (Marc Deslauriers <marc.deslauriers@ubuntu.com>
Wed, 23 May 2018 13:29:29 -0400)
Jan 31 11:17:24 cshp214 libvirtd[25368]: hostname: cshp214
Jan 31 11:17:24 cshp214 libvirtd[25368]: Configured security driver
"none" disables default policy to create confined guests
Jan 31 11:17:25 cshp214 libvirtd[25368]: unsupported configuration:
Security driver apparmor not enabled
Can anyone help me?
Il 30/01/19 13:37, Rohit Yadav ha scritto:
>
> Hi Ugo,
>
>
> This will be a one-time procedure, and the KVM host and the VMs do not
> need a reboot but the provisionCertificate API will restart the
> libvirtd process (just check if that can have any side effects for
> your VMs/distro, on most modern distros restarting libvirtd does not
> have any side-effects on existing running VMs).
>
>
> - Rohit
>
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> @shapeblue
>
> ------------------------------------------------------------------------
> *From:* Ugo Vasi <ugo.vasi@procne.it>
> *Sent:* Wednesday, January 30, 2019 4:47:09 PM
> *To:* users@cloudstack.apache.org; Rohit Yadav
> *Subject:* Re: secure hosts communications
> Hi Rohit,
> I have a 4.11.2.0 ACS infrastructure (Ubuntu 16.04 with KVM hypervisor)
> I see that all the hosts are in unsecure state from the UI and so the
> live migration don't works (we had trubles with mgmt server).
>
> I read in the documentation that launching the provisionCertificate API
> (by pressing the appropriate button in the UI) the certificates will be
> renewed/regenerated for already connected agents/hosts.
>
> I do not understand if provisioning should be done manually on each host
> or if the procedure should be done only once.
>
> Do this procedure reboot the host or the instances that it contains?
>
>
> Thanks
>
>
>
> Il 27/11/18 09:49, Rohit Yadav ha scritto:
> > Hi Richard,
> >
> >
> > Please read:
> http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/hosts.html#security
> >
> >
> > 4.11.2 is out, please consider using it instead of 4.11.1 as it has
> several bugfixes etc.
> >
> > In short, with all of your KVM hosts up and connected to mgmt
> server, first change the auth strictness global setting to true, then
> using API secure the hosts using the provisionCertificate API. In the
> UI, go to your hosts that don't show up as secure and click on the key
> button (a new button) to secure the host which calls the
> provisionCertificate API as well.
> >
> >
> > - Rohit
> >
> > <https://cloudstack.apache.org>
> >
> >
> >
> > ________________________________
> > From: Richard Persaud <richard.persaud@macys.com>
> > Sent: Monday, November 26, 2018 8:19:56 PM
> > To: users@cloudstack.apache.org
> > Subject: RE: secure hosts communications
> >
> > Thank you, Rohit.
> >
> > I am using 4.11.1 with a full KVM environment. They are showing
> unsecure with strictness set to true.
> >
> > What configuration needs to be adjusted to have the KVM hosts show
> secure?
> >
> > Regards,
> >
> > Richard Persaud
> >
> > From: Rohit Yadav <rohit.yadav@shapeblue.com>
> > Sent: Saturday, November 24, 2018 2:02 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: secure hosts communications
> >
> > ⚠ EXT MSG:
> >
> > Richard,
> >
> >
> > Starting 4.11, agent and management servers will use an in-built CA
> framework to secured hosts. Only in case of KVM hosts you may see an
> insecure state, otherwise all KVM hosts (agents) and SSVM/CPVM agents
> will by default in Up state will be secured. There is an auth
> strictness setting that should be true.
> >
> >
> >
> > - Rohit
> >
> > <https://cloudstack.apache.org>
> >
> >
> >
> > ________________________________
> > From: Richard Persaud
> <richard.persaud@macys.com<mailto:richard.persaud@macys.com>>
> > Sent: Saturday, November 24, 2018 4:21:24 AM
> > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> > Subject: secure hosts communications
> >
> > Hello,
> >
> > Is there straight-forward to enable secure communications between
> the management and the hosts?
> >
> > I have looked at many documentations but am still unable to get the
> hosts to show a "secure" state.
> >
> > Regards,
> >
> > Richard Persaud
> >
> >
> > rohit.yadav@shapeblue.com<mailto:rohit.yadav@shapeblue.com>
> >
> www.shapeblue.com<https://isolate.menlosecurity.com/0/eJyrViotylGyUsooKSmw0tcvLy_XK85ILEhNyilN1UvOz1XSUSrKV7Iy1FEqyUwBqjM0MFaqBQDf4BCe>
> > Amadeus House, Floral Street, London WC2E 9DPUK
> > @shapeblue
> >
> >
> >
> >
> > * This is an EXTERNAL EMAIL. Stop and think before clicking a link
> or opening attachments.
> >
> > rohit.yadav@shapeblue.com
> > www.shapeblue.com <http://www.shapeblue.com>
> > Amadeus House, Floral Street, London WC2E 9DPUK
> > @shapeblue
> >
> >
> >
> >
> >
> >
>
>
> --
>
> *Ugo Vasi* / System Administrator
> ugo.vasi@procne.it <mailto:ugo.vasi@procne.it>
>
>
>
>
> *Procne S.r.l.*
> +39 0432 486 523
> via Cotonificio, 45
> 33010 Tavagnacco (UD)
> www.procne.it <http://www.procne.it> <http://www.procne.it/>
>
>
> Le informazioni contenute nella presente comunicazione ed i relativi
> allegati possono essere riservate e sono, comunque, destinate
> esclusivamente alle persone od alla Società sopraindicati. La
> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
> "Codice in materia di protezione dei dati personali". Se avete ricevuto
> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
> info@procne.it <mailto:info@procne.it>.
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <mailto:ugo.vasi@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <mailto:info@procne.it>.
|