cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ugo Vasi <ugo.v...@procne.it.INVALID>
Subject Re: secure hosts communications
Date Thu, 31 Jan 2019 10:50:51 GMT
Hi Rohit,
I tryed renew certificate but it failed!
Now libvirt does not restart and agent is disconnected:

agent log:
2019-01-31 11:17:07,530 INFO 
[resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] 
(Certificate Renewal Timer:null) (logid:fe1554cc) Restarting libvirt 
after certificate provisioning/renewal
2019-01-31 11:17:07,567 INFO  [cloud.agent.Agent] 
(AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2019-01-31 11:17:07,568 WARN  [cloud.agent.Agent] (Certificate Renewal 
Timer:null) (logid:fe1554cc) Failed to execute post certificate renewal 
command:
java.lang.IllegalStateException: Shutdown in progress
         at 
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
         at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
         at 
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157)
         at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
         at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
         at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
         at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
         at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
         at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
         at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
         at java.util.TimerThread.mainLoop(Timer.java:555)
         at java.util.TimerThread.run(Timer.java:505)
2019-01-31 11:17:09,797 INFO  [cloud.agent.AgentShell] (main:null) 
(logid:) Agent started
2019-01-31 11:17:09,800 INFO  [cloud.agent.AgentShell] (main:null) 
(logid:) Implementation Version is 4.11.2.0
2019-01-31 11:17:09,802 INFO  [cloud.agent.AgentShell] (main:null) 
(logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2019-01-31 11:17:09,815 INFO  [cloud.agent.AgentShell] (main:null) 
(logid:) Defaulting to using properties file for storage
2019-01-31 11:17:09,816 INFO  [cloud.agent.AgentShell] (main:null) 
(logid:) Defaulting to the constant time backoff algorithm
2019-01-31 11:17:09,828 INFO  [cloud.utils.LogUtils] (main:null) 
(logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2019-01-31 11:17:09,850 INFO  [cloud.agent.AgentShell] (main:null) 
(logid:) Using default Java settings for IPv6 preference for agent 
connection
2019-01-31 11:17:09,998 INFO  [cloud.agent.Agent] (main:null) (logid:) 
id is 5
2019-01-31 11:17:10,030 INFO  [kvm.resource.LibvirtConnection] 
(main:null) (logid:) No existing libvirtd connection found. Opening a 
new one
2019-01-31 11:17:10,175 ERROR [cloud.agent.AgentShell] (main:null) 
(logid:) Unable to start agent:
com.cloud.utils.exception.CloudRuntimeException: Failed to connect 
socket to '/var/run/libvirt/libvirt-sock': No such file or directory
         at 
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914)
         at com.cloud.agent.Agent.<init>(Agent.java:190)
         at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
         at 
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
         at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
         at com.cloud.agent.AgentShell.start(AgentShell.java:512)
         at com.cloud.agent.AgentShell.main(AgentShell.java:547)
(logs repeat)

syslog:


Jan 31 11:17:07 cshp214 sh[5065]: INFO 
[resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] 
(Certificate Renewal Timer:) (logid:fe1554cc) Restarting libvirt after 
certificate provisioning/renewal
Jan 31 11:17:07 cshp214 systemd[1]: Stopping CloudStack Agent...
Jan 31 11:17:07 cshp214 sh[5065]: INFO  [cloud.agent.Agent] 
(AgentShutdownThread:) (logid:) Stopping the agent: Reason = sig.kill
Jan 31 11:17:07 cshp214 sh[5065]: WARN  [cloud.agent.Agent] (Certificate 
Renewal Timer:) (logid:fe1554cc) Failed to execute post certificate 
renewal command:
Jan 31 11:17:07 cshp214 sh[5065]: java.lang.IllegalStateException: 
Shutdown in progress
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
java.lang.Runtime.removeShutdownHook(Runtime.java:239)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
java.util.TimerThread.mainLoop(Timer.java:555)
Jan 31 11:17:07 cshp214 sh[5065]: #011at 
java.util.TimerThread.run(Timer.java:505)
Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading data: 
Input/output error
Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading data: 
Input/output error
Jan 31 11:17:08 cshp214 systemd[1]: Stopped CloudStack Agent.
Jan 31 11:17:08 cshp214 systemd[1]: Stopping Virtualization daemon...
Jan 31 11:17:08 cshp214 systemd[1]: Stopped Virtualization daemon.
Jan 31 11:17:08 cshp214 systemd[1]: Starting Virtualization daemon...
Jan 31 11:17:08 cshp214 systemd[1]: Started Virtualization daemon.
Jan 31 11:17:08 cshp214 systemd[1]: Started CloudStack Agent.
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN No appenders could be 
found for logger (com.cloud.agent.AgentShell).
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN Please initialize the 
log4j system properly.
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN See 
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jan 31 11:17:09 cshp214 sh[25387]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Agent started
Jan 31 11:17:09 cshp214 sh[25387]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Implementation Version is 4.11.2.0
Jan 31 11:17:09 cshp214 sh[25387]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) agent.properties found at 
/etc/cloudstack/agent/agent.properties
Jan 31 11:17:09 cshp214 sh[25387]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Defaulting to using properties file for storage
Jan 31 11:17:09 cshp214 sh[25387]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Defaulting to the constant time backoff algorithm
Jan 31 11:17:09 cshp214 sh[25387]: INFO  [cloud.utils.LogUtils] (main:) 
(logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
Jan 31 11:17:09 cshp214 sh[25387]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Using default Java settings for IPv6 preference for 
agent connection
Jan 31 11:17:09 cshp214 sh[25387]: INFO  [cloud.agent.Agent] (main:) 
(logid:) id is 5
Jan 31 11:17:10 cshp214 sh[25387]: INFO [kvm.resource.LibvirtConnection] 
(main:) (logid:) No existing libvirtd connection found. Opening a new one
--
Jan 31 11:17:16 cshp214 snmpd[2460]: error on subcontainer 'ia_addr' 
insert (-1)
Jan 31 11:17:16 cshp214 snmpd[2460]: message repeated 3 times: [ error 
on subcontainer 'ia_addr' insert (-1)]
Jan 31 11:17:20 cshp214 systemd[1]: cloudstack-agent.service: Service 
hold-off time over, scheduling restart.
Jan 31 11:17:20 cshp214 systemd[1]: Stopped CloudStack Agent.
Jan 31 11:17:20 cshp214 systemd[1]: Started CloudStack Agent.
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN No appenders could be 
found for logger (com.cloud.agent.AgentShell).
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN Please initialize the 
log4j system properly.
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN See 
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jan 31 11:17:21 cshp214 sh[25457]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Agent started
Jan 31 11:17:21 cshp214 sh[25457]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Implementation Version is 4.11.2.0
Jan 31 11:17:21 cshp214 sh[25457]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) agent.properties found at 
/etc/cloudstack/agent/agent.properties
Jan 31 11:17:21 cshp214 sh[25457]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Defaulting to using properties file for storage
Jan 31 11:17:21 cshp214 sh[25457]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Defaulting to the constant time backoff algorithm
Jan 31 11:17:21 cshp214 sh[25457]: INFO  [cloud.utils.LogUtils] (main:) 
(logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
Jan 31 11:17:21 cshp214 sh[25457]: INFO  [cloud.agent.AgentShell] 
(main:) (logid:) Using default Java settings for IPv6 preference for 
agent connection
Jan 31 11:17:21 cshp214 sh[25457]: INFO  [cloud.agent.Agent] (main:) 
(logid:) id is 5
Jan 31 11:17:21 cshp214 sh[25457]: INFO [kvm.resource.LibvirtConnection] 
(main:) (logid:) No existing libvirtd connection found. Opening a new one
Jan 31 11:17:21 cshp214 sh[25457]: libvirt: XML-RPC error : Failed to 
connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Jan 31 11:17:21 cshp214 sh[25457]: ERROR [cloud.agent.AgentShell] 
(main:) (logid:) Unable to start agent:
Jan 31 11:17:21 cshp214 sh[25457]: 
com.cloud.utils.exception.CloudRuntimeException: Failed to connect 
socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Jan 31 11:17:21 cshp214 sh[25457]: #011at 
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914)
Jan 31 11:17:21 cshp214 sh[25457]: #011at 
com.cloud.agent.Agent.<init>(Agent.java:190)
Jan 31 11:17:21 cshp214 sh[25457]: #011at 
com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
Jan 31 11:17:21 cshp214 sh[25457]: #011at 
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
Jan 31 11:17:21 cshp214 sh[25457]: #011at 
com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
Jan 31 11:17:21 cshp214 sh[25457]: #011at 
com.cloud.agent.AgentShell.start(AgentShell.java:512)
Jan 31 11:17:21 cshp214 sh[25457]: #011at 
com.cloud.agent.AgentShell.main(AgentShell.java:547)
Jan 31 11:17:21 cshp214 sh[25457]: Unable to start agent: Failed to 
connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Main 
process exited, code=exited, status=67/n/a
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Unit 
entered failed state.
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Failed 
with result 'exit-code'.
Jan 31 11:17:21 cshp214 dnsmasq[4000]: read /etc/hosts - 13 addresses
Jan 31 11:17:21 cshp214 dnsmasq[4000]: read 
/var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jan 31 11:17:21 cshp214 dnsmasq-dhcp[4000]: read 
/var/lib/libvirt/dnsmasq/default.hostsfile
Jan 31 11:17:22 cshp214 snmpd[2460]: Connection from UDP: 
[127.0.0.1]:37699->[127.0.0.1]:161
Jan 31 11:17:24 cshp214 snmpd[2460]: message repeated 2 times: [ 
Connection from UDP: [127.0.0.1]:37699->[127.0.0.1]:161]
Jan 31 11:17:24 cshp214 libvirtd[25368]: libvirt version: 1.3.1, 
package: 1ubuntu10.24 (Marc Deslauriers <marc.deslauriers@ubuntu.com> 
Wed, 23 May 2018 13:29:29 -0400)
Jan 31 11:17:24 cshp214 libvirtd[25368]: hostname: cshp214
Jan 31 11:17:24 cshp214 libvirtd[25368]: Configured security driver 
"none" disables default policy to create confined guests
Jan 31 11:17:25 cshp214 libvirtd[25368]: unsupported configuration: 
Security driver apparmor not enabled


Can anyone help me?

Il 30/01/19 13:37, Rohit Yadav ha scritto:
>
> Hi Ugo,
>
>
> This will be a one-time procedure, and the KVM host and the VMs do not 
> need a reboot but the provisionCertificate API will restart the 
> libvirtd process (just check if that can have any side effects for 
> your VMs/distro, on most modern distros restarting libvirtd does not 
> have any side-effects on existing running VMs).
>
>
> - Rohit
>
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> @shapeblue
>
> ------------------------------------------------------------------------
> *From:* Ugo Vasi <ugo.vasi@procne.it>
> *Sent:* Wednesday, January 30, 2019 4:47:09 PM
> *To:* users@cloudstack.apache.org; Rohit Yadav
> *Subject:* Re: secure hosts communications
> Hi Rohit,
> I have a 4.11.2.0 ACS infrastructure (Ubuntu 16.04 with KVM hypervisor)
> I see that all the hosts are in unsecure state from the UI and so the
> live migration don't works (we had trubles with mgmt server).
>
> I read in the documentation that launching the provisionCertificate API
> (by pressing the appropriate button in the UI) the certificates will be
> renewed/regenerated for already connected agents/hosts.
>
> I do not understand if provisioning should be done manually on each host
> or if the procedure should be done only once.
>
> Do this procedure reboot the host or the instances that it contains?
>
>
> Thanks
>
>
>
> Il 27/11/18 09:49, Rohit Yadav ha scritto:
> > Hi Richard,
> >
> >
> > Please read: 
> http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/hosts.html#security
> >
> >
> > 4.11.2 is out, please consider using it instead of 4.11.1 as it has 
> several bugfixes etc.
> >
> > In short, with all of your KVM hosts up and connected to mgmt 
> server, first change the auth strictness global setting to true, then 
> using API secure the hosts using the provisionCertificate API. In the 
> UI, go to your hosts that don't show up as secure and click on the key 
> button (a new button) to secure the host which calls the 
> provisionCertificate API as well.
> >
> >
> > - Rohit
> >
> > <https://cloudstack.apache.org>
> >
> >
> >
> > ________________________________
> > From: Richard Persaud <richard.persaud@macys.com>
> > Sent: Monday, November 26, 2018 8:19:56 PM
> > To: users@cloudstack.apache.org
> > Subject: RE: secure hosts communications
> >
> > Thank you, Rohit.
> >
> > I am using 4.11.1 with a full KVM environment. They are showing 
> unsecure with strictness set to true.
> >
> > What configuration needs to be adjusted to have the KVM hosts show 
> secure?
> >
> > Regards,
> >
> > Richard Persaud
> >
> > From: Rohit Yadav <rohit.yadav@shapeblue.com>
> > Sent: Saturday, November 24, 2018 2:02 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: secure hosts communications
> >
> > ⚠ EXT MSG:
> >
> > Richard,
> >
> >
> > Starting 4.11, agent and management servers will use an in-built CA 
> framework to secured hosts. Only in case of KVM hosts you may see an 
> insecure state, otherwise all KVM hosts (agents) and SSVM/CPVM agents 
> will by default in Up state will be secured. There is an auth 
> strictness setting that should be true.
> >
> >
> >
> > - Rohit
> >
> > <https://cloudstack.apache.org>
> >
> >
> >
> > ________________________________
> > From: Richard Persaud 
> <richard.persaud@macys.com<mailto:richard.persaud@macys.com>>
> > Sent: Saturday, November 24, 2018 4:21:24 AM
> > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> > Subject: secure hosts communications
> >
> > Hello,
> >
> > Is there straight-forward to enable secure communications between 
> the management and the hosts?
> >
> > I have looked at many documentations but am still unable to get the 
> hosts to show a "secure" state.
> >
> > Regards,
> >
> > Richard Persaud
> >
> >
> > rohit.yadav@shapeblue.com<mailto:rohit.yadav@shapeblue.com>
> > 
> www.shapeblue.com<https://isolate.menlosecurity.com/0/eJyrViotylGyUsooKSmw0tcvLy_XK85ILEhNyilN1UvOz1XSUSrKV7Iy1FEqyUwBqjM0MFaqBQDf4BCe>
> > Amadeus House, Floral Street, London  WC2E 9DPUK
> > @shapeblue
> >
> >
> >
> >
> > * This is an EXTERNAL EMAIL. Stop and think before clicking a link 
> or opening attachments.
> >
> > rohit.yadav@shapeblue.com
> > www.shapeblue.com <http://www.shapeblue.com>
> > Amadeus House, Floral Street, London  WC2E 9DPUK
> > @shapeblue
> >
> >
> >
> >
> >
> >
>
>
> -- 
>
> *Ugo Vasi* / System Administrator
> ugo.vasi@procne.it <mailto:ugo.vasi@procne.it>
>
>
>
>
> *Procne S.r.l.*
> +39 0432 486 523
> via Cotonificio, 45
> 33010 Tavagnacco (UD)
> www.procne.it <http://www.procne.it> <http://www.procne.it/>
>
>
> Le informazioni contenute nella presente comunicazione ed i relativi
> allegati possono essere riservate e sono, comunque, destinate
> esclusivamente alle persone od alla Società sopraindicati. La
> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
> "Codice in materia di protezione dei dati personali". Se avete ricevuto
> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
> info@procne.it <mailto:info@procne.it>.
>


-- 

*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <mailto:ugo.vasi@procne.it>




*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>


Le informazioni contenute nella presente comunicazione ed i relativi 
allegati possono essere riservate e sono, comunque, destinate 
esclusivamente alle persone od alla Società sopraindicati. La 
diffusione, distribuzione e/o copiatura del documento trasmesso da parte 
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi 
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003 
"Codice in materia di protezione dei dati personali". Se avete ricevuto 
questo messaggio per errore, vi preghiamo di distruggerlo e di informare 
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail 
info@procne.it <mailto:info@procne.it>.


Mime
View raw message