cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jumani <David.Jum...@shapeblue.com>
Subject Re: ACS 4.13.1 failed to create PVLAN network
Date Mon, 01 Jun 2020 05:08:57 GMT
Hi Jerry, could you trace the DHCP packet coming out of the vm and the router ?


ovs-appctl ofproto/trace vlan_tci=0x0000,dl_src=<vm_mac>
ovs-appctl ofproto/trace vlan_tci=0x0000,dl_src=<vm_mac>,dl_dst=ff:ff:ff:ff:ff:ff
ovs-appctl ofproto/trace dl_vlan=<primary_vlan>,dl_src=<vm_mac>,dl_dst=ff:ff:ff:ff:ff:ff

________________________________
From: li jerry <div8cn@hotmail.com>
Sent: Saturday, May 30, 2020 1:13 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: ACS 4.13.1 failed to create PVLAN network

hi David, I cleared all the experimental environment and redeployed the test.
pvlan can work normally, and a VM with a PVLAN network + an isolated network can also work
normally.

However, I found that when the VR where the PVLAN is located and the VM are not in the same
HOST, the VM cannot obtain the IP of the PVLAN.

I checked ovs flows and the flow table was issued.

(VM has Pvlan Network and Isolated Network, Isolated Network can DHCP to IP, Pvlan Network
can not get.

(Migrate VM to HOST where PVLAN VR is located, then VM can get all IPs)


Does PVLAN need other switch support?

My switch all port trunk all

-----邮件原件-----
发件人: David Jumani <David.Jumani@shapeblue.com>
发送时间: 2020年5月29日 12:36
收件人: users@cloudstack.apache.org
主题: Re: ACS 4.13.1 failed to create PVLAN network

That's great!
There will be no communication between the devices on a PVLAN if they're isolated except DHCP
(since isolated should not be able to communicate with each other).
About multiple NICs, in my setup, I've had a VM attached to a PVLAN as well as an Isolated
and it worked. Haven't tried multiple PVLANs though ________________________________
From: li jerry <div8cn@hotmail.com>
Sent: Thursday, May 28, 2020 2:05 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: ACS 4.13.1 failed to create PVLAN network

Thanks to David, Boris and all friends.

I am on the master branch of clone github.com, and then merge PR [https://github.com/apache/cloudstack/pull/4040].

I have successfully implemented a PVLAN network in my test environment.
Realize no communication between VM and VM; Communication between VM and DHCP (dhcp request)

However, during the test, the following two problems were encountered.

1. VM does not support multiple NICs
  When creating vm> 1 NIC, the cloudstack-agent not create ovs flows.


2. L3 PVLAN, Secondary Isolated VLAN Type = Isolated. VM cannot get userdata information
  I tested in the VM and found that only DHCP requests can be sent to the VR, and other requests
HTTP 80 AND 8080 can not communicate.


Is this a bug? Or am I doing something wrong?

-----邮件原件-----
发件人: David Jumani <David.Jumani@shapeblue.com>
发送时间: 2020年5月27日 19:38
收件人: users@cloudstack.apache.org
主题: Re: ACS 4.13.1 failed to create PVLAN network

It's off master, I've built it using the PR that I shared, and yes, two bridges via ovs in
my setup too!
________________________________
From: li jerry <div8cn@hotmail.com>
Sent: Wednesday, May 27, 2020 3:34 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: 回复: ACS 4.13.1 failed to create PVLAN network

Thanks David, is your environment 4.13.1 or 4.15?

I can test it according to your version.

In 4.13.1 I used openvswitch and created two bridges through ovs-vsctl

-Jerry

发件人: David Jumani<mailto:David.Jumani@shapeblue.com>
发送时间: 2020年5月27日 16:28
收件人: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
主题: Re: ACS 4.13.1 failed to create PVLAN network

Hi Jerry,

The way PVLAN works on KVM is that it uses OpenFlow rules to emulate PVLAN, so OVS must be
used for Linux networking, not the default Linux bridge. Apart from that, there were certain
issues with it which have been addressed in the PR provided by Boris. It's been tested and
working on OVS 2.9.2 When creating an L3 PVLAN, the VR is automatically created only when
an instance is brought up on that network.
The steps I followed are :

  1.  Create the PVLAN network
  2.  Create an instance and attach it to the network
  3.  Wait until the router comes up

The OpenFlow scripts run in the background on the agent Once the instance is up, it should
work as expected

I haven't tried adding a live host to a PVLAN network, but it worked when I tried it following
the above steps.
You can test the PVLAN connectivity by pinging another host using the interface attached to
the PVLAN network ________________________________
From: Boris Stoyanov <boris.stoyanov@shapeblue.com>
Sent: Wednesday, May 27, 2020 1:45 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: ACS 4.13.1 failed to create PVLAN network

I can't advise if simply merging this code in 4.13 will work fine, it may cause some side
issues since it's a big gap in code bases, therefore I'll advise you to install/upgrade you
test env to this PR ( 4.15 )

Bobby.

On 27.05.20, 11:07, "li jerry" <div8cn@hotmail.com> wrote:

    Thank you Boris for the information.

    I now go to merge this PR into 4.13.1 for testing. Provide test results later.

    -Jerry

    发件人: Boris Stoyanov<mailto:boris.stoyanov@shapeblue.com>
    发送时间: 2020年5月27日 16:01
    收件人: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
    主题: Re: ACS 4.13.1 failed to create PVLAN network

    Hi Li,

    Can you try your tests with this PR: https://github.com/apache/cloudstack/pull/4040
    This one ^^ is enabling it on L2 networks, to be honest I'm not sure about L3 as I'm currently
testing it.

    But to be able to test I'll need to facilitate an L3 network (arping requires IP), and
then specify the interface to the other L2 network attached to the same VM.
    Please note that this is 4.15+ and it's currently under testing, so do it in a testing
lab.

    Thanks,

    On 26.05.20, 18:14, "li jerry" <div8cn@hotmail.com> wrote:

        Dear All

        Who has successfully deployed a PVLAN network on ACS 4.13.1?
        I created the PVLAN network through the following process, but it failed

        Server:
        CentOS7
        CloudStack 4.13.1

        Zone 1:

        l  Physical Network1 :Public and Guest , Isolation method=VLAN, KVM traffic label=br1

        l  Physical Network2 :  Management , Isolation method=VLAN, KVM traffic label=br2

        SystemVM Template (KVM): Version 4.11.3

        Hypervisor:
        CentOS7.7 3.10.0-1062
        Openvswitch-2.12.0
        libvirt 4.5.0
        QEMU 1.5.3



        1.     Create L3 Guest Network, name= Pvlan-Net-01, vlan=700, isolatedpvlan=1700

        ( this is api command)

        command=createNetwork&zoneId=409b04ea-d128-48ac-8e33-4df700da89cc&networkOfferingId=9778a4ab-0de8-4440-9879-a488416e0572&physicalnetworkid=ca0768c8-f068-4d88-b7bd-2766414a6415&name=Pvlan-Net-01&displayText=Pvlan-Net-01&vlan=700&bypassVlanOverlapCheck=false&isolatedpvlan=1700&acltype=domain&gateway=172.17.0.1&netmask=255.255.0.0&startip=172.17.1.1&endip=172.17.1.254&networkdomain=hyperx.com&response=json&_=1590497900407


        2.     attache network to VM vm1 (this VM is running and has an isolated network 10.0.0.x
/ 24)

        3.     attache failed, management throws the following error:



        this is management log

        2020-05-26 21:59:40,268 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-905dd7d7
job-44/job-45 ctx-4c296832) (logid:2009fc05) Seq 4-6374282322589515787: Received:  { Ans:
, MgmtId: 113349236140, via: 4(2222), Ver: v1, Flags: 10, { StartAnswer, CheckSshAnswer, GetDomRVersionAnswer,
NetworkUsageAnswer, Answer, Answer, Answer, Answer, Answer } }

        2020-05-26 21:59:40,280 DEBUG [o.a.c.n.t.AdvancedNetworkTopology] (Work-Job-Executor-1:ctx-905dd7d7
job-44/job-45 ctx-4c296832) (logid:2009fc05) SETUP DHCP PVLAN RULES

        2020-05-26 21:59:40,292 DEBUG [c.c.n.r.NetworkHelperImpl] (Work-Job-Executor-1:ctx-905dd7d7
job-44/job-45 ctx-4c296832) (logid:2009fc05) Router requires upgrade. Unable to send command
to router:5, router template version : null, minimal required version : 4.10.0

        2020-05-26 21:59:40,294 WARN  [o.a.c.n.t.AdvancedNetworkVisitor] (Work-Job-Executor-1:ctx-905dd7d7
job-44/job-45 ctx-4c296832) (logid:2009fc05) Timed Out

        com.cloud.exception.ResourceUnavailableException: Resource [VirtualRouter:5] is unreachable:
Unable to send command. Router requires upgrade

        at com.cloud.network.router.NetworkHelperImpl.sendCommandsToRouter(NetworkHelperImpl.java:175)

        at org.apache.cloudstack.network.topology.AdvancedNetworkVisitor.visit(AdvancedNetworkVisitor.java:185)

        at com.cloud.network.rules.DhcpPvlanRules.accept(DhcpPvlanRules.java:61)

        at org.apache.cloudstack.network.topology.AdvancedNetworkTopology.setupDhcpForPvlan(AdvancedNetworkTopology.java:131)

        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeStart(VirtualNetworkApplianceManagerImpl.java:2080)

        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1170)

        at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:904)

        at com.cloud.network.router.NetworkHelperImpl.start(NetworkHelperImpl.java:277)

        at com.cloud.network.router.NetworkHelperImpl.startVirtualRouter(NetworkHelperImpl.java:356)

        at com.cloud.network.router.NetworkHelperImpl.startRouters(NetworkHelperImpl.java:341)

        at org.cloud.network.router.deployment.RouterDeploymentDefinition.deployVirtualRouter(RouterDeploymentDefinition.java:205)

        at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:278)

        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1380)

        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1715)

        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createNicForVm(NetworkOrchestrator.java:3767)

        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateAddVmToNetwork(VirtualMachineManagerImpl.java:3504)

        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateAddVmToNetwork(VirtualMachineManagerImpl.java:5264)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)

        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5326)

        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)

        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:603)

        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)

        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)

        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)

        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)

        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)

        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:551)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

        2020-05-26 21:59:40,297 INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-905dd7d7
job-44/job-45 ctx-4c296832) (logid:2009fc05) The guru did not like the answers so stopping
VM[DomainRouter|r-5-VM]

        2020-05-26 21:59:40,302 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-905dd7d7
job-44/job-45 ctx-4c296832) (logid:2009fc05) Seq 4-6374282322589515789: Sending  { Cmd , MgmtId:
113349236140, via: 4(2222), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.188.7","forceStop":false,"volumesToDisconnect":[],"vmName":"r-5-VM","executeInSequence":false,"wait":0}}]
}





        Agent error

        20-05-26 22:53:44,801 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-2:null)
(logid:) Execution is successful.

        2020-05-26 22:53:44,802 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-2:null)
(logid:) Looking for libvirtd connection at: qemu:///system

        2020-05-26 22:53:44,807 DEBUG [cloud.agent.Agent] (UgentTask-2:null) (logid:) Sending
ping: Seq 4-4:  { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"r-11-VM":{"state":"PowerOn","host":"2222"},"r-4-VM":{"state":"PowerOn","host":"2222"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":4,"wait":0}}]
}

        2020-05-26 22:53:44,903 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:)
Received response: Seq 4-4:  { Ans: , MgmtId: 113349236140, via: 4, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":4,"wait":0},"result":true,"wait":0}}]
}

        2020-05-26 22:53:48,475 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Trying to connect to 169.254.208.61

        2020-05-26 22:53:48,477 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5)
Processing command: com.cloud.agent.api.check.CheckSshCommand

        2020-05-26 22:53:48,477 DEBUG [resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper]
(agentRequest-Handler-3:null) (logid:03678ec5) Ping command port, 169.254.208.61:3922

        2020-05-26 22:53:48,477 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Trying to connect to 169.254.208.61

        2020-05-26 22:53:48,477 DEBUG [resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper]
(agentRequest-Handler-3:null) (logid:03678ec5) Ping command port succeeded for vm r-11-VM

        2020-05-26 22:53:48,477 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5)
Processing command: com.cloud.agent.api.GetDomRVersionCmd

        2020-05-26 22:53:48,480 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
get_template_version.sh 169.254.208.61

        2020-05-26 22:53:48,482 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Executing while with timeout : 1800000

        2020-05-26 22:53:48,986 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Execution is successful.

        2020-05-26 22:53:48,987 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Executing script in VR: get_template_version.sh

        2020-05-26 22:53:48,988 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5)
Processing command: com.cloud.agent.api.NetworkUsageCommand

        2020-05-26 22:53:48,988 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
netusage.sh 169.254.208.61 -c

        2020-05-26 22:53:48,989 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Executing while with timeout : 3600000

        2020-05-26 22:53:49,571 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Execution is successful.

        2020-05-26 22:53:49,572 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5)
Processing command: com.cloud.agent.api.routing.AggregationControlCommand

        2020-05-26 22:53:49,572 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5)
Processing command: com.cloud.agent.api.routing.SetMonitorServiceCommand

        2020-05-26 22:53:49,573 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5)
Processing command: com.cloud.agent.api.routing.AggregationControlCommand

        2020-05-26 22:53:49,573 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Transforming com.cloud.agent.api.routing.SetMonitorServiceCommand to ConfigItems

        2020-05-26 22:53:49,602 DEBUG [virtualnetwork.facade.AbstractConfigItemFacade] (agentRequest-Handler-3:null)
(logid:03678ec5) Transformed filename: monitor_service.json to: monitor_service.json.e2aeaa96-5a74-4753-8edf-6a040717a8aa

        2020-05-26 22:53:49,604 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Aggregate action timeout in seconds is 600

        2020-05-26 22:53:49,605 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Creating file in VR, with ip: 169.254.208.61, file: VR-1e88cb1c-f82d-4994-9a80-b8c5fc22fd9e.cfg

        2020-05-26 22:53:50,236 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
vr_cfg.sh 169.254.208.61 -c /var/cache/cloud/VR-1e88cb1c-f82d-4994-9a80-b8c5fc22fd9e.cfg

        2020-05-26 22:53:50,239 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Executing while with timeout : 600600

        2020-05-26 22:53:52,488 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Execution is successful.

        2020-05-26 22:53:52,488 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null)
(logid:03678ec5) Executing script in VR: vr_cfg.sh





        Note:

        1.     I used the same template to create the Isolated network and attache to the
VM, everything works fine.

        2.     When I created the PVLAN VR, I used the virsh console to enter the VM and executed
the get_template_version.sh script, which output cloudstack 4.11.3


        I do n’t know what causes the PVLAN network to be created!
        Any help would be greatly appreciated!
        If you need more detailed information, please let me know
        Thank you



        -Jerry




    boris.stoyanov@shapeblue.com
    www.shapeblue.com<http://www.shapeblue.com>
    3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
    @shapeblue






boris.stoyanov@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK @shapeblue




David.Jumani@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK @shapeblue




David.Jumani@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK @shapeblue




David.Jumani@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK @shapeblue




David.Jumani@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message