cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wido den Hollander <w...@widodh.nl>
Subject Re: IPV6 in Isolated/VPC networks
Date Mon, 19 Jul 2021 09:08:10 GMT


Op 17-07-2021 om 06:28 schreef Hean Seng:
> I think if doing this way ,  since you were to implement on peering ip
> between vr and phsical router , then would need keep /56 or 48 at
> Clodustack ?  We can only add /64 subnet to Cloudstack only (instead of
> keep the /56 or 48 there).
> 

We can have a /64 at CloudStack in which all VRs talk with the router of 
the ISP.

That is large enough as a interconnect subnet between the VRs and routers.

 From there you can route /56, /48 or which size you want towards the VR.

 From the interconnect /64 you can also grab IPs which you use for 
loadbalancing purposes over different VMs.


> I  saw other software provider do is adding /64 subnet to their system,
> and  after that allocate subnet to the VM (from the previous added list).
> 
> May be considering the OSPF if really on this.  It really a nightmare for
> maintaining 1000 or few thousand of BGP session.   You can imagine your
> Cisco Router list of few thousand BGP session there.
> 

Yes, but I would suggest that both OSPFv3 and BGP should work. Not 
everybody will have 1000 accounts on their environment.

Even static routes should be supported.

Wido

> 
> 
> 
> 
> On Fri, Jul 16, 2021 at 11:17 PM Wido den Hollander <wido@widodh.nl> wrote:
> 
>>
>>
>> Op 16-07-2021 om 16:42 schreef Hean Seng:
>>> Hi Wido,
>>>
>>> In current setup,  each Cloudstack have own VR, so in this new  IPv6
>> subnet
>>> allocation , each VR (which have Frr) will need to have peering with ISP
>>> router (and either BGP or Static Route) , and there is 1000 Acocunts,  it
>>> will 1000 BGP session with ISP router ,  Am I right for this ? or I
>>> understand wrong .
>>>
>>
>> Yes, that is correct. A /56 would also be sufficient or a /60 which is
>> enough to allocate a few /64 subnets.
>>
>> 1000 BGP connections isn't really a problem for a proper router at the
>> ISP. OSPF(v3) would be better, but as I said that's poorly supported.
>>
>> The ISP could also install 1000 static routes, but that means that the
>> ISP's router needs to have those configured.
>>
>> http://docs.frrouting.org/en/latest/ospf6d.html
>> (While looking up this URL I see that Frr recently put in a lot of work
>> in OSPFv3, seems better now)
>>
>>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each
>>> devices have own IP. It just how to realize in easy way.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <wido@widodh.nl>
>> wrote:
>>>
>>>>
>>>>
>>>> Op 16-07-2021 om 05:54 schreef Hean Seng:
>>>>> Hi Wido,
>>>>>
>>>>> My initial thought is not like this,  it is the /48 at ISP router, and
>>>> /64
>>>>> subnet assign to AdvanceZoneVR,   AdvanceZoneVR responsible is
>>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM,  and not
>> routing
>>>>> the traffic,   in the VM that get the IPv6 IP will default route to ISP
>>>>> router as gw.   It can may be a bridge over via Advancezone-VR.
>>>>>
>>>>
>>>> How would you bridge this? That sounds like NAT?
>>>>
>>>> IPv6 is meant to be routed. Not to be translated or bridged in any way.
>>>>
>>>> The way a made the drawing is exactly how IPv6 should work in a VPC
>>>> environment.
>>>>
>>>> Traffic flows through the VR where it can do firewalling of the traffic.
>>>>
>>>>> However, If do as the way described in the drawing, then i suppose will
>>>> be
>>>>> another kind of virtual router going to introduce , to get hold the /48
>>>> in
>>>>> this virtual router right ?
>>>>>
>>>>
>>>> It can be the same VR. But keep in mind that IPv6 != IPv4.
>>>>
>>>> The VR will get Frr as a new daemon which can talk BGP with the upper
>>>> network to route traffic.
>>>>
>>>>> After this,  The Advance Zone, NAT's  VR will peer with this new IPv6
>> VR
>>>>> for getting the IPv6 /64 prefix ?
>>>>>
>>>>
>>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT.
>>>>
>>>>> If do in this way, then I guess  you just only need Static route, with
>>>>> peering ip both end  as one /48 can have a lot of /64 on it.  And
>>>> hardware
>>>>> budgeting for new IPv6-VR will become very important, as all traffic
>> will
>>>>> need to pass over it .
>>>>>
>>>>
>>>> Routing or NAT is the same for the VR. You don't need a very beefy VR
>>>> for this.
>>>>
>>>>> It will be like
>>>>>
>>>>> ISP Router  ------ >  (new IPV6-VR ) ---- > AdvanceZone-VR ---->
VM
>>>>>
>>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering
>> on
>>>>> OSPF instead of  BGP , otherwise few thousand of AdvanceZone-VR wil
>> have
>>>>> few thousand of BGP session. on new-IPv6-VR
>>>>>
>>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct,
>>   ,
>>>>> otherwise ISP router will be full of /64 prefix route either on BGP(
>> Many
>>>>> BGP Session) , or  Many Static route .   If few thousand account, ti
>> will
>>>>> be few thousand of BGP session with ISP router or few thousand static
>>>> route
>>>>> which  is not possible .
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <wido@widodh.nl>
>>>> wrote:
>>>>>
>>>>>> But you still need routing. See the attached PNG (and draw.io XML).
>>>>>>
>>>>>> You need to route the /48 subnet TO the VR which can then route it
to
>>>>>> the Virtual Networks behind the VR.
>>>>>>
>>>>>> There is no other way then routing with either BGP or a Static route.
>>>>>>
>>>>>> Wido
>>>>>>
>>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng:
>>>>>>> Or explain like this :
>>>>>>>
>>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network
admin
>>>>>>> assigned to Cloudstack
>>>>>>> 2) Cloudsack allocated the subnet (that generated from step1)
to
>>>> Virtual
>>>>>>> Router, one Virtual Router have one subniet /64
>>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64
>>>>>>> allocated to VR)  to VM
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heanseng@gmail.com
>>>>>>> <mailto:heanseng@gmail.com>> wrote:
>>>>>>>
>>>>>>>        Hi Wido,
>>>>>>>
>>>>>>>        I think the /48 is at physical router as gateway , and
subnet
>> of
>>>> /64
>>>>>>>        at VR of Cloudstack.   Cloudstack only keep which /48
prefix
>> and
>>>>>>>        vlan information of this /48 to be later split the  /64.
to VR.
>>>>>>>
>>>>>>>        And the instances is getting singe IPv6 of /64  IP.  
The VR is
>>>>>>>        getting /64.  The default gateway shall goes to /48 of
physical
>>>>>>>        router ip .   In this case ,does not need any BGP router
.
>>>>>>>
>>>>>>>
>>>>>>>        Similar concept as IPv4 :
>>>>>>>
>>>>>>>        /48 subnet of IPv6 is equivalent to current /24 subnet
of IPv4
>>>> that
>>>>>>>        created in Network.
>>>>>>>        and /64  of IPv6 is equivalent to single IP of IPv4 assign
to
>> VM.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>        On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander <
>>>> wido@widodh.nl
>>>>>>>        <mailto:wido@widodh.nl>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>            Op 14-07-2021 om 16:44 schreef Hean Seng:
>>>>>>>             > Hi
>>>>>>>             >
>>>>>>>             > I replied in another thread, i think do not
need
>> implement
>>>>>>>            BGP or OSPF,
>>>>>>>             > that would be complicated .
>>>>>>>             >
>>>>>>>             > We only need assign  IPv6 's /64 prefix to Virtual
>> Router
>>>>>>>            (VR) in NAT
>>>>>>>             > zone, and the VR responsible to deliver single
IPv6 to
>> VM
>>>> via
>>>>>>>            DHCP6.
>>>>>>>             >
>>>>>>>             > In VR, you need to have Default IPv6 route to
 Physical
>>>>>>>            Router's /48. IP
>>>>>>>             > as IPv6 Gateway.  Thens should be done .
>>>>>>>             >
>>>>>>>             > Example :
>>>>>>>             > Physical Router Interface
>>>>>>>             >   IPv6 IP : 2000:aaaa::1/48
>>>>>>>             >
>>>>>>>             > Cloudstack  virtual router : 2000:aaaa:200:201::1/64
>> with
>>>>>>>            default ipv6
>>>>>>>             > route to router ip 2000:aaaa::1
>>>>>>>             > and Clodustack Virtual router dhcp allocate
IP to VM ,
>> and
>>>>>>>            VM will have
>>>>>>>             > default route to VR. IPv6 2000:aaaa:200:201::1
>>>>>>>             >
>>>>>>>             > So in cloudstack need to allow  user to enter
,  IPv6
>>>>>>>            gwateway , and
>>>>>>>             > the  /48 Ipv6 prefix , then it will self allocate
the
>> /64
>>>> ip
>>>>>>>            to the VR ,
>>>>>>>             > and maintain make sure not ovelap allocation
>>>>>>>             >
>>>>>>>             >
>>>>>>>
>>>>>>>            But NAT is truly not the solution with IPv6. IPv6
is
>> supposed
>>>> to
>>>>>> be
>>>>>>>            routable. In addition you should avoid DHCPv6 as much
as
>>>>>>>            possible as
>>>>>>>            that's not really the intended use-case for address
>> allocation
>>>>>>>            with IPv6.
>>>>>>>
>>>>>>>            In order to route an /48 IPv6 subnet to the VR you
have a
>> few
>>>>>>>            possibilities:
>>>>>>>
>>>>>>>            - Static route from the upperlying routers which are
>> outside
>>>> of
>>>>>>>            CloudStack
>>>>>>>            - BGP
>>>>>>>            - OSPFv3 (broken in most cases!)
>>>>>>>            - DHCPv6 Prefix Delegation
>>>>>>>
>>>>>>>            BGP and/or Static routes are still the best bet here.
>>>>>>>
>>>>>>>            So what you do is that you tell CloudStack that you
will
>> route
>>>>>>>            2001:db8::/48 to the VR, the VR can then use that
to split
>> it
>>>> up
>>>>>>>            into
>>>>>>>            multiple /64 subnets going towards the instances:
>>>>>>>
>>>>>>>            - 2001:db8::/64
>>>>>>>            - 2001:db8:1::/64
>>>>>>>            - 2001:db8:2::/64
>>>>>>>            ...
>>>>>>>            - 2001:db8:f::/64
>>>>>>>
>>>>>>>            And go on.
>>>>>>>
>>>>>>>            In case of BGP you indeed have to tell the VR a few
things:
>>>>>>>
>>>>>>>            - It's own AS number
>>>>>>>            - The peer's address(es)
>>>>>>>
>>>>>>>            With FRR you can simply say:
>>>>>>>
>>>>>>>            neighbor 2001:db8:4fa::179 remote-as external
>>>>>>>
>>>>>>>            The /48 you need to have at the VR anyway in case
of
>> either a
>>>>>>>            static
>>>>>>>            route or BGP.
>>>>>>>
>>>>>>>            We just need to add a NullRoute on the VR for that
/48 so
>> that
>>>>>>>            traffic
>>>>>>>            will not be routed to the upper gateway in case of
the VR
>>>> can't
>>>>>>>            find a
>>>>>>>            route.
>>>>>>>
>>>>>>>            Wido
>>>>>>>
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >
>>>>>>>             > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli
>>>>>>>             > <Alex.Mattioli@shapeblue.com
>>>>>>>            <mailto:Alex.Mattioli@shapeblue.com>
>>>>>>>            <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>            <mailto:Alex.Mattioli@shapeblue.com>>>
wrote:
>>>>>>>             >
>>>>>>>             >     Hi Wido,
>>>>>>>             >     That's pretty much in line with our thoughts,
thanks
>>>> for
>>>>>>>            the input.
>>>>>>>             >     I believe we agree on the following points
then:
>>>>>>>             >
>>>>>>>             >     - FRR with BGP (no OSPF)
>>>>>>>             >     - Route /48 (or/56) down to the VR
>>>>>>>             >     - /64 per network
>>>>>>>             >     - SLACC for IP addressing
>>>>>>>             >
>>>>>>>             >     I believe the next big question is then
"on which
>> level
>>>>>>>            of ACS do we
>>>>>>>             >     manage AS numbers?".  I see two options:
>>>>>>>             >     1) Private AS number on a per-zone basis
>>>>>>>             >     2) Root Admin assigned AS number on a domain/account
>>>> basis
>>>>>>>             >     3) End-user driven AS number on a per network
basis
>>>> (for
>>>>>>>            bring your
>>>>>>>             >     own AS and IP scenario)
>>>>>>>             >
>>>>>>>             >     Thoughts?
>>>>>>>             >
>>>>>>>             >     Cheers
>>>>>>>             >     Alex
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >     -----Original Message-----
>>>>>>>             >     From: Wido den Hollander <wido@widodh.nl
>>>>>>>            <mailto:wido@widodh.nl> <mailto:wido@widodh.nl
>>>>>>>            <mailto:wido@widodh.nl>>>
>>>>>>>             >     Sent: 13 July 2021 15:08
>>>>>>>             >     To: dev@cloudstack.apache.org
>>>>>>>            <mailto:dev@cloudstack.apache.org>
>>>>>>>            <mailto:dev@cloudstack.apache.org
>>>>>>>            <mailto:dev@cloudstack.apache.org>>;
>>>>>>>             >     Alex Mattioli <Alex.Mattioli@shapeblue.com
>>>>>>>            <mailto:Alex.Mattioli@shapeblue.com>
>>>>>>>             >     <mailto:Alex.Mattioli@shapeblue.com
>>>>>>>            <mailto:Alex.Mattioli@shapeblue.com>>>
>>>>>>>             >     Cc: Wei Zhou <Wei.Zhou@shapeblue.com
>>>>>>>            <mailto:Wei.Zhou@shapeblue.com>
>>>>>>>             >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>            <mailto:Wei.Zhou@shapeblue.com>>>; Rohit
Yadav
>>>>>>>             >     <rohit.yadav@shapeblue.com
>>>>>>>            <mailto:rohit.yadav@shapeblue.com>
>>>>>>>            <mailto:rohit.yadav@shapeblue.com
>>>>>>>            <mailto:rohit.yadav@shapeblue.com>>>;
>>>>>>>             >     Gabriel Beims Bräscher <gabriel@pcextreme.nl
>>>>>>>            <mailto:gabriel@pcextreme.nl>
>>>>>>>             >     <mailto:gabriel@pcextreme.nl <mailto:
>>>> gabriel@pcextreme.nl
>>>>>>>>>
>>>>>>>             >     Subject: Re: IPV6 in Isolated/VPC networks
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >     On 7/7/21 1:16 PM, Alex Mattioli wrote:
>>>>>>>             >      > Hi all,
>>>>>>>             >      > @Wei Zhou<mailto:Wei.Zhou@shapeblue.com
>>>>>>>            <mailto:Wei.Zhou@shapeblue.com>
>>>>>>>             >     <mailto:Wei.Zhou@shapeblue.com
>>>>>>>            <mailto:Wei.Zhou@shapeblue.com>>> @Rohit
>>>>>>>             >     Yadav<mailto:rohit.yadav@shapeblue.com
>>>>>>>            <mailto:rohit.yadav@shapeblue.com>
>>>>>>>             >     <mailto:rohit.yadav@shapeblue.com
>>>>>>>            <mailto:rohit.yadav@shapeblue.com>>> and
myself are
>>>>>>>            investigating how
>>>>>>>             >     to enable IPV6 support on Isolated and VPC
networks
>> and
>>>>>>>            would like
>>>>>>>             >     your input on it.
>>>>>>>             >      > At the moment we are looking at implementing
FRR
>>>> with
>>>>>>>            BGP (and
>>>>>>>             >     possibly OSPF) on the ACS VR.
>>>>>>>             >      >
>>>>>>>             >      > We are looking for requirements, recommendations,
>>>>>>>            ideas, rants,
>>>>>>>             >     etc...etc...
>>>>>>>             >      >
>>>>>>>             >
>>>>>>>             >     Ok! Here we go.
>>>>>>>             >
>>>>>>>             >     I think that you mean that the VR will actually
>> route
>>>> the
>>>>>>>            IPv6
>>>>>>>             >     traffic and for that you need to have a
way of
>> getting
>>>> a
>>>>>>>            subnet
>>>>>>>             >     routed to the VR.
>>>>>>>             >
>>>>>>>             >     BGP is probably you best bet here. Although
OSPFv3
>>>>>>>            technically
>>>>>>>             >     supports this it is very badly implemented
in Frr
>> for
>>>>>>>            example.
>>>>>>>             >
>>>>>>>             >     Now FRR is a very good router and one of
the fancy
>>>>>>>            features it
>>>>>>>             >     supports is BGP Unnumered. This allows for
auto
>>>>>>>            configuration of BGP
>>>>>>>             >     over a L2 network when both sides are sending
Router
>>>>>>>            Advertisements.
>>>>>>>             >     This is very easy for flexible BGP configurations
>> where
>>>>>>>            both sides
>>>>>>>             >     have dynamic IPs.
>>>>>>>             >
>>>>>>>             >     What you want to do is that you get a /56,
/48 or
>>>>>>>            something which is
>>>>>>>             >      >/64 bits routed to the VR.
>>>>>>>             >
>>>>>>>             >     Now you can sub-segment this into separate
/64
>> subnets.
>>>>>>>            You don't
>>>>>>>             >     want to go smaller then a /64 is that prevents
you
>> from
>>>>>>>            using SLAAC
>>>>>>>             >     for IPv6 address configuration. This is
how it works
>>>> for
>>>>>>>            Shared
>>>>>>>             >     Networks now in Basic and Advanced Zones.
>>>>>>>             >
>>>>>>>             >     FRR can now also send out the Router Advertisements
>> on
>>>>>>>            the downlinks
>>>>>>>             >     sending out:
>>>>>>>             >
>>>>>>>             >     - DNS servers
>>>>>>>             >     - DNS domain
>>>>>>>             >     - Prefix (/64) to be used
>>>>>>>             >
>>>>>>>             >     There is no need for DHCPv6. You can calculate
the
>> IPv6
>>>>>>>            address the
>>>>>>>             >     VM will obtain by using the MAC and the
prefix.
>>>>>>>             >
>>>>>>>             >     So in short:
>>>>>>>             >
>>>>>>>             >     - Using BGP you routed a /48 to the VR
>>>>>>>             >     - Now you split this into /64 subnets towards
the
>>>>>>>            isolated networks
>>>>>>>             >
>>>>>>>             >     Wido
>>>>>>>             >
>>>>>>>             >      > Alex Mattioli
>>>>>>>             >      >
>>>>>>>             >      >
>>>>>>>             >      >
>>>>>>>             >      >
>>>>>>>             >
>>>>>>>             >
>>>>>>>             >
>>>>>>>             > --
>>>>>>>             > Regards,
>>>>>>>             > Hean Seng
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>        --
>>>>>>>        Regards,
>>>>>>>        Hean Seng
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Hean Seng
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
> 
> 

Mime
View raw message