Ethernet Over MPLS -EoMPLS
Lets talk about the basics and the configurational part of the Ethernet over MPLS. Whenever you think about MPLS, it is a core technology of the service Provider. If you would like to understand more on MPLS and VPLS, you can go through the below mentioned links for more details
 |
| Fig 1.1- EoMPLS tunnel between Sites |
EoMPLS comes in two flavors: port-based (everything goes on a trunk port, preserving 802.1q VLAN info) or VLAN-based (traffic in an access VLAN gets transported). VLAN-based can transport different VLANs to different endpoints, or mix and match L2 and L3 activities on a port (via subinterfaces). It is configured using dot1q sub interfaces.
Note that the technology will in principle do that for you. I'm not necessarily recommending you do that, because the cluster may behave oddly if the network between HERE and THERE goes flaky, drops packets for a while, etc. But EoMPLS is a great thing to have in your bag of tricks, for when you need it.
Why EoMPLS? It is downright handy when you have a VLAN that needs to be HERE and THERE, with a routed network in the middle. For example, cluster heartbeat between servers in two data centers. You do need MPLS capable switches that can also do EoMPLS at wire speed. Unlike MPLS VPN, there is no requirement for MBGP.
The other warning about EoMPLS is it might be a bit dangerous to your network's health. It is so easy to set up, that you can easily dig yourself a nice big hole, full or L2 over L3 "spaghetti". Which complicates troubleshooting, defeats structured design, ignores the carefully crafted routed core in your network, and causes warts. Well, 3 out of 4 anyway.
EoMPLS versus VPLS
Like QinQ tunneling (802.1q tunneling of switched traffic), EoMPLS just takes the frame that comes in a port (somehow), transports it across the middle, and spits it out the paired port on the other end. There is no examination of MAC address, no learning of source MAC address, in short, no switching logic applied.VPLS is Virtual Private LAN Service. It is EoMPLS on steroids. In VPLS, the switch can have several EoMPLS tunnels and make a switching decision, as to which one to use. VPLS in the 6500 (7600) requires extra hardware assistance. It does not work in a "vanilla" 6500 (assuming the 6500 can somehow be considered vanilla is kind of a big assumption, I know).
Traditional VPLS
If you want switching logic, you can cheat a bit. If you cable the port doing EoMPLS to another switch, that other switch does normal switching. You can even cable the EoMPLS to a non-EoMPLS port on the same switch. This is called a "loopback cable". (I do wish folks used a term that couldn't be confused with a loopback interface. Something like "humdinger" or "frobozz" perhaps?) This is mildly wasteful of ports, but works just fine.So if you want to transport traffic from any of a bunch of ports on a switch to another switch using EoMPLS, add one more port to the VLAN, cable it physically to the EoMPLS port, and it'll work.
Design Example
Three switches, Switch A, Switch B and Switch C . Switches A and B are connected to C and Switch C is a Core Switch in this scenarioI enabled MPLS globally and on the uplink / downlink interfaces between the Switch A, Switch B, and Switch C switches. In practice, one would enable them on all infrastructure uplinks, downlinks, and crosslinks in the distribution and core layers, and to selected closets if you need EoMPLS anywhere, anytime. The technique does not work if there is a routed path along which MPLS is not enabled.
Note: to support the MPLS labels, you'll want jumbo support consistently configured throughout as well. This requires per-interface configuration on uplinks, downlinks, and crosslinks. (The same interfaces that will be doing MPLS labels.)
Sample Configuration
!
mpls ip
!
int Ten1/1
mtu 9216 mpls ip
!
int Ten1/2
int Ten1/2
mtu 9216
mpls ip
!
Make sure you will not configure MPLS on the port that will be connected at L2 via EoMPLS. Just the paths in between the two endpoints.
I created two xconnect pseudowires, one from swA to swC, the other from swB to swC. As noted, the pseudo-wires can either be port-based (all VLANs) or VLAN-based (just one VLAN). I tried it both ways. Due to late hours, I did not test VLAN-based extensively.
Port-Based EoMPLS
The first xconnect went from Gig 4/1 to Gig 2/2, the second from Gig 4/1 to Gig 2/3 (Switch A and Switch B to Switch C). The addresses shown are the loopback address for the switch on the other end. The number is the circuit ID, which allows the two switches to recognize the two ends of one connection.Here's some captured text from switch B:
Switch A#show run int gig 4/1
interface GigabitEthernet4/1
mtu 9216
no ip address
xconnect 10.159.0.2 201 encapsulation mpls
spanning-tree portfast trunk
swB#show mpls l2 vc
Local intf Local circuit Dest address VC ID Status
------------- -------------------- --------------- ---------- ----------
Gi4/1 Ethernet 10.159.0.2 201 UP
Switch B#show mpls l2 bind
Destination Address: 10.159.0.2, VC ID: 201
Local Label: 72
Cbit: 1, VC Type: Ethernet, GroupID: 0
MTU: 9216, Interface Desc: w-ecc-01 G06-314 U31 ONBD1
VCCV: CC Type: RA [2]
CV Type: LSPV [2]
Remote Label: 76
Cbit: 1, VC Type: Ethernet, GroupID: 0
MTU: 9216, Interface Desc: n/a
VCCV: CC Type: RA [2]
CV Type: LSPV [2]
And for switch A:
Switch A#show run int gig 4/1
...
interface GigabitEthernet4/1
mtu 9216
no ip address
xconnect 10.159.0.2 200 encapsulation mpls
spanning-tree portfast trunk
end
Switch A#show mpls l2 vc
Local intf Local circuit Dest address VC ID Status
------------- -------------------- --------------- ---------- ----------
Gi4/1 Ethernet 10.159.0.2 200 UP
And for switch C:
Switch C#show run int gig 2/2
... interface GigabitEthernet2/2
mtu 9216
no ip address
xconnect 10.112.0.128 200 encapsulation mpls
spanning-tree portfast trunk
...
interface GigabitEthernet2/3
mtu 9216
no ip address
xconnect 10.112.0.130 201 encapsulation mpls
spanning-tree portfast trunk
Switch C#show mpls l2 vc
Local intf Local circuit Dest address VC ID Status
------------- -------------------- --------------- ---------- ----------
Gi2/2 Ethernet 10.112.0.128 200 UP
Gi2/3 Ethernet 10.112.0.130 201 UP
Troubleshooting EoMPLS
Along the way, I ran into two problems. The first took some time to resolve. The xconnect was not coming up, and debug showed an authorization problem. It turned out the MTU on the physical port (for port-based) has to be set, and to at least 1504, to accommodate VLAN 802.1q tagging. Subsequent experimentation showed that the xconnect verifies that the two end physical ports have the same MTU. (This would be a lovely nasty time-killer for a CCIE lab test, I suspect.)Caution: not all 6500 blades support jumbos (8000 to 9216 byte MTU).
The second (minor) problem was that the xconnect does not come up unless the physical port is up (something connected to it, link status). If you're trying to configure it without two devices plugged into the two ends, it's going to be difficult to see that word "up". (I was somewhat expecting it to behave more like a GRE tunnel: configure it and if it is happy, it shows as "up".)
I mention these as possible gotchas when doing xconnect. They could consume time trouble-shooting if you don’t expect this behavior.
Testing EoMPLS
This was testing by plugging in my two test PCs, addressed with 1.1.1.10 and 1.1.1.11. Those were certainly not in the global routing table in the lab.When I did so, I could ping between the PCs, despite their being connected to two different switches with only routing of 10.0.0.0 networks in between. I varied the PC connection points to test all combinations (pairs of ports). They worked. (Output not captured.)
I also verified that when one PC was on swA and the other on swB, they could not ping each other (nor even ARP each other). There is no local switching of traffic coming out one pseudowire back into another. (For that, SIP or ES hardware is required with VPLS functionality).
However, I patched port 2/2 on swC to 2/5, and 2/3 to 2/6, and did “no shutdown” on the latter two ports. They defaulted to both being in VLAN 1. I was then able to ping between the edge-connected PCs. Neat!
The following capture shows that ports 2/5 and 2/6 were doing normal MAC-based LAN switching, and there was no MAC learning on ports 2/2 and 2/3:
Switch C#show mac-address-table dyn int gig 2/5
Displaying entries from Line card 2:
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
Module 2:
* 1 000b.db99.6dec dynamic Yes 220 Gi2/5
Switch C#show mac-address-table dyn int gig 2/6
Displaying entries from Line card 2:
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
Module 2:
* 1 0015.c5b5.f3e4 dynamic Yes 225 Gi2/6
Switch C#show mac-address-table dyn int gig 2/6 2
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
No entries present.
Switch C#show mac-address-table dyn int gig 2/2 3
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
No entries present.
Once this was worked, I was pleased with the extreme simplicity of adding xconnects.Note also that enabling jumbos and MPLS on the infrastructure only needs to be done once, no matter how many xconnects are to be built for various purposes.
For new deployments and upgrades, we now enable jumbos on the infrastructure since consistently doing it supports all sorts of later needs. And doing it in ad hoc fashion is an invitation to all sorts of fun, especially with OSPF. (Adjacencies stay up with MTU mismatches, but won't come up after the link bounces -- even months after you changed the configuration.)
Note: redundant xconnects for High Availability require special handling of Spanning Tree. Note however that re-routing and MPLS will keep an xconnect up if there is any routed path fully supporting MPLS between the endpoints, so xconnects should be rather robust.
