Lab Introduction This lab tests point-to-multipoint (P2MP) OPSF with DMVPN phase 3 to enable spoke-to-spoke traffic and understand how OSPF metric/cos
Lab Introduction
This lab tests point-to-multipoint (P2MP) OPSF with DMVPN phase 3 to enable spoke-to-spoke traffic and understand how OSPF metric/cost works in point-to-multipoint scenario.
I use C7200 15.2(4) M6 (IOS: c7200-adventerprisek9-mz.152-4.M6.bin) as router on GNS3. Please note:
- Not all C7200 IOS support DMVPN Phase 3 with GNS3. For example, when I used later version 15.2(4) S7 (IOS: c7200-adventerprisek9-mz.152-4.S7.bin), it says ‘ip nhrp redirect’ failed to initialize.
- CSR1000v (IOS-XE) supports DMVPN Phase 3 with GNS3; however, it shows some different results in terms of troubleshooting. It should be ideal to use CSR1000v if RAM is not an issue.
- Crypto is not configured to protect tunnels in this lab for simplicity.
Lab topology is as below:
HUB1_1
G1/0 : 200.1.1.101/24 (global routing instance)
Tunnel 101: 172.16.1.101/24 (global routing instance), OSPF 1 area 0
Lo 1: 10.66.1.101/24 (global routing instance), OSPF 1 area 1
interface Tunnel 101
ip address 172.16.1.101 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication GREEN
ip nhrp map multicast dynamic
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp redirect
ip ospf network point-to-multipoint
ip ospf 1 area 0
tunnel source GigabitEthernet1/0
tunnel mode gre multipoint
tunnel key 101
HUB1_2
G1/0 : 200.1.1.102/24 (global routing instance)
Tunnel 101: 172.16.1.102/24 (GREEN_IVRF), OSPF 1 area 0, cost 1001
Lo 1: 10.66.2.102/24 (GREEN_IVRF), OSPF 1 area 1
interface Tunnel 101
vrf forwarding GREEN_IVRF
ip address 172.16.1.102 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication GREEN
ip nhrp map multicast dynamic
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp redirect
ip ospf network point-to-multipoint
ip ospf 1 area 0
ip ospf cost 1001
tunnel source GigabitEthernet1/0
tunnel mode gre multipoint
tunnel key 101
SPOKE1_1
G1/0 : 200.1.1.103/24 (global routing instance)
Tunnel 101: 172.16.1.103/24 (GREEN_IVRF), OPSF 1 area 0
Lo 1: 10.66.3.103/24 (GREEN_IVRF), OSPF 1 area 1
interface Tunnel 101
vrf forwarding GREEN_IVRF
ip address 172.16.1.103 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication GREEN
ip nhrp map multicast 200.1.1.101
ip nhrp map multicast 200.1.1.102
ip nhrp map 172.16.1.101 200.1.1.101
ip nhrp map 172.16.1.102 200.1.1.102
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp nhs 172.16.1.101 cluster 1
ip nhrp nhs 172.16.1.102 priority 2 cluster 1
ip nhrp nhs cluster 1 max-connections 2
ip nhrp nhs fallback 25
ip nhrp shortcut
ip ospf network point-to-multipoint
ip ospf 1 area 0
tunnel source GigabitEthernet1/0
tunnel mode gre multipoint
tunnel key 101
SPOKE1_2
G1/0 : 200.1.1.104/24 (global routing instance)
Tunnel 101: 172.16.1.104/24 (GREEN_IVRF), OSPF 1 area 0
Lo 1: 10.66.4.104/24 (GREEN_IVRF), OSPF 1 area 1
interface Tunnel 101
vrf forwarding GREEN_IVRF
ip address 172.16.1.104 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication GREEN
ip nhrp map multicast 200.1.1.101
ip nhrp map multicast 200.1.1.102
ip nhrp map 172.16.1.101 200.1.1.101
ip nhrp map 172.16.1.102 200.1.1.102
ip nhrp network-id 101
ip nhrp holdtime 300
ip nhrp nhs 172.16.1.101 cluster 1
ip nhrp nhs 172.16.1.102 priority 2 cluster 1
ip nhrp nhs cluster 1 max-connections 2
ip nhrp nhs fallback 25
ip nhrp shortcut
ip ospf network point-to-multipoint
ip ospf 1 area 0
tunnel source GigabitEthernet1/0
tunnel mode gre multipoint
tunnel key 101
Verification 1 – show ip route
If we ‘show ip route’ on SPOKE1_1 and SPOKE1_2, spoke-to-spoke route appears traversing via HUB and with dual-trip metric (SPOKE1_1 to HUB1_1 to SPOKE1_2). Please note ‘10.66.4.104/32 [110/2001]’. Is the spoke-to-spoke direct tunnel really working?
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O IA 10.66.1.101/32 [110/1001] via 172.16.1.101, 01:11:28, Tunnel101
O IA 10.66.2.102/32 [110/1001] via 172.16.1.102, 00:14:50, Tunnel101
C 10.66.3.0/24 is directly connected, Loopback1
L 10.66.3.103/32 is directly connected, Loopback1
O IA 10.66.4.104/32 [110/2001] via 172.16.1.101, 00:02:49, Tunnel101
172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
C 172.16.1.0/24 is directly connected, Tunnel101
O 172.16.1.101/32 [110/1000] via 172.16.1.101, 01:15:38, Tunnel101
O 172.16.1.102/32 [110/1000] via 172.16.1.102, 00:14:50, Tunnel101
L 172.16.1.103/32 is directly connected, Tunnel101
O 172.16.1.104/32 [110/2000] via 172.16.1.101, 00:02:49, Tunnel101
Verification 2 – ‘show ip nhrp’ and ‘show dmvpn’
Before issuing any traffic, we execute ‘show ip nhrp’ on SPOKEs. It only shows nhrp entries to hubs.
172.16.1.101/32 via 172.16.1.101
Tunnel101 created 02:30:32, never expire
Type: static, Flags: used
NBMA address: 200.1.1.101
172.16.1.102/32 via 172.16.1.102
Tunnel101 created 02:30:32, never expire
Type: static, Flags: used
NBMA address: 200.1.1.102
However, when we ping from SPOKE1_2 to SPOKE1_1 lo1 address ‘ping vrf GREEN_IVRF 10.66.3.103’; SPOKE1_1 receives NHRP entries to SPOKE1_2 as dynamic type (172.16.1.104/32 via 172.16.1.104). Please also note ‘flags: router used’, which indicates spoke-to-spoke tunnel as temporary not ‘never expire’.
10.66.3.0/24 via 172.16.1.103
Tunnel101 created 00:00:07, expire 00:04:52
Type: dynamic, Flags: router unique local
NBMA address: 200.1.1.103
(no-socket)
172.16.1.101/32 via 172.16.1.101
Tunnel101 created 02:32:31, never expire
Type: static, Flags: used
NBMA address: 200.1.1.101
172.16.1.102/32 via 172.16.1.102
Tunnel101 created 02:32:31, never expire
Type: static, Flags: used
NBMA address: 200.1.1.102
172.16.1.104/32 via 172.16.1.104
Tunnel101 created 00:00:07, expire 00:04:51
Type: dynamic, Flags: router used
NBMA address: 200.1.1.104
SPOKE1_2 also receives NHRP entries to SPOKE1_1 as dynamic type (172.16.1.103/32 via 172.16.1.103).
172.16.1.101/32 via 172.16.1.101
Tunnel101 created 01:34:28, never expire
Type: static, Flags: used
NBMA address: 200.1.1.101
172.16.1.102/32 via 172.16.1.102
Tunnel101 created 01:34:28, never expire
Type: static, Flags: used
NBMA address: 200.1.1.102
172.16.1.103/32 via 172.16.1.103
Tunnel101 created 00:10:01, expire 00:02:35
Type: dynamic, Flags: router implicit
NBMA address: 200.1.1.103
172.16.1.104/32 via 172.16.1.104
Tunnel101 created 00:10:01, expire 00:02:35
Type: dynamic, Flags: router unique local
NBMA address: 200.1.1.104
‘show dmvpn detail’ also demonstrates spoke-to-spoke dynamic tunnel.
Legend: Attrb –> S – Static, D – Dynamic, I – Incomplete
N – NATed, L – Local, X – No Socket
# Ent –> Number of NHRP entries with same NBMA peer
NHS Status: E –> Expecting Replies, R –> Responding, W –> Waiting
UpDn Time –> Up or Down Time for a Tunnel
=======================================================
Interface Tunnel101 is up/up, Addr. is 172.16.1.103, VRF “GREEN_IVRF”
Tunnel Src./Dest. addr: 200.1.1.103/MGRE, Tunnel VRF “”
Protocol/Transport: “multi-GRE/IP”, Protect “”
Interface State Control: Disabled
nhrp event-publisher : Disabled
IPv4 NHS:
172.16.1.101 RE priority = 0 cluster = 1
172.16.1.102 RE priority = 2 cluster = 1
Type:Spoke, Total NBMA Peers (v4/v6): 4# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
—– ————— ————— —– ——– —– —————–
1 200.1.1.103 172.16.1.103 UP 01:26:03 DLX 10.66.3.0/24
1 200.1.1.101 172.16.1.101 UP 00:50:08 S 172.16.1.101/32
1 200.1.1.102 172.16.1.102 UP 01:38:28 S 172.16.1.102/32
1 200.1.1.104 172.16.1.104 UP 00:07:02 D 172.16.1.104/32
One HUB1_1, we also see nhrp redirect information as below:
I/F NBMA address Destination Drop Count ExpiryTunnel101 200.1.1.104 10.66.3.103 36 00:00:05
Verification 3 – debug nhrp
Let’s have a detailed look on what exactly happened on HUB1_1, SPOKE1_1 and SPOKE1_2 by ‘debug nhrp’.
HUB1_1: provides SPOKE physical WAN-facing IP to allow dynamic tunnel to be established between SPOKEs.
*May 4 20:34:41.415: NHRP: Attempting to Redirect, remote_nbma: 200.1.1.104
SPOKE1_2 adds SPOKE1_1 tunnel endpoint (172.16.1.103) as NHRP entry and sends packet via 172.16.1.103 directly. When no traffic traversing SPOKE1_1 and SPOKE1_2, the dynamic spoke-to-spoke tunnel will be deleted.
*May 4 20:32:55.131: NHRP: Adding Tunnel Endpoints (VPN: 172.16.1.103, NBMA: 200.1.1.103)
*May 4 20:32:55.131: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.1.103, NBMA: 200.1.1.103)
*May 4 20:32:55.131: NHRP: Cache Delete: Converting prefix: ‘10.66.3.0’ in cache: 0x6B0072B8
SPOKE 1_1 also adds SPOKE1_2 tunnel endpoint (172.16.1.104) as NHRP entry and sends packet via 172.16.1.104 directly.
*May 4 20:31:42.639: NHRP: Successfully attached NHRP subblock for Tunnel Endpoints (VPN: 172.16.1.104, NBMA: 200.1.1.104)
*May 4 20:31:42.655: NHRP: Attempting to send packet via DEST 172.16.1.104
*May 4 20:31:42.655: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 200.1.1.101
*May 4 20:31:42.655: NHRP: Send Resolution Reply via Tunnel101 vrf 1, packet size: 133
*May 4 20:31:42.655: src: 172.16.1.103, dst: 172.16.1.104
Verification 4 – traceroute
Traceroute result is interesting. When first time traceroute, the traffic traverses via HUB; however, traffic directly goes through SPOKE when tried second time after the dynamic spoke-to-spoke tunnel established.
Type escape sequence to abort.
Tracing the route to 10.66.3.103
VRF info: (vrf in name/id, vrf out name/id)
1 172.16.1.101 5 msec
172.16.1.102 5 msec
172.16.1.101 5 msec
Type escape sequence to abort.
Tracing the route to 10.66.3.103
VRF info: (vrf in name/id, vrf out name/id)
1 172.16.1.103 5 msec 5 msec *
When traffic going through spoke-to-spoke directly. Do a quick ‘show ip route’. 2 records are marked with % now. % means ‘next hop override’. Traffic to SPOKE1_2’s loopback 10.66.4.104 had HUB1_1’s tunnel IP (172.16.1.101) as next-hop according to the routing table; however, the next-hop is now overridden by SPOKE1_2’s tunnel IP (172.16.1.104). The traffic will now take 172.16.1.104 as next hop to reach 10.66.4.104 with OSPF metric 2001.
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O IA 10.66.1.101/32 [110/1001] via 172.16.1.101, 01:11:28, Tunnel101
O IA 10.66.2.102/32 [110/1001] via 172.16.1.102, 00:14:50, Tunnel101
C 10.66.3.0/24 is directly connected, Loopback1
L 10.66.3.103/32 is directly connected, Loopback1
O IA% 10.66.4.104/32 [110/2001] via 172.16.1.101, 00:02:49, Tunnel101
172.16.0.0/16 is variably subnetted, 5 subnets, 2 masks
C 172.16.1.0/24 is directly connected, Tunnel101
O 172.16.1.101/32 [110/1000] via 172.16.1.101, 01:15:38, Tunnel101
O 172.16.1.102/32 [110/1000] via 172.16.1.102, 00:14:50, Tunnel101
L 172.16.1.103/32 is directly connected, Tunnel101
O % 172.16.1.104/32 [110/2000] via 172.16.1.101, 00:02:49, Tunnel101
Verification 5 – OSPF neighbor
SPOKEs only form OPSF neighbor with HUBs, but not other SPOKEs.
Neighbor ID Pri State Dead Time Address Interface
200.1.1.101 0 FULL/ – 00:01:51 172.16.1.101 Tunnel101
200.1.1.102 0 FULL/ – 00:01:47 172.16.1.102 Tunnel101
Other Useful Commands
Clear ip nhrp
Show ip nhrp nhs
Show ip nhrp nhs redundancy
Conclusion
1) SPOKEs only form routing neighbors with HUBs, but not other SPOKEs.
2) SPOKE to SPOKE OSPF routing metric is calculated as if traffic traversing HUB; i.e. SPOKE1_1 to SPOKE1_2 metric = 1000+1000 = 2000. When spoke-to-spoke traffic initiates, routing metric stays same but next-hop is overriden to spoke.
3) When spoke-to-spoke traffic initiates, HUB will redirect traffic to spoke and provide spoke WAN-facing physical interface IP. Dynamic tunnel between SPOKE1_1 and SPOKE1_2 will established based on information provided by HUB. When spoke-to-spoke traffic ends, the dynamic tunnel will be terminated.
COMMENTS