Troubleshoot DMVPN Phase 2 Spoke-to-Spoke Tunnel (2024)

Introduction

This document describes how to troubleshoot a phase 2 spoke-to-spoke DMVPN tunnel when it does not establishes.

Prerequisites

Requirements

Cisco recommendsthat you have knowledge on the next topics:

Dynamic Multipoint Virtual Private Network (DMVPN)
IKE/IPSEC protocols
Next Hop Resolution Protocol (NHRP)

Components Used

This document is based on this software version:

Cisco CSR1000V (VXE) - Version17.03.08

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document describes how to configure and use different troubleshooting tools on a common DMVPN issue.The issue is failed negotiation of a phase 2 DMVPN tunnel, where the source spoke, the DMVPN state shows UP with the correct Non-Broadcast Multi-Access (NBMA)/Tunnel mapping to the destination spoke. However, on the destination spoke anincorrectmapping is displayed.

Theoretical Background

It is important to understand how spoke-to-spoke tunnels are established when having a DMVPN Phase 2 set-up. This section provides a brief theoretical summary of the NHRP process during this phase.

In DMVPN Phase 2 you can build dynamic spoke-to-spoke tunnels on demand.This is possible because, on all devices within the DMVPN cloud (huband spokes) the mode of the tunnel interface changes to Generic Routing Encapsulation (GRE) multipoint. One of the key features of this phase is that the hub is not perceived as the next-hop by the other devices. Instead, all spokes have the routing information of each other.When establishing a spoke-to-spoke tunnel in phase 2, an NHRP process gets triggered where the spokes learn the information about other spokes, and makes a mapping between the NBMA and tunnel IP addresses.

The next steps list how the NHRP resolution process is triggered:

When the source spoke tries to reach the LAN of the destination spoke, it does a route lookuptriggering the resolution request message to get the NBMA address of the destination spoke. The source spoke send this initial message to the hub.
The hubreceives the resolution request and forwards it to the destination spoke.
The destination spoke sends the resolution reply to the source spoke. If the tunnel configuration has an IPSEC profile linked:
- The NHRP resolution process is delayed until IKE/IPSEC protocols can establish.
- The destination spoke initiates and establish theIKE/IPSEC tunnels.
- Then, the NHRP process is resumed and the destination spoke sends theresolution reply to the source spoke using the IPSEC tunnel as transport method.

NHRP Message Flow Between the Spokes on Phase 2

Note: Before the resolution process can start, all the spokes must be already registered with the HUB.

Topology

This diagram shows the topology used for the scenario:

Network Diagram and IP Subnets Used

Troubleshooting Steps

In this scenario the spoke-to-spoke tunnel between Spoke1 and Spoke2 is not established, affecting the communication between their local resources (represented by loopback interfaces) as they are not able to reach each other.

SPOKE1#ping 192.168.2.2 source loopback1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Initial Validation

When encountering such a scenario, it is important to begin by validating the tunnel configuration and ensure that both devices have the correct values within it. To review the tunnel configuration run the command show running-config interface tunnel<ID>.

Spoke 1 tunnel configuration:

SPOKE1#show running-config interface tunnel10
Building configuration...

Current configuration : 341 bytes
!
interface Tunnel10
ip address 10.10.10.1 255.255.255.0
no ip redirects
ip nhrp authentication DMVPN
ip nhrp map 10.10.10.10 172.20.10.10
ip nhrp map multicast 172.20.10.10
ip nhrp network-id 10
ip nhrp nhs 10.10.10.10
tunnel source GigabitEthernet1
tunnel mode gre multipoint
tunnel protection IPSEC profile IPSEC_Profile_1
end

Spoke 2 tunnel configuration:

SPOKE2#show running-config interface tunnel10
Building configuration...

Current configuration : 341 bytes
!
interface Tunnel10
ip address 10.10.10.2 255.255.255.0
no ip redirects
ip nhrp authentication DMVPN
ip nhrp map 10.10.10.10 172.20.10.10
ip nhrp map multicast 172.20.10.10
ip nhrp network-id 10
ip nhrp nhs 10.10.10.10
tunnel source GigabitEthernet1
tunnel mode gre multipoint
tunnel protection IPSEC profile IPSEC_Profile_1
end

On the configuration you need to validate that the mapping to the HUB is correct, the NHRP authentication string is matching between the devices, both spokes have the same DMVPN phase configured, and, if IPSEC protection is used, verify that the correct crypto configuration is applied.

If the configuration is correct and it includes IPSEC protection, it is necessary to verify that the IKE and IPSEC protocols are working correctly. This is because NHRP uses the IPSEC tunnel as transport method to fully negotiate. To verify the state of the IKE/IPSEC protocols run the command show crypto IPSEC sa peer x.x.x.x (where x.x.x.x is the NBMAIP address of the spoke you are trying to establish the tunnel with).

Note: To verify if the IPSEC tunnel is up, the inbound and outbound Encapsulation Security Payload (ESP) section must have the tunnel information (SPI, transform-set, and so on). All the values shown on this section must match on both ends.

Note: If any issues with IKE/IPSEC are identified, the troubleshooting must focus on those protocols.

IKE/IPSEC tunnel status on Spoke1:

SPOKE1#show crypto IPSEC sa peer 172.22.200.2

interface: Tunnel10
Crypto map tag: Tunnel10-head-0, local addr 172.21.100.1

protected vrf: (none)
local ident (addr/mask/prot/port): (172.21.100.1/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (172.22.200.2/255.255.255.255/47/0)
current_peer 172.22.200.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 172.21.100.1, remote crypto endpt.: 172.22.200.2
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet1
current outbound spi: 0x6F6BF94A(1869347146)
PFS (Y/N): N, DH group: none

inbound esp sas:
spi: 0x84502A19(2219846169)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Transport, }
conn id: 2049, flow_id: CSR:49, sibling_flags FFFFFFFF80000008, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4608000/28716)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0x6F6BF94A(1869347146)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Transport, }
conn id: 2050, flow_id: CSR:50, sibling_flags FFFFFFFF80000008, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4608000/28716)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

Troubleshooting Tools

After doing the initial validation and corroborating the configuration and the IKE/IPSEC protocols (if needed) are not causing the communication problem, you can use the tools presented on this section to continue the troubleshooting.

Useful Commands

The command show dmvpn interface tunnel<ID>give youDMVPN-specific session information (NBMA/Tunnel IP addresses, state of the tunnel, up/down time and attribute). You can use the detailkeyword to show details from the crypto session/socket. It is important to mention that the state of the tunnel must match on both ends.

Spoke 1 show dmvpn interface tunnel<ID>output:

SPOKE1#show dmvpn interface tunnel10
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override, B - BGP
C - CTS Capable, I2 - Temporary
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================Interface: Tunnel10, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
 2 172.20.10.10 10.10.10.2 UP 00:00:51 I2
 10.10.10.10 UP 02:53:27 S

Spoke 2 show dmvpn interface tunnel<ID>output:

SPOKE2#show dmvpn interface tunnel10
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override, B - BGP
C - CTS Capable, I2 - Temporary
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================Interface: Tunnel10, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
 1 172.21.100.1 10.10.10.1 UP 00:03:53 D
 1 172.20.10.10 10.10.10.10 UP 02:59:14 S

The output on each device shows different information for each spoke. In the Spoke1 table, you can see that the entry for Spoke 2 does not include the correct NBMA IP address, and the attribute appears incomplete (I2). On the other hand, the Spoke2 table shows the correct mapping (NBMA/Tunnel IP addresses) and the state as up indicating that the tunnel is fully negotiated.

The next commands can be helpful during the troubleshooting process:

show ip nhrp: Display NHRP mapping information
show ip nhrp traffic interface tunnel10:Displays NHRP traffic statistics

Note: For command specifications (syntax, description, keywords, example), please consult the Command Reference:Cisco IOS Security Command Reference: Commands S to Z

Debugs

After verifying the previous information and confirmed that the tunnel is experiencing negotiation issues, it is necessary to enable debugs to observe how the NHRP packets are being exchanged. The next debugs mustbe enabled on all the involved devices:

debug dmvpn condition peer NBMA x.x.x.x (where x.x.x.x is the remote device IP address).
debug dmvpn all all: this command enables ISAKMP, IKEv2, IPSEC, DMVPN and NHRP debugging commands.

Tip: It is recommended to use the peer condition command every time you enable the debugs so you can see the negotiation of that specific tunnel.

To see the complete NHRP flow, the next debugs commands were used on each device:

Spoke1

debug dmvpn condition peer NBMA 172.22.200.2
debug dmvpn condition peer NBMA 172.20.10.10
debug dmvpn all all

HUB

debug dmvpn condition peer NBMA 172.21.100.1
debug dmvpn condition peer NBMA 172.22.200.2
debug dmvpn all all

Spoke2

debug dmvpn condition peer NBMA 172.21.100.1
debug dmvpn condition peer NBMA 172.20.10.10
debug dmvpn all all

Note: The debugs must be enabled and collected simultaneously on all the devices involved.

Debugs enabled on all the devices are displayedwith the command show debug:

ROUTER#show debug
IOSXE Conditional Debug Configs:

Conditional Debug Global State: Stop

IOSXE Packet Tracing Configs:

Packet Infra debugs:

Ip Address Port
------------------------------------------------------|----------

NHRP:
NHRP protocol debugging is on
NHRP activity debugging is on
NHRP detail debugging is on
NHRP extension processing debugging is on
NHRP cache operations debugging is on
NHRP routing debugging is on
NHRP rate limiting debugging is on
NHRP errors debugging is on
NHRP events debugging is on

Cryptographic Subsystem:
Crypto ISAKMP debugging is on
Crypto ISAKMP Error debugging is on
Crypto IPSEC debugging is on
Crypto IPSEC Error debugging is on
Crypto secure socket events debugging is on
IKEV2:
IKEv2 error debugging is on
IKEv2 default debugging is on
IKEv2 packet debugging is on
IKEv2 packet hexdump debugging is on
IKEv2 internal debugging is on
Tunnel Protection Debugs:
Generic Tunnel Protection debugging is on

DMVPN:
DMVPN error debugging is on
DMVPN UP/DOWN event debugging is on
DMVPN detail debugging is on
DMVPN packet debugging is on
DMVPN all level debugging is on

After collecting all the debugs, you must start analyzing the debugs on the source spoke (Spoke1),this allows you to trace the negotiation from the beginning.

Spoke1 debug output:

-------------------- [IKE/IPSEC DEBUG OUTPUTS OMITTED]--------------------

*Feb 1 01:31:34.657: ISAKMP: (1016):Old State = IKE_QM_R_QM2 New State = IKE_QM_PHASE2_COMPLETE
*Feb 1 01:31:34.657: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*Feb 1 01:31:34.657: IPSEC(key_engine_enable_outbound): rec'd enable notify from ISAKMP
*Feb 1 01:31:34.657: CRYPTO_SS(TUNNEL SEC): Sending MTU Changed message
*Feb 1 01:31:34.661: IPSEC-IFC MGRE/Tu10(172.21.100.1/172.22.200.2): Got MTU message mtu 1458
*Feb 1 01:31:34.661: IPSEC-IFC MGRE/Tu10(172.21.100.1/172.22.200.2): connection lookup returned 80007F2B7055F5A8
*Feb 1 01:31:34.662: CRYPTO_SS(TUNNEL SEC): Sending Socket Up message
*Feb 1 01:31:34.662: IPSEC-IFC MGRE/Tu10(172.21.100.1/172.22.200.2): connection lookup returned 80007F2B7055F5A8
*Feb 1 01:31:34.662: IPSEC-IFC MGRE/Tu10(172.21.100.1/172.22.200.2): tunnel_protection_socket_up
*Feb 1 01:31:34.662: IPSEC-IFC MGRE/Tu10(172.21.100.1/172.22.200.2): Signalling NHRP
*Feb 1 01:31:36.428: NHRP: Checking for delayed event NULL/10.10.10.2 on list (Tunnel10 vrf: global(0x0))
*Feb 1 01:31:36.429: NHRP: No delayed event node found.
*Feb 1 01:31:36.429: NHRP: There is no VPE Extension to construct for the request
*Feb 1 01:31:36.429: NHRP: Sending NHRP Resolution Request for dest: 10.10.10.2 to nexthop: 10.10.10.2 using our src: 10.10.10.1 vrf: global(0x0)
*Feb 1 01:31:36.429: NHRP: Attempting to send packet through interface Tunnel10 via DEST dst 10.10.10.2
*Feb 1 01:31:36.429: NHRP-DETAIL: First hop route lookup for 10.10.10.2 yielded 10.10.10.2, Tunnel10
*Feb 1 01:31:36.429: NHRP: Send Resolution Request via Tunnel10 vrf: global(0x0), packet size: 85
*Feb 1 01:31:36.429: src: 10.10.10.1, dst: 10.10.10.2
*Feb 1 01:31:36.429: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Feb 1 01:31:36.429: shtl: 4(NSAP), sstl: 0(NSAP)
*Feb 1 01:31:36.429: pktsz: 85 extoff: 52
*Feb 1 01:31:36.429: (M) flags: "router auth src-stable nat ", reqid: 10
*Feb 1 01:31:36.429: src NBMA: 172.21.100.1
*Feb 1 01:31:36.429: src protocol: 10.10.10.1, dst protocol: 10.10.10.2
*Feb 1 01:31:36.429: (C-1) code: no error(0), flags: none
*Feb 1 01:31:36.429: prefix: 0, mtu: 9976, hd_time: 600
*Feb 1 01:31:36.429: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Feb 1 01:31:36.429: Responder Address Extension(3):
*Feb 1 01:31:36.429: Forward Transit NHS Record Extension(4):
*Feb 1 01:31:36.429: Reverse Transit NHS Record Extension(5):
*Feb 1 01:31:36.429: Authentication Extension(7):
*Feb 1 01:31:36.429: type:Cleartext(1), data:DMVPN
*Feb 1 01:31:36.429: NAT address Extension(9):
*Feb 1 01:31:36.430: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.20.10.10
*Feb 1 01:31:36.430: NHRP: 109 bytes out Tunnel10
*Feb 1 01:31:36.430: NHRP-RATE: Retransmitting Resolution Request for 10.10.10.2, reqid 10, (retrans ivl 4 sec)
*Feb 1 01:31:39.816: NHRP: Checking for delayed event NULL/10.10.10.2 on list (Tunnel10 vrf: global(0x0))
*Feb 1 01:31:39.816: NHRP: No delayed event node found.
*Feb 1 01:31:39.816: NHRP: There is no VPE Extension to construct for the request
*Feb 1 01:31:39.817: NHRP: Sending NHRP Resolution Request for dest: 10.10.10.2 to nexthop: 10.10.10.2 using our src: 10.10.10.1 vrf: global(0x0)
*Feb 1 01:31:39.817: NHRP: Attempting to send packet through interface Tunnel10 via DEST dst 10.10.10.2
*Feb 1 01:31:39.817: NHRP-DETAIL: First hop route lookup for 10.10.10.2 yielded 10.10.10.2, Tunnel10
*Feb 1 01:31:39.817: NHRP: Send Resolution Request via Tunnel10 vrf: global(0x0), packet size: 85
*Feb 1 01:31:39.817: src: 10.10.10.1, dst: 10.10.10.2
*Feb 1 01:31:39.817: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Feb 1 01:31:39.817: shtl: 4(NSAP), sstl: 0(NSAP)
*Feb 1 01:31:39.817: pktsz: 85 extoff: 52
*Feb 1 01:31:39.817: (M) flags: "router auth src-stable nat ", reqid: 10
*Feb 1 01:31:39.817: src NBMA: 172.21.100.1
*Feb 1 01:31:39.817: src protocol: 10.10.10.1, dst protocol: 10.10.10.2
*Feb 1 01:31:39.817: (C-1) code: no error(0), flags: none
*Feb 1 01:31:39.817: prefix: 0, mtu: 9976, hd_time: 600
*Feb 1 01:31:39.817: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Feb 1 01:31:39.817: Responder Address Extension(3):
*Feb 1 01:31:39.817: Forward Transit NHS Record Extension(4):
*Feb 1 01:31:39.817: Reverse Transit NHS Record Extension(5):
*Feb 1 01:31:39.817: Authentication Extension(7):
*Feb 1 01:31:39.817: type:Cleartext(1), data:DMVPN
*Feb 1 01:31:39.817: NAT address Extension(9):
*Feb 1 01:31:39.817: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.20.10.10
*Feb 1 01:31:39.818: NHRP: 109 bytes out Tunnel10
*Feb 1 01:31:39.818: NHRP-RATE: Retransmitting Resolution Request for 10.10.10.2, reqid 10, (retrans ivl 8 sec)
*Feb 1 01:31:46.039: NHRP: Checking for delayed event NULL/10.10.10.2 on list (Tunnel10 vrf: global(0x0))
*Feb 1 01:31:46.040: NHRP: No delayed event node found.
*Feb 1 01:31:46.040: NHRP: There is no VPE Extension to construct for the request

Once the Spoke1 NHRP process begins, the logs show that the device is sending the NHRP resolution request. The packet has some important information like the src NMBA and src protocol that are the NBMA IP address and tunnel IP address of the source spoke (Spoke1). You can also see the dst protocol value that has the tunnel IP address of the destination spoke (Spoke2). This is indicating that Spoke1 is asking for the NBMA address of Spoke2 to complete the mapping.Also on the packet, you can find the reqid value that can help you track the packet along the path. This value is going to remain the same through the whole process, and can be helpful to track an specific flow of the NHRP negotiation. The packet has another values that are important to the negotiation like the NHRP authentication string.

After the device sends the NHRP resolution request, the logs show that a retransmission is sent. This is because the device is not seeing the NHRP resolution response so it sends the packet again. As Spoke1 is not seeing the response, it is necessaryto track that packet on the next device in the path, meaning the HUB.

HUB debug output:

*Feb 1 01:31:34.262: NHRP: Receive Resolution Request via Tunnel10 vrf: global(0x0), packet size: 85
*Feb 1 01:31:34.262: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Feb 1 01:31:34.262: shtl: 4(NSAP), sstl: 0(NSAP)
*Feb 1 01:31:34.263: pktsz: 85 extoff: 52
*Feb 1 01:31:34.263: (M) flags: "router auth src-stable nat ", reqid: 10 
*Feb 1 01:31:34.263: src NBMA: 172.21.100.1
*Feb 1 01:31:34.263: src protocol: 10.10.10.1, dst protocol: 10.10.10.2
*Feb 1 01:31:34.263: (C-1) code: no error(0), flags: none
*Feb 1 01:31:34.263: prefix: 0, mtu: 9976, hd_time: 600
*Feb 1 01:31:34.263: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Feb 1 01:31:34.263: Responder Address Extension(3):
*Feb 1 01:31:34.263: Forward Transit NHS Record Extension(4):
*Feb 1 01:31:34.263: Reverse Transit NHS Record Extension(5):
*Feb 1 01:31:34.263: Authentication Extension(7):
*Feb 1 01:31:34.263: type:Cleartext(1), data:DMVPN 
*Feb 1 01:31:34.263: NAT address Extension(9):
*Feb 1 01:31:34.263: NHRP-DETAIL: netid_in = 10, to_us = 0
*Feb 1 01:31:34.263: NHRP-DETAIL: Resolution request for afn 1 received on interface Tunnel10 , for vrf: global(0x0) label: 0
*Feb 1 01:31:34.263: NHRP-DETAIL: Multipath IP route lookup for 10.10.10.2 in vrf: global(0x0) yielded Tunnel10, pfx:10.10.10.0/24 (netid_in:10 if_in:Tunnel10)
*Feb 1 01:31:34.263: NHRP: Route lookup for destination 10.10.10.2 in vrf: global(0x0) yielded interface Tunnel10, prefixlen 24
*Feb 1 01:31:34.263: NHRP-DETAIL: netid_out 10, netid_in 10
*Feb 1 01:31:34.263: NHRP: Forwarding request due to authoritative request.
*Feb 1 01:31:34.263: NHRP-ATTR: NHRP Resolution Request packet is forwarded to 10.10.10.2 using vrf: global(0x0)
*Feb 1 01:31:34.263: NHRP: Attempting to forward to destination: 10.10.10.2 vrf: global(0x0)
*Feb 1 01:31:34.264: NHRP: Forwarding: NHRP SAS picked source: 10.10.10.10 for destination: 10.10.10.2
*Feb 1 01:31:34.264: NHRP: Attempting to send packet through interface Tunnel10 via DEST dst 10.10.10.2
*Feb 1 01:31:34.264: NHRP-DETAIL: First hop route lookup for 10.10.10.2 yielded 10.10.10.2, Tunnel10
*Feb 1 01:31:34.264: NHRP: Forwarding Resolution Request via Tunnel10 vrf: global(0x0), packet size: 105
*Feb 1 01:31:34.264: src: 10.10.10.10, dst: 10.10.10.2
*Feb 1 01:31:34.264: (F) afn: AF_IP(1), type: IP(800), hop: 254, ver: 1
*Feb 1 01:31:34.264: shtl: 4(NSAP), sstl: 0(NSAP)
*Feb 1 01:31:34.264: pktsz: 105 extoff: 52
*Feb 1 01:31:34.264: (M) flags: "router auth src-stable nat ", reqid: 10 
*Feb 1 01:31:34.264: src NBMA: 172.21.100.1
*Feb 1 01:31:34.264: src protocol: 10.10.10.1, dst protocol: 10.10.10.2
*Feb 1 01:31:34.264: (C-1) code: no error(0), flags: none
*Feb 1 01:31:34.264: prefix: 0, mtu: 9976, hd_time: 600
*Feb 1 01:31:34.264: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Feb 1 01:31:34.264: Responder Address Extension(3):
*Feb 1 01:31:34.264: Forward Transit NHS Record Extension(4):
*Feb 1 01:31:34.264: (C-1) code: no error(0), flags: none
*Feb 1 01:31:34.264: prefix: 0, mtu: 9976, hd_time: 600
*Feb 1 01:31:34.264: addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 255
*Feb 1 01:31:34.264: client NBMA: 172.20.10.10
*Feb 1 01:31:34.264: client protocol: 10.10.10.10
*Feb 1 01:31:34.264: Reverse Transit NHS Record Extension(5):
*Feb 1 01:31:34.264: Authentication Extension(7):
*Feb 1 01:31:34.264: type:Cleartext(1), data:DMVPN 
*Feb 1 01:31:34.265: NAT address Extension(9):
*Feb 1 01:31:34.265: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.22.200.2
*Feb 1 01:31:34.265: NHRP: 129 bytes out Tunnel10

Using the value of thereqid, you can observe that the HUB receives the resolution request sent by Spoke1. In the packet, the values of src NBMA and src protocol are the information from Spoke1, and the value of dst protocol is the tunnel IP of Spoke2, as it was seen on the debugs of Spoke1. When the HUB receives the resolution request, it performs a route lookup and forwards the packet to Spoke2. In the forwarded packet, the HUB adds an extension containing its own information (NBMA IP address and tunnel IP address).

The previous debugs show that the HUB is correctly forwarding the resolution request to spoke 2. Therefore, the next step is to confirm Spoke2 is receiving it, processing it correctly, and sending to Spoke1 the resolution reply.

Spoke2 debug output:

-------------------- [IKE/IPSEC DEBUG OUTPUTS OMITTED]--------------------

*Feb 1 01:31:34.647: ISAKMP: (1015):Old State = IKE_QM_IPSEC_INSTALL_AWAIT New State = IKE_QM_PHASE2_COMPLETE
*Feb 1 01:31:34.647: NHRP: Process delayed resolution request src:10.10.10.1 dst:10.10.10.2 vrf: global(0x0)
*Feb 1 01:31:34.648: NHRP-DETAIL: Resolution request for afn 1 received on interface Tunnel10 , for vrf: global(0x0) label: 0
*Feb 1 01:31:34.648: NHRP-DETAIL: Multipath IP route lookup for 10.10.10.2 in vrf: global(0x0) yielded Tunnel10, pfx:10.10.10.0/24 (netid_in:10 if_in:Tunnel10)
*Feb 1 01:31:34.648: NHRP: Route lookup for destination 10.10.10.2 in vrf: global(0x0) yielded interface Tunnel10, prefixlen 24
*Feb 1 01:31:34.648: NHRP-ATTR: smart spoke feature and attributes are not configured
*Feb 1 01:31:34.648: NHRP: Request was to us. Process the NHRP Resolution Request.
*Feb 1 01:31:34.648: NHRP-DETAIL: Multipath IP route lookup for 10.10.10.2 in vrf: global(0x0) yielded Tunnel10, pfx:10.10.10.0/24 (netid_in:10 if_in:Tunnel10)
*Feb 1 01:31:34.648: NHRP: nhrp_rtlookup for 10.10.10.2 in vrf: global(0x0) yielded interface Tunnel10, prefixlen 24, label none(0)
*Feb 1 01:31:34.648: NHRP: Request was to us, responding with ouraddress
*Feb 1 01:31:34.648: NHRP: Checking for delayed event 10.10.10.1/10.10.10.2 on list (Tunnel10 vrf: global(0x0))
*Feb 1 01:31:34.648: NHRP: No delayed event node found.
*Feb 1 01:31:34.648: IPSEC-IFC MGRE/Tu10: Checking to see if we need to delay for src 172.22.200.2 dst 172.21.100.1
*Feb 1 01:31:34.648: IPSEC-IFC MGRE/Tu10: crypto_ss_listen_start already listening
*Feb 1 01:31:34.648: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): Opening a socket with profile IPSEC_Profile_1
*Feb 1 01:31:34.648: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): connection lookup returned 80007F1CE6348C68
*Feb 1 01:31:34.648: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): Socket is already open. Ignoring.
*Feb 1 01:31:34.648: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): connection lookup returned 80007F1CE6348C68
*Feb 1 01:31:34.648: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): tunnel is already open!
*Feb 1 01:31:34.648: NHRP: No need to delay processing of resolution event NBMA src:172.22.200.2 NBMA dst:172.21.100.1
*Feb 1 01:31:34.648: NHRP-MEF: No vendor private extension in NHRP packet
*Feb 1 01:31:34.649: NHRP-CACHE: Tunnel10: Cache update for target 10.10.10.1/32 vrf: global(0x0) label none next-hop 10.10.10.1
*Feb 1 01:31:34.649: 172.21.100.1 (flags:0x2080)
*Feb 1 01:31:34.649: NHRP: Adding Tunnel Endpoints (VPN: 10.10.10.1, NBMA: 172.21.100.1)
*Feb 1 01:31:34.649: IPSEC-IFC MGRE/Tu10: crypto_ss_listen_start already listening
*Feb 1 01:31:34.649: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): Opening a socket with profile IPSEC_Profile_1
*Feb 1 01:31:34.649: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): connection lookup returned 80007F1CE6348C68
*Feb 1 01:31:34.649: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): Found an existing tunnel endpoint
*Feb 1 01:31:34.649: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): tunnel_protection_stop_pending_timer 80007F1CE6348C68
*Feb 1 01:31:34.649: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): Socket is already open. Ignoring.
*Feb 1 01:31:34.653: NHRP: Successfully attached NHRP subblock for Tunnel Endpoints (VPN: 10.10.10.1, NBMA: 172.21.100.1)
*Feb 1 01:31:34.653: NHRP: Peer capability:0
*Feb 1 01:31:34.653: NHRP-CACHE: Inserted subblock node(1 now) for cache: Target 10.10.10.1/32 nhop 10.10.10.1
*Feb 1 01:31:34.653: NHRP-CACHE: Converted internal dynamic cache entry for 10.10.10.1/32 interface Tunnel10 vrf: global(0x0) to external
*Feb 1 01:31:34.653: NHRP-EVE: NHP-UP: 10.10.10.1, NBMA: 172.21.100.1
*Feb 1 01:31:34.653: NHRP-MEF: No vendor private extension in NHRP packet
*Feb 1 01:31:34.653: NHRP-CACHE: Tunnel10: Internal Cache add for target 10.10.10.2/32 vrf: global(0x0) label none next-hop 10.10.10.2
*Feb 1 01:31:34.653: 172.22.200.2 (flags:0x20)
*Feb 1 01:31:34.653: NHRP: Attempting to send packet through interface Tunnel10 via DEST dst 10.10.10.1
*Feb 1 01:31:34.654: NHRP-DETAIL: First hop route lookup for 10.10.10.1 yielded 10.10.10.1, Tunnel10
*Feb 1 01:31:34.654: NHRP: Send Resolution Reply via Tunnel10 vrf: global(0x0), packet size: 133
*Feb 1 01:31:34.654: src: 10.10.10.2, dst: 10.10.10.1
*Feb 1 01:31:34.654: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Feb 1 01:31:34.654: shtl: 4(NSAP), sstl: 0(NSAP)
*Feb 1 01:31:34.654: pktsz: 133 extoff: 60
*Feb 1 01:31:34.654: (M) flags: "router auth dst-stable unique src-stable nat ", reqid: 10
*Feb 1 01:31:34.654: src NBMA: 172.21.100.1
*Feb 1 01:31:34.654: src protocol: 10.10.10.1, dst protocol: 10.10.10.2
*Feb 1 01:31:34.654: (C-1) code: no error(0), flags: none
*Feb 1 01:31:34.654: prefix: 32, mtu: 9976, hd_time: 599
*Feb 1 01:31:34.654: addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 255
*Feb 1 01:31:34.654: client NBMA: 172.22.200.2
*Feb 1 01:31:34.654: client protocol: 10.10.10.2
*Feb 1 01:31:34.654: Responder Address Extension(3):
*Feb 1 01:31:34.654: (C) code: no error(0), flags: none
*Feb 1 01:31:34.654: prefix: 0, mtu: 9976, hd_time: 600
*Feb 1 01:31:34.654: addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 255
*Feb 1 01:31:34.654: client NBMA: 172.22.200.2
*Feb 1 01:31:34.654: client protocol: 10.10.10.2
*Feb 1 01:31:34.654: Forward Transit NHS Record Extension(4):
*Feb 1 01:31:34.654: (C-1) code: no error(0), flags: none
*Feb 1 01:31:34.654: prefix: 0, mtu: 9976, hd_time: 600
*Feb 1 01:31:34.654: addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 255
*Feb 1 01:31:34.654: client NBMA: 172.20.10.10
*Feb 1 01:31:34.654: client protocol: 10.10.10.10
*Feb 1 01:31:34.654: Reverse Transit NHS Record Extension(5):
*Feb 1 01:31:34.654: Authentication Extension(7):
*Feb 1 01:31:34.654: type:Cleartext(1), data:DMVPN
*Feb 1 01:31:34.655: NAT address Extension(9):
*Feb 1 01:31:34.655: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.21.100.1
*Feb 1 01:31:34.655: NHRP: 157 bytes out Tunnel10
*Feb 1 01:31:34.655: IPSEC-IFC MGRE/Tu10(172.22.200.2/172.21.100.1): connection lookup returned 80007F1CE6348C68
*Feb 1 01:31:34.655: NHRP-DETAIL: Deleted delayed event on interfaceTunnel10 dest: 172.21.100.1

The reqidmatches the valueseen in the previous outputs, with this, it is confirmed that the NHRP resolution request packet sent by Spoke1 reaches Spoke2. This packet triggers a route lookup on Spoke2, and realizes that the resolution request is for itself, therefore, Spoke2 adds the information from Spoke1 to its NHRP table. Before sending the resolution reply packet back to Spoke1, the deviceadds its own information (NBMA IP address and tunnel IP address) so that Spoke1 can use that packet to add that information to its database.

Based on all the debugs seen, the NHRP Resolution Reply send from Spoke2 is not arriving to Spoke1. The HUB can be discarded from the issue as it is receiving and forwarding the NHRP Resolution Request packet as expected. Therefore, the next step is to take captures between Spoke1 and Spoke2 to get more details about the issue.

Embedded Packet Capture

The embedded packet capture feature allows you to analyze the traffic passing through the device. The first step to configure it is to create anaccess-list including the traffic you want to capture on both traffic flows (inbound and outbound).

For this scenario, the NBMAIP addresses are used:

ip access-list extended filter
10 permit ip host 172.21.100.1 host 172.22.200.2
20 permit ip host 172.22.200.2 host 172.21.100.1

Then, configure the capture using the commandmonitor capture <CAPTURE_NAME> access-list <ACL_NAME> buffer size 10interface <WAN_INTERFACE> both and start the capture with the command monitor capture <CAPTURE_NAME> start.

Capture configuration on Spoke1 and Spoke2:

monitor capture CAP access-list filter buffer size 10interface GigabitEthernet1 both
monitor capture CAP start

To display the output of the capture use the command show monitor capture <CAPTURE_NAME> buffer brief.

Capture output Spoke1:

SPOKE1#show monitor capture CAP buffer brief
----------------------------------------------------------------------------
#  size  timestamp  source  destination dscp protocol
----------------------------------------------------------------------------
 0 210 0.000000  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 1 150 0.014999  172.21.100.1 -> 172.22.200.2  48 CS6 UDP
 2 478 0.028990  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 3 498 0.049985  172.21.100.1 -> 172.22.200.2  48 CS6 UDP
 4 150 0.069988  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 5 134 0.072994  172.21.100.1 -> 172.22.200.2  48 CS6 UDP
 6 230 0.074993  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 7 230 0.089992  172.21.100.1 -> 172.22.200.2  48 CS6 UDP
 8 118 0.100993  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 9 218 0.108988  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 10  70 0.108988  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 11 218 1.907994  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 12  70 1.907994  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 13 218 5.818003  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 14  70 5.818003  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 15 218  12.559969  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 16  70  12.559969  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 17 218  26.859001  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 18  70  26.859001  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 19 218  54.378978  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 20  70  54.378978  172.21.100.1 -> 172.22.200.2  0 BE  ICMP

Capture output Spoke2:

SPOKE2#show monitor capture CAP buffer brief
----------------------------------------------------------------------------
#  size  timestamp  source  destination dscp protocol
----------------------------------------------------------------------------
 0 210 0.000000  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 1 150 0.015990  172.21.100.1 -> 172.22.200.2  48 CS6 UDP
 2 478 0.027998  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 3 498 0.050992  172.21.100.1 -> 172.22.200.2  48 CS6 UDP
 4 150 0.069988  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 5 134 0.072994  172.21.100.1 -> 172.22.200.2  48 CS6 UDP
 6 230 0.074993  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 7 230 0.089992  172.21.100.1 -> 172.22.200.2  48 CS6 UDP
 8 118 0.099986  172.22.200.2  -> 172.21.100.1 48 CS6 UDP
 9 218 0.108988  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 10  70 0.108988  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 11 218 1.907994  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 12  70 1.909001  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 13 218 5.817011  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 14  70 5.818002  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 15 218  12.559968  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 16  70  12.560960  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 17 218  26.858009  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 18  70  26.859001  172.21.100.1 -> 172.22.200.2  0 BE  ICMP
 19 218  54.378978  172.22.200.2  -> 172.21.100.1 48 CS6 ESP
 20  70  54.379970  172.21.100.1 -> 172.22.200.2  0 BE  ICMP

The output of the captures shows that the initial packets are UDP traffic, indicating the IKE/IPSEC negotiation. After that, Spoke2 sends the resolution reply to Spoke1, which is seen as ESP traffic (packet 9). After this, the expected traffic flow is ESP, however, the next packet seen is ICMP traffic coming from Spoke1 to Spoke2.

To analyze deeper the packet, you can export the pcap file from the device by runningthe commandshow monitor capture <CAPTURE_NAME> buffer dump. Then use adecoder tool to convert the dump output to a pcap file so you can open it with Wireshark.

Note: Cisco has a packet analyzer where you can find capture configuration, examples and a decoder:Cisco TAC Tool - Packet Capture Config Generator and Analyzer

Wireshark output:

Capture Output on Wireshark

The content of the ICMP packet has the error message Destination unreachable (Communication administratively filtered). This indicates thatthere is some kind of filter, such as a router ACL or firewall affecting the traffic along the path. Most the of times, the filter is configured on the device sending the packet (in this case, Spoke1) but middle devices can send it as well.

Note: The Wireshark output is the same on both spokes.

Cisco IOS® XE Datapath Packet Trace Feature

The Cisco IOSXE datapath packet trace featureis used to analyze how the device is processing the traffic. To configure it, you need to create an access-list including the traffic you want to capture on both traffic flows (inbound and outbound).

For this scenario, the NBMA IP addresses are used.

ip access-list extended filter
10 permit ip host 172.21.100.1 host 172.22.200.2
20 permit ip host 172.22.200.2 host 172.21.100.1

Then, configure the fia-trace feature and set the debugging condition to use the access-list. Finally, start the condition.

debug platform packet-trace packet 1024 fia-trace
debug platform condition ipv4 access-list filter both
debug platform condition start

debug platform packet-trace packet <count> fia-trace: enables detailed fia trace, stoppingit once the amountof packets configured have been captured
debug platform condition ipv4 access-list <ACL-NAME> both: setsa condition on the device using the access-list previously configured
debug platform condition start: starts the condition

To review the output of the fia-trace use the next commands.

show platform packet-trace statistics
show platform packet-trace summary
show platform packet-trace packet <number>

Spoke1 show platform packet-trace statistics output:

SPOKE1#show platform packet-trace statistics
Packets Summary
Matched 18
Traced 18
Packets Received
Ingress 11
Inject 7
Count Code Cause
4 2 QFP destination lookup
3 9 QFP ICMP generated packet
Packets Processed
Forward 7
Punt 8
Count Code Cause
5 11 For-us data
3 26 QFP ICMP generated packet
Drop 3
Count Code Cause
3 8 Ipv4Acl
Consume 0

PKT_DIR_IN
Dropped Consumed Forwarded
INFRA 0 0 0
TCP 0 0 0
UDP 0 0 5
IP 0 0 5
IPV6 0 0 0
ARP 0 0 0

PKT_DIR_OUT
Dropped Consumed Forwarded
INFRA 0 0 0
TCP 0 0 0
UDP 0 0 0
IP 0 0 0
IPV6 0 0 0
ARP 0 0 0

In the show platform packet-trace statistics output, you can see the counters for the packets processed by the device. This allows you to see the inbound and outbound packets, and check if the device is dropping any packets, along with the drop reason.

In the output shown, Spoke1 is dropping some packets with the description Ipv4Acl. To further analyze those packets, the command show platform packet-trace summary can be used.

Spoke1 show platform packet-trace summaryoutput:

SPOKE1#show platform packet-trace summary
Pkt  Input  Output State Reason
0  Gi1  internal0/0/rp:0 PUNT  11 (For-us data)
1  INJ.2  Gi1  FWD  
2  Gi1  internal0/0/rp:0 PUNT  11 (For-us data)
3  INJ.2  Gi1  FWD  
4  Gi1  internal0/0/rp:0 PUNT  11 (For-us data)
5  INJ.2  Gi1  FWD  
6  Gi1  internal0/0/rp:0 PUNT  11 (For-us data)
7  INJ.2  Gi1  FWD  
8  Gi1  internal0/0/rp:0 PUNT  11 (For-us data)
9  Gi1  Gi1  DROP  8  (Ipv4Acl)
10 Gi1  internal0/0/recycle:0  PUNT  26 (QFP ICMP generated packet)
11 INJ.9  Gi1  FWD  
12 Gi1  Gi1  DROP  8  (Ipv4Acl)
13 Gi1  internal0/0/recycle:0  PUNT  26 (QFP ICMP generated packet)
14 INJ.9  Gi1  FWD  
15 Gi1  Gi1  DROP  8  (Ipv4Acl)
16 Gi1  internal0/0/recycle:0  PUNT  26 (QFP ICMP generated packet)
17 INJ.9  Gi1  FWD  
18 Gi1  Gi1  DROP  8  (Ipv4Acl)
19 Gi1  internal0/0/recycle:0  PUNT  26 (QFP ICMP generated packet)
20 INJ.9  Gi1  FWD  
21 Gi1  Gi1  DROP  8  (Ipv4Acl)
22 Gi1  internal0/0/recycle:0  PUNT  26 (QFP ICMP generated packet)
23 INJ.9  Gi1  FWD  
24 Gi1  Gi1  DROP  8  (Ipv4Acl)
25 Gi1  internal0/0/recycle:0  PUNT  26 (QFP ICMP generated packet)
26 INJ.9  Gi1  FWD

With this output, you can see each packet arriving and leaving the device, as well as ingress and egress interfaces. The status of the packet is also shown, indicating whether it was forwarded, dropped, or processed internally (punt).

In this example, this output helped to identify the packets that are being dropped by the device. Using the command show platform packet-trace packet <PACKET_NUMBER>, you can see how the device processes that specific packet.

Spoke1 show platform packet-trace packet <PACKET_NUMBER>output:

SPOKE1#show platform packet-trace packet 9
Packet: 9 CBUG ID: 9
Summary
Input : GigabitEthernet1
Output : GigabitEthernet1
State : DROP 8 (Ipv4Acl)
Timestamp
Start : 366032715676920 ns (02/01/2024 04:30:15.708990 UTC)
Stop : 366032715714128 ns (02/01/2024 04:30:15.709027 UTC)
Path Trace
 Feature: IPV4(Input)
 Input : GigabitEthernet1
 Output : <unknown>
 Source : 172.22.200.2
 Destination : 172.21.100.1
 Protocol : 50 (ESP)
 Feature: DEBUG_COND_INPUT_PKT
 Entry : Input - 0x812707d0
  Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 194 ns
 Feature: IPV4_INPUT_DST_LOOKUP_ISSUE
 Entry : Input - 0x8129bf74
  Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 769 ns
 Feature: IPV4_INPUT_ARL_SANITY
 Entry : Input - 0x812725cc
  Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 307 ns
 Feature: EPC_INGRESS_FEATURE_ENABLE
 Entry : Input - 0x812782d0
 Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 6613 ns
 Feature: IPV4_INPUT_DST_LOOKUP_CONSUME
 Entry : Input - 0x8129bf70
 Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 272 ns
 Feature: STILE_LEGACY_DROP
 Entry : Input - 0x812a7650
  Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 278 ns
 Feature: INGRESS_MMA_LOOKUP_DROP
 Entry : Input - 0x812a1278
 Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 697 ns
 Feature: INPUT_DROP_FNF_AOR
 Entry : Input - 0x81297278
 Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 676 ns
 Feature: INPUT_FNF_DROP
 Entry : Input - 0x81280f24
 Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 1018 ns
 Feature: INPUT_DROP_FNF_AOR_RELEASE
 Entry : Input - 0x81297274
 Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 174 ns
 Feature: INPUT_DROP
 Entry : Input - 0x8126e568
 Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 116 ns
 Feature: IPV4_INPUT_ACL
 Entry : Input - 0x81271f70
 Input : GigabitEthernet1
 Output : <unknown>
 Lapsed time : 12915 ns

In the first part, you can see the ingress and egress interface, and the state of the packet. This is followed by the second part of the output where you can find the source and destination IP addresses and the protocol.

Each subsequent phase shows how the device process this particular packet.This offers insights into any configurations like Network Address Translation (NAT) or access-list or other factors that could be impacting it.

In this case, it can be identified that the protocol of the packet is ESP, the source IP is the NBMA IP address of Spoke2, and the destination IP is the NBMA IP address of Spoke1. This indicates that this is the missing packet in the NHRP negotiation. Also, it is observed, that no egress interface is specified in any phase, suggesting that something affected the traffic before it could be forwarded. On the penultimate phase you can see that the device is dropping the inbound traffic on the interface specified (GigabitEthernet1). The last phase shows an input access-list, suggesting that there canbe some configuration on the interface causing the drop.

Note:If after using all the troubleshooting tools listed in this document, the spokes involved in the negotiation are not showing any signs that they are dropping or affecting the traffic, that concludes the troubleshooting on those devices.

The next step must be to check on middle devices between them, such as firewalls, switches, and ISP.

Solution

If such a scenario is seen, the next step is to check the interface shown in the previous outputs.This involves checking the configuration to verify if there is anything affecting the traffic.

WAN interface configuration:

SPOKE1#show running-configuration interface gigabitEthernet1
Building configuration...

Current configuration : 150 bytes
!
interface GigabitEthernet1
ip address 172.21.100.1 255.255.255.0
ip access-group ESP_TRAFFIC in
negotiation auto
no mop enabled
no mop sysid
end

As part of its configuration, the interface has an access-group applied. It is important to verify that the hosts configured on the access-list are not interfering with the traffic used for the NHRP negotiation.

SPOKE1#show access-lists ESP_TRAFFIC
Extended IP access list ESP_TRAFFIC
10 deny esp host 172.21.100.1 host 172.22.200.2
20 deny esp host 172.22.200.2 host 172.21.100.1 (114 matches)
30 permit ip any any (22748 matches)

The second statement of the access-list is denying the communication between the NBMA IP address of Spoke2 and theNBMA IP address of Spoke1, causing the drop previously seen. After removing the access-group from the interface, the communication between the two spokes is successful:

SPOKE1#ping 192.168.2.2 source loopback1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds:
Packet sent with a source address of 192.168.1.1 
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/2/3 ms

The IPSEC tunnel is up and now it is showing encaps and decaps on both devices:

Spoke1:

SPOKE1#show crypto IPSEC sa peer 172.22.200.2

interface: Tunnel10
Crypto map tag: Tunnel10-head-0, local addr 172.21.100.1

protected vrf: (none)
local ident (addr/mask/prot/port): (172.21.100.1/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (172.22.200.2/255.255.255.255/47/0)
current_peer 172.22.200.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 6, #pkts encrypt: 6, #pkts digest: 6
#pkts decaps: 7, #pkts decrypt: 7, #pkts verify: 7
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 172.21.100.1, remote crypto endpt.: 172.22.200.2
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet1
current outbound spi: 0x9392DA81(2475874945)
PFS (Y/N): N, DH group: none

inbound esp sas:
spi: 0xBF8F523D(3213840957)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Transport, }
conn id: 2073, flow_id: CSR:73, sibling_flags FFFFFFFF80000008, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4607998/28783)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0x9392DA81(2475874945)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Transport, }
conn id: 2074, flow_id: CSR:74, sibling_flags FFFFFFFF80000008, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4607999/28783)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

Spoke2:

SPOKE2#show crypto IPSEC sa peer 172.21.100.1

interface: Tunnel10
Crypto map tag: Tunnel10-head-0, local addr 172.22.200.2

protected vrf: (none)
local ident (addr/mask/prot/port): (172.22.200.2/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (172.21.100.1/255.255.255.255/47/0)
current_peer 172.21.100.1 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 7, #pkts encrypt: 7, #pkts digest: 7
#pkts decaps: 6, #pkts decrypt: 6, #pkts verify: 6
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0

local crypto endpt.: 172.22.200.2, remote crypto endpt.: 172.21.100.1
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet1
current outbound spi: 0xBF8F523D(3213840957)
PFS (Y/N): N, DH group: none

inbound esp sas:
spi: 0x9392DA81(2475874945)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Transport, }
conn id: 2073, flow_id: CSR:73, sibling_flags FFFFFFFF80000008, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4607998/28783)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

inbound ah sas:

inbound pcp sas:

outbound esp sas:
spi: 0xBF8F523D(3213840957)
transform: esp-256-aes esp-sha256-hmac ,
in use settings ={Transport, }
conn id: 2074, flow_id: CSR:74, sibling_flags FFFFFFFF80000008, crypto map: Tunnel10-head-0
sa timing: remaining key lifetime (k/sec): (4607999/28783)
IV size: 16 bytes
replay detection support: Y
Status: ACTIVE(ACTIVE)

outbound ah sas:

outbound pcp sas:

Now the DMVPN table of Spoke1 is showing the correct mapping on both entries:

SPOKE1#show dmvpnLegend: Attrb --> S - Static, D - Dynamic, I - Incomplete
 N - NATed, L - Local, X - No Socket
 T1 - Route Installed, T2 - Nexthop-override, B - BGP
 C - CTS Capable, I2 - Temporary
 # Ent --> Number of NHRP entries with same NBMA peer
 NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
 UpDn Time --> Up or Down Time for a Tunnel
==========================================================================
Interface: Tunnel10, IPv4 NHRP Details
Type:Spoke, NHRP Peers:2,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
 1 172.22.200.2  10.10.10.2 UP 00:01:31  D
 1 172.20.10.10 10.10.10.10 UP 1d05h  S

Troubleshoot DMVPN Phase 2 Spoke-to-Spoke Tunnel (2024)

FAQs

How to troubleshoot DMVPN tunnel? ›

Verify the basic connectivity

Ping from the hub to the spoke with NBMA addresses and reverse. These pings must go directly out the physical interface, not through the DMVPN tunnel. ...
Also, use traceroute to check the path that the encrypted tunnel packets take.
Use the debug and show commands to verify no connectivity:

Mar 24, 2023

Show Me More ›

Which common issue causes intermittent DMVPN tunnel flaps? ›

When DMVPN tunnels flap, check the neighborship between the routers as issues with neighborship formation between routers may cause the DMVPN tunnel to flap. In order to resolve this problem, make sure the neighborship between the routers is always up.

Get More Info ›

What is tunnel key in DMVPN? ›

Traffic engineering - The tunnel key can also be used as a parameter to route different types of traffic via different tunnels. You can also use the tunnel keys to enforce specific routing policies.

See Details ›

What is the difference between DMVPN phase 2 and 3? ›

In Phase 2: The traffic goes through the hub until an IPsec tunnel has been formed between the two communicating spokes. In Phase 3: The traffic goes through the hub until the spoke gets an NHRP resolution and the CEF next-hop is overwritten/changed.

Discover More ›

How do I check tunnel status on Cisco router? ›

You can use the standard show interface command on a tunnel interface to see a considerable amount of useful information about it: Router1# show interface Tunnel5 Tunnel5 is up, line protocol is up Hardware is Tunnel Internet address is 192.168.

Tell Me More ›

How do I reset my VPN tunnel? ›

Select. Network. IPSec Tunnels. and select the tunnel you want to refresh or restart.
In the row for that tunnel, under the Status column, click. Tunnel Info. .
At the bottom of the Tunnel Info screen, click the action you want: Refresh. —Updates the onscreen statistics. Restart.

View Details ›

What are the phases of DMVPN? ›

DMVPN Phases Explained

DMVPN Phase 1: All traffic flows through the hub. The hub is used in the network's control and data plane paths. DMVPN Phase 2: Allows spoke-to-spoke tunnels. Spoke-to-spoke communication does not need the hub in the actual data plane.

Discover More ›

How to configure DMVPN on cisco router? ›

Four Steps To Fully Configure Cisco DMVPN

Configure the DMVPN Hub.
Configure the DMVPN Spoke(s)
Protect the mGRE tunnels with IPSecurity (optional)
Configure Routing Between DMVPN mGRE Tunnels (static routing or routing protocol)

View Details ›

What is NBMa in DMVPN? ›

NHRP was designed to advertise network layer information across non broadcast multiaccess networks. NHRP calls these addresses NBMA addresses. Because DMVPN uses NHRP to advertise the underlying network layer addresses, these addresses are called NBMA address or the underlay addresses.

Keep Reading ›

What is DMVPN for dummies? ›

DMVPN (Dynamic Multipoint VPN) is a routing technique we can use to build a VPN network with multiple sites without having to statically configure all devices. It's a “hub and spoke” network where the spokes will be able to communicate with each other directly without having to go through the hub.

Is DMVPN still used? ›

DVPN is still used in many networks. It requires the network administrator to understand and configure IPsec and dynamic routing protocol such as EIGRP.

Explore More ›

What describes DMVPN hub and spoke configuration? ›

Simplified hub/spoke router configuration

With DMVPN, multiple tunnel interfaces for each branch (spoke) VPN are not required. Instead, the simple hub-and-spoke configuration provides on-demand mesh connectivity with dynamic routing and IP Multicast. DMVPN also supports "zero touch" deployment to add more remote sites.

Learn More Now ›

What is a difference between FlexVPN and DMVPN? ›

IPSec: One key difference between FlexVPN and default Dynamic Multipoint VPN (DMVPN) is the protocol used for negotiating IPsec Security Associations (SAs). While DMVPN defaults to using Internet Key Exchange version 1 (IKEv1), FlexVPN utilizes IKEv2.

Learn More Now ›

What is the difference between DMVPN and Sdwan? ›

DMVPN requires more manual configuration and troubleshooting, but it also offers more granular control and flexibility. On the other hand, SD-WAN is simpler and more automated, and it can optimize WAN performance with multiple links and intelligent path selection.

Read The Full Story ›

What is NHRP in DMVPN? ›

Next Hop Resolution Protocol (NHRP) is a resolution protocol that allows a Next Hop Client (NHC) to dynamically register with Next Hop Servers (NHSs). With the Dynamic Multipoint Virtual Private Network (DMVPN) design, the NHC is the spoke router, and the NHS is the hub router.

View Details ›

What is the difference between VPN and DMVPN? ›

VPNs connect each remote site to the company headquarters. A DMVPN creates a mesh VPN topology. In this "hub and spoke" mesh, each remote site's router is configured to connect to the company's VPN hub device to provide access to the required resources.

Discover More ›

How to check VPN tunnel status in checkpoint cli? ›

Run Tunnels on Gateway View

In the SmartView Monitor client, click the Tunnels branch in the Tree View.
In the Tunnels branch (Custom or Predefined), double-click the Tunnels on Gateway view. A list of the Security Gateways shows.
Select the Security Gateway, whose Tunnels and their status you want to see.
Click OK.

How do I enable VPN tunnel? ›

Overview.
Step 1: Create a VPN Gateway.
Step 2: Create a Customer Gateway.
Step 3: Create a VPN Tunnel.
Step 4: Load the Configuration of the Local Gateway.
Step 5: Configure a Routing Table.
Step 6: Activate a VPN Tunnel.

Jan 9, 2024

Discover More ›

Troubleshoot DMVPN Phase 2 Spoke-to-Spoke Tunnel (2024)

Introduction

Prerequisites

Requirements

Components Used

Background Information

Theoretical Background

Topology

Troubleshooting Steps

Initial Validation

Troubleshooting Tools

Useful Commands

Debugs

Embedded Packet Capture

Cisco IOS® XE Datapath Packet Trace Feature

Solution

FAQs

How to troubleshoot DMVPN tunnel? ›

Is DMVPN still used? ›

References