Like I always say, Mist has certainly spoiled me when it comes to troubleshooting; ability to troubleshoot network and Wi-Fi issues remotely and in a shorter time period, find the root cause and come up with a plan to fix the issue have been possible because of what all Juniper Mist offers in their UI. I’ve written up few posts in the past around troubleshooting Mist or Marvis. This particular posts is about an interesting issue that I encountered.
Scenario:
- Two sites connected together via P2P link.
- DHCP server at each location.
- Clients at Site A are getting IPs from the DHCP at Site B after upgrading some access layer switches.
- General DHCP issues.
- And lastly, me having no clue about the site setup.
This is all the information had to go by, I had my call scheduled with the customer and right before the call I was trying to think how best to figure this out. First I needed to understand what is happening exactly. Since I was remote, how am I going to accomplish this?
This is where the Mist AI came in handy.
Step 1 – Packet Captures:
Ability to do pcaps and get dynamic pcaps is just amazing.
I started a wired pcap on all the APs at Site A and filtered all the DHCP messages, because why not.
Got the pcap,let’s check it out.
Step 2 – PCAP Analysis:
I see the DHCP Discover, but right after that I see two DHCP Offer. Notice how they both came from two different DHCP servers.
Let’s dig more into them. Notice how the VLAN showing up is “1” for both of them. This gave me a pretty good idea on what is going on and all I had to do now was validate some connections.
Take a look at the high level overview of the setup below. Even though there were routes on the core and the Switch for 10.10.0.0/24 and 10.20.0.0/24 networks, everything was Layer 2 with Native VLAN1. Which means VLAN1 was stretched between the two sites.
When the clients were sending the DHCP broadcast, it was a broadcast in VLAN1 which included VLAN1 for both sites and since both DHCP servers were in VLAN1 they were receiving that broadcast and responding with an offer.
Ability to analyze pcaps to find out the root cause and troubleshoot is a crucial skill and there is so much to learn there always, and ability to get pcaps without having to go onsite is ♥️. Hopefully, this post will be helpful to someone else. Would love to hear your thoughts if you have made it this far. Happy Troubleshooting.