Whenever I needed to trace physical connections from a switch to a server (and vice versa), I mostly used the show mac-address table command on the Catalyst switches. This would show which MAC address is connected to which port on the switch.
By using finding out which MAC address belongs to which interface (on HP ILO you would see the MAC addresses assigned to the NICs or on Linux you can run ip link) on the server side I was able to trace down how exactly the servers were connected to the switches.
This worked fine in the past on Cisco Catalyst switches. But then I ran into a problem with newer Nexus switches.
On the physical server, I first needed to find the MAC addresses of the two physical NICs being connected as a bonding interface to the switch's port channel. I've used ip link, as mentioned before, to find this:
ck@bookworm:~$ ip link | egrep "enp1[1,2]" -A1
2: enp11s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether aa:bb:cc:dd:ee:ef brd ff:ff:ff:ff:ff:ff
3: enp12s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether aa:bb:cc:dd:ee:ef brd ff:ff:ff:ff:ff:ff permaddr aa:bb:cc:dd:ee:ff
Note: The Cisco CIMC did not show the MAC addresses for this PCIe module (UCS VIC 1225).
With the information of the MAC address, let's find out on the Nexus switches to which interface this MAC address is connected to:
nexus1# show mac address-table | grep 252a
* 10 aabb.ccdd.eeef dynamic 6960 F F Po2
Umm... the MAC address was found and matches the one seen on the Linux server. But instead of telling me to which physical interface this is connected on this switch, I only see that it's connected to the Po2 / port-channel 2 interface. D'uh! I know that (I did the LACP configuration on the Debian server), but still need to find out which server NIC is connected to which switch interface.
On my research I came across an interesting article (Tracing a Layer 2 Path on Cisco Nexus Switches), which introduced me to the Cisco Discovery Protocol - cdp in short. From the documentation on the Cisco learning network, CDP is described as:
CDP is a Cisco proprietary protocol that is used for collecting directly connected neighbor device information like hardware, software, device name details and many more...
Cool. Sounds as if this does automatically what I've used to do in the past (manually do detective work and hunt down mac addresses)!
Let's put this to the test and show the connected neighbors on this Port-Channel 2 (po2)!
First let's find out which switch interfaces are members of this Po2 channel:
nexus1# sh port-channel sum | grep Po2
2 Po2(SU) Eth LACP Eth1/2(P)
So this would be interface Eth1/2 on each switch (stacked switch setup).
Let's see if the mentioned interface shows up in the local CDP database:
nexus1# sh cdp ne | grep Eth1/2
nexus1#
Nothing! Interestingly the full output of sh cdp ne showed only RJ45 connections, using a Gbit SFP transceiver. But not a single interface using a DAC cable connected to a 10G SFP+ transceiver showed up in the output!
After another round of research, it seems that CDP does not work on etherchannels (the terminology used in older Catalyst switches):
This is a known issue and is documented as an internal cosmetic bug. Logically speaking, CDP is not supported on etherchannel interfaces
Hmm.. not entirely sure if this is still true today, because sh cdp ne shows the server interfaces behind the port-channel interfaces using a Gbit SFP.
Was it time to give up yet?
Yet another round of research, this time in one of these huge release notes PDFs from the Cisco Nexus 3000, led me (by chance) to an important hint while searching for the string "nei" (looking for neighbor information):
Looks like the show lacp command also allows to show the neighbor information? Could this be the missing piece? Let's find out:
nexus1# show lacp neighbor interface port-channel2
Flags: S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode
port-channel2 neighbors
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/2 65535,a-bb-cc-dd-ee-ef 0x2 176734 SA
LACP Partner Partner Partner
Port Priority Oper Key Port State
255 0xf 0x3f
nexus2# show lacp neighbor interface port-channel2
Flags: S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode
port-channel2 neighbors
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/2 65535,a-bb-cc-dd-ee-ef 0x1 176731 SA
LACP Partner Partner Partner
Port Priority Oper Key Port State
255 0xf 0x3f
The output does indeed reveal the MAC address of the Linux server! Finally!
Now there's only one problem: The MAC address of the bonded interface is the same on both switch interfaces (see the Linux ip link output at the beginning of the post). But luckily the sh lacp nei output shows an additional column "Port Number". On the first switch (nexus1) we can see the port is 0x2 (second port on the server NIC) while on the second switch (nexus2) the port indicates 0x1 (first port).
This way I was finally able to successfully trace the physical layer 2 connection from the Nexus switch interfaces to the Cisco server running Linux!
Although CDP did not help in tracing the SFP+ connections, CDP worked surprisingly well for the physical connections using a Gbit SFP and a CAT6 RJ45 connection.
The switch side shows the connections to the servers and their corresponding network interface:
switch>show cdp nei
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
[...]
server2.example.com
Gig 1/0/14 152 R T Debian GN enp4s0f1
server2.example.com
Gig 1/0/13 152 R T Debian GN enp4s0f0
server1.example.com
Gig 1/0/15 152 R T Debian GN enp4s0f0
server1.example.com
Gig 1/0/16 152 R T Debian GN enp4s0f1
[...]
But CDP is not only giving helpful information on the switch side. On the server side (Debian 12) the ip link output also shows the corresponding switch interface!
ck@debian:~$ ip link show enp4s0f0
4: enp4s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
alias connected to switch (Gi1/0/13)
This is pretty cool (and very helpful)! But, as mentioned before, only seems to work on RJ45 connections.
After talking with a Cisco network engineer about that issue I got some interesting feedback: The Linux server needs to have a specific service (ladvd) running in order to make CDP work correctly over LACP/bond interfaces with SFP+ transceivers!
On a Debian 12, this package can be installed right from the Debian repos:
root@bookworm:~# apt-cache search ladvd
ladvd - LLDP/CDP sender
root@bookworm:~# apt-cache show ladvd
Package: ladvd
Version: 1.1.2-2
Installed-Size: 134
Maintainer: Debian QA Group <packages@qa.debian.org>
Architecture: amd64
Depends: libbsd0 (>= 0.6.0), libc6 (>= 2.34), libcap-ng0 (>= 0.7.9), libevent-2.1-7 (>= 2.1.8-stable), libmnl0 (>= 1.0.3-4~), libpcap0.8 (>= 1.5.1), libpci3 (>= 1:3.5.1), libteam5 (>= 1.7), adduser, pciutils
Description-en: LLDP/CDP sender
ladvd sends link layer advertisements on all available interfaces.
This makes connected hosts visible on managed switches. By default it
will run as a privilege-separated daemon.
Description-md5: bd87495c24188621bf5a5d4799e76d71
Multi-Arch: foreign
Homepage: https://github.com/sspans/ladvd/
Tag: implemented-in::c, interface::daemon, role::program,
works-with::network-traffic
Section: net
Priority: optional
Filename: pool/main/l/ladvd/ladvd_1.1.2-2_amd64.deb
Size: 47800
MD5sum: b62ce7940a10e8d64a1811c2bc92a006
SHA256: 9ce1b53b07897b726ba1bf4045151e2d13f0b945480a2b59a8637668c5bf1a65
root@bookworm:~# apt install ladvd
After installing the ladvd package, the ladvd.service is now running in the background.
After roughly one minute, both the Linux side and the Nexus side showed the physical connection between the interfaces:
root@bookworm:~# ip link | egrep "enp1[1,2]" -A2
2: enp11s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether aa:bb:cc:dd:ee:ef brd ff:ff:ff:ff:ff:ff
alias connected to nexus2(XXXXXXXXXXX) (Eth1/2)
3: enp12s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether aa:bb:cc:dd:ee:ef brd ff:ff:ff:ff:ff:ff permaddr aa:bb:cc:dd:ee:ff
alias connected to nexus1(XXXXXXXXXXX) (Eth1/2)
nexus1# sh cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID Local Intrfce Hldtme Capability Platform Port ID
server2.example.com
Eth1/2 167 R T Debian GNU/Li enp12s0
[...]
This means: With the ladvd service running on the (Debian) Linux server, the CDP exchange works correctly, even with LACP port-channels and SFP+ modules!
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder