Tracing a layer 2 link on a Cisco Nexus switch with port-channel interfaces using SFP+ transceivers

Written by - 0 comments

Published on - last updated on July 17th 2024 - Listed in Network Hardware


Whenever I needed to trace physical connections from a switch to a server (and vice versa), I mostly used the show mac-address table command on the Catalyst switches. This would show which MAC address is connected to which port on the switch.

By using finding out which MAC address belongs to which interface (on HP ILO you would see the MAC addresses assigned to the NICs or on Linux you can run ip link) on the server side I was able to trace down how exactly the servers were connected to the switches.

This worked fine in the past on Cisco Catalyst switches. But then I ran into a problem with newer Nexus switches.

No physical interfaces, only port-channels

On the physical server, I first needed to find the MAC addresses of the two physical NICs being connected as a bonding interface to the switch's port channel. I've used ip link, as mentioned before, to find this:

ck@bookworm:~$ ip link | egrep "enp1[1,2]" -A1
2: enp11s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether aa:bb:cc:dd:ee:ef brd ff:ff:ff:ff:ff:ff
3: enp12s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether aa:bb:cc:dd:ee:ef brd ff:ff:ff:ff:ff:ff permaddr aa:bb:cc:dd:ee:ff

Note: The Cisco CIMC did not show the MAC addresses for this PCIe module (UCS VIC 1225).

With the information of the MAC address, let's find out on the Nexus switches to which interface this MAC address is connected to:

nexus1# show mac address-table | grep 252a
* 10       aabb.ccdd.eeef    dynamic   6960       F    F  Po2

Umm... the MAC address was found and matches the one seen on the Linux server. But instead of telling me to which physical interface this is connected on this switch, I only see that it's connected to the Po2 / port-channel 2 interface. D'uh! I know that (I did the LACP configuration on the Debian server), but still need to find out which server NIC is connected to which switch interface.

Cisco Discovery Protocol (CDP) to the help?

On my research I came across an interesting article (Tracing a Layer 2 Path on Cisco Nexus Switches), which introduced me to the Cisco Discovery Protocol - cdp in short. From the documentation on the Cisco learning network, CDP is described as:

CDP is a Cisco proprietary protocol that is used for collecting directly connected neighbor device information like hardware, software, device name details and many more...

Cool. Sounds as if this does automatically what I've used to do in the past (manually do detective work and hunt down mac addresses)!

Let's put this to the test and show the connected neighbors on this Port-Channel 2 (po2)!

First let's find out which switch interfaces are members of this Po2 channel:

nexus1# sh port-channel sum | grep Po2
2     Po2(SU)     Eth      LACP      Eth1/2(P)

So this would be interface Eth1/2 on each switch (stacked switch setup).

Let's see if the mentioned interface shows up in the local CDP database:

nexus1# sh cdp ne | grep Eth1/2
nexus1#

Nothing! Interestingly the full output of sh cdp ne showed only RJ45 connections, using a Gbit SFP transceiver. But not a single interface using a DAC cable connected to a 10G SFP+ transceiver showed up in the output!

After another round of research, it seems that CDP does not work on etherchannels (the terminology used in older Catalyst switches):

This is a known issue and is documented as an internal cosmetic bug. Logically speaking, CDP is not supported on etherchannel interfaces

Hmm.. not entirely sure if this is still true today, because sh cdp ne shows the server interfaces behind the port-channel interfaces using a Gbit SFP.

Was it time to give up yet?

Time to give up yet? Never!

LACP config knows the information

Yet another round of research, this time in one of these huge release notes PDFs from the Cisco Nexus 3000, led me (by chance) to an important hint while searching for the string "nei" (looking for neighbor information):

Looks like the show lacp command also allows to show the neighbor information? Could this be the missing piece? Let's find out:

nexus1# show lacp neighbor interface port-channel2
Flags:  S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
port-channel2 neighbors
Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/2      65535,a-bb-cc-dd-ee-ef 0x2             176734      SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            255                    0xf                         0x3f

nexus2# show lacp neighbor interface port-channel2
Flags:  S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
port-channel2 neighbors
Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/2      65535,a-bb-cc-dd-ee-ef 0x1             176731      SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            255                    0xf                         0x3f

The output does indeed reveal the MAC address of the Linux server! Finally!

Now there's only one problem: The MAC address of the bonded interface is the same on both switch interfaces (see the Linux ip link output at the beginning of the post). But luckily the sh lacp nei output shows an additional column "Port Number". On the first switch (nexus1) we can see the port is 0x2 (second port on the server NIC) while on the second switch (nexus2) the port indicates 0x1 (first port).

This way I was finally able to successfully trace the physical layer 2 connection from the Nexus switch interfaces to the Cisco server running Linux!

Something else I learned (about CDP)

Although CDP did not help in tracing the SFP+ connections, CDP worked surprisingly well for the physical connections using a Gbit SFP and a CAT6 RJ45 connection.

The switch side shows the connections to the servers and their corresponding network interface:

switch>show cdp nei
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
[...]
server2.example.com
                 Gig 1/0/14        152              R T   Debian GN enp4s0f1
server2.example.com
                 Gig 1/0/13        152              R T   Debian GN enp4s0f0

server1.example.com
                 Gig 1/0/15        152              R T   Debian GN enp4s0f0
server1.example.com
                 Gig 1/0/16        152              R T   Debian GN enp4s0f1
[...]

But CDP is not only giving helpful information on the switch side. On the server side (Debian 12) the ip link output also shows the corresponding switch interface!

ck@debian:~$ ip link show enp4s0f0
4: enp4s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond1 state UP mode DEFAULT group default qlen 1000
    link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
    alias connected to switch (Gi1/0/13)

This is pretty cool (and very helpful)! But, as mentioned before, only seems to work on RJ45 connections.

Making CDP work with LACP/bond interfaces (with SFP+)

After talking with a Cisco network engineer about that issue I got some interesting feedback: The Linux server needs to have a specific service (ladvd) running in order to make CDP work correctly over LACP/bond interfaces with SFP+ transceivers!

On a Debian 12, this package can be installed right from the Debian repos:

root@bookworm:~# apt-cache search ladvd
ladvd - LLDP/CDP sender

root@bookworm:~# apt-cache show ladvd
Package: ladvd
Version: 1.1.2-2
Installed-Size: 134
Maintainer: Debian QA Group <packages@qa.debian.org>
Architecture: amd64
Depends: libbsd0 (>= 0.6.0), libc6 (>= 2.34), libcap-ng0 (>= 0.7.9), libevent-2.1-7 (>= 2.1.8-stable), libmnl0 (>= 1.0.3-4~), libpcap0.8 (>= 1.5.1), libpci3 (>= 1:3.5.1), libteam5 (>= 1.7), adduser, pciutils
Description-en: LLDP/CDP sender
 ladvd sends link layer advertisements on all available interfaces.
 This makes connected hosts visible on managed switches. By default it
 will run as a privilege-separated daemon.
Description-md5: bd87495c24188621bf5a5d4799e76d71
Multi-Arch: foreign
Homepage: https://github.com/sspans/ladvd/
Tag: implemented-in::c, interface::daemon, role::program,
 works-with::network-traffic
Section: net
Priority: optional
Filename: pool/main/l/ladvd/ladvd_1.1.2-2_amd64.deb
Size: 47800
MD5sum: b62ce7940a10e8d64a1811c2bc92a006
SHA256: 9ce1b53b07897b726ba1bf4045151e2d13f0b945480a2b59a8637668c5bf1a65

root@bookworm:~# apt install ladvd

After installing the ladvd package, the ladvd.service is now running in the background.

After roughly one minute, both the Linux side and the Nexus side showed the physical connection between the interfaces:

root@bookworm:~# ip link | egrep "enp1[1,2]" -A2
2: enp11s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether aa:bb:cc:dd:ee:ef brd ff:ff:ff:ff:ff:ff
    alias connected to nexus2(XXXXXXXXXXX) (Eth1/2)
3: enp12s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether aa:bb:cc:dd:ee:ef brd ff:ff:ff:ff:ff:ff permaddr aa:bb:cc:dd:ee:ff
    alias connected to nexus1(XXXXXXXXXXX) (Eth1/2)

nexus1# sh cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute


Device-ID             Local Intrfce Hldtme Capability  Platform      Port ID
server2.example.com
                    Eth1/2         167    R T       Debian GNU/Li enp12s0       
[...]

This means: With the ladvd service running on the (Debian) Linux server, the CDP exchange works correctly, even with LACP port-channels and SFP+ modules!


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder