Netonix to Cisco LACP issue?

[aaron@futuretk.com]

I'm troubleshooting an odd disconnect issue with one of 5 Netonix Switches in the same rack. The Netonix Switch has been replaced. The general behavior is that I loose connection to to the switch over the LACP bond and historically I've had a tech reboot the switch and it all comes right back up. Today I was near by and was able to connect directly to the switch to further diagnose.

Netonix WS-12-400-AC on 1.4.9 firmware and dual Fiber SFP modules
Cisco 6500 Switch with dual 16 port WS-X6416-GBIC cards and Cisco fiber GBIC modules

During the event the Cisco log showed simultaneous disconnects and reconnects of both ports facing the offending Netonix on roughly 20 second and 10 second intervals.

.Feb 22 19:33:33.457: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to up
.Feb 22 19:33:33.533: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to up
Feb 22 19:33:33.461: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to up
Feb 22 19:33:33.533: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to up
.Feb 22 19:33:52.385: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to down
.Feb 22 19:33:52.613: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to down
Feb 22 19:33:52.385: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to down
Feb 22 19:33:52.613: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to down
.Feb 22 19:34:03.189: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to up
.Feb 22 19:34:03.769: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to up
Feb 22 19:34:03.185: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to up
Feb 22 19:34:03.765: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to up
.Feb 22 19:34:22.509: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to down
.Feb 22 19:34:22.697: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to down
Feb 22 19:34:22.513: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to down
Feb 22 19:34:22.717: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to down

On the Netonix I showed both SFP ports up and connected at 1G, but the LACP status was Not Active on both ports. I disabled both SFP ports, clicked Save/Apply, enabled both SFP ports, clicked Save/Apply, and a few seconds later it came back up and started working again.

This is the Netonix logs from when I got up there and plugged my laptop into port 9, through when I unplugged it. There were no log entries in the previous 48 hours leading up to the event.

Feb 22 13:33:36 Port: link state changed to 'up' (10M-F) on port 9
Feb 22 13:33:36 STP: set port 9 to discarding
Feb 22 13:33:37 Port: link state changed to 'down' on port 9
Feb 22 13:33:40 Port: link state changed to 'up' (1G) on port 9
Feb 22 13:33:40 STP: set port 9 to discarding
Feb 22 13:33:43 STP: set port 9 to learning
Feb 22 13:33:43 STP: set port 9 to forwarding
Feb 22 13:34:11 UI: User 'admin' logged in from 10.1.1.222
Feb 22 13:34:54 UI: Configuration changed by admin (10.1.1.222)
Feb 22 13:34:54 UI: Port 13 Enable: changed from 'Enabled' to 'Disabled'
Feb 22 13:34:54 UI: Port 14 Enable: changed from 'Enabled' to 'Disabled'
Feb 22 13:35:09 monitor: restarting vtss_appl
Feb 22 13:35:11 STP: set port 1 to discarding
Feb 22 13:35:11 STP: set port 4 to discarding
Feb 22 13:35:11 STP: set port 9 to discarding
Feb 22 13:35:13 Port: link state changed to 'up' (1G) on port 14
Feb 22 13:35:13 Port: link state changed to 'up' (1G) on port 14
Feb 22 13:35:14 STP: set port 9 to learning
Feb 22 13:35:14 STP: set port 9 to forwarding
Feb 22 13:35:14 STP: set port 4 to learning
Feb 22 13:35:14 STP: set port 4 to forwarding
Feb 22 13:35:14 STP: set port 1 to learning
Feb 22 13:35:14 STP: set port 1 to forwarding
Feb 22 13:35:14 UI: Configuration changed by admin (10.1.1.222)
Feb 22 13:35:14 UI: Port 13 Enable: changed from 'Disabled' to 'Enabled'
Feb 22 13:35:14 UI: Port 14 Enable: changed from 'Disabled' to 'Enabled'
Feb 22 13:35:20 LACP: starting negotiation with partner 00-18-74-97-24-00
Feb 22 13:35:24 LACP: LACP changed state to Active on port 13 (key 4)
Feb 22 13:35:25 LACP: LACP changed state to Active on port 14 (key 4)
Feb 22 13:38:45 sSMTP[3209]: Sent mail for SharpSwitch4@futuretk.com (221 smtp.ellipse.net Mail Server ** ABUSE IS STRICTLY PROHIBITED **) uid=0 username=admin outbytes=7378
Feb 22 13:39:39 Port: link state changed to 'down' on port 9
Feb 22 13:39:39 STP: set port 9 to discarding

These are the Cisco Logs during the same time period of time.(Don't mind the 6 hours Time Zone setting difference)

.Feb 22 19:34:57.117: %LINK-3-UPDOWN: Interface GigabitEthernet8/4, changed state to down
.Feb 22 19:34:57.381: %LINK-3-UPDOWN: Interface GigabitEthernet9/4, changed state to down
Feb 22 19:34:57.116: %LINK-SP-3-UPDOWN: Interface GigabitEthernet8/4, changed state to down
Feb 22 19:34:57.380: %LINK-SP-3-UPDOWN: Interface GigabitEthernet9/4, changed state to down
.Feb 22 19:35:17.026: %LINK-3-UPDOWN: Interface GigabitEthernet8/4, changed state to up
.Feb 22 19:35:17.946: %LINK-3-UPDOWN: Interface GigabitEthernet9/4, changed state to up
Feb 22 19:35:16.974: %LINK-SP-3-UPDOWN: Interface GigabitEthernet8/4, changed state to up
Feb 22 19:35:17.950: %LINK-SP-3-UPDOWN: Interface GigabitEthernet9/4, changed state to up
.Feb 22 19:35:21.970: %LINK-3-UPDOWN: Interface Port-channel4, changed state to up
.Feb 22 19:35:22.094: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel4, changed state to up
.Feb 22 19:35:22.098: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to up
Feb 22 19:35:22.070: %LINK-SP-3-UPDOWN: Interface Port-channel4, changed state to up
.Feb 22 19:35:22.778: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to up
Feb 22 19:35:22.074: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet8/4, changed state to up
Feb 22 19:35:22.202: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface Port-channel4, changed state to up
Feb 22 19:35:22.754: %LINEPROTO-SP-5-UPDOWN: Line protocol on Interface GigabitEthernet9/4, changed state to up

I have attached a quick drawing of how the two devices are connected. All 5 Netonix switches are connected in the same way. SFP13 connects to a port on Module 8, and SFP14 connects to a port on Module 9. This issue is only impacting this one switch in the stack.

I've also attached a screen shots of the Status, Ports, and LAG tabs in the Netonix.

Thank you
Aaron Clark

[sirhc]

I would try:

1) Try setting the speed / duplex to AUTO on the SFP modules preferably in both Netonix and Cisco

2) swap out the SFP modules in the switch and if no love swap out the modules in the CISCO