I have been trying to figure this out for a while, this happens on a few switches in the network at times. We have swapped out switches when this has happened in the past and have had the same results with the new switches. I am going to focus on the latest one to start having this problem below.
Yesterday, one of the switches we have had been working just fine until around noon when all but a few ports stopped passing traffic. We assumed this was an STP issue but if we touched anything on the switch or changed any settings (other than power cycle) the switch would still pass traffic between the ports that did not drop. during this time, we could ping the switch, but the web GUI would not load and nothing done via ssh helped.
after that first lockup, we power cycled the switch and everything came back up, but the problem reoccurred every few hours until earlier today when we turned off STP. After turning off STP, the switch does not lock up like it was, but every once in a while a group of ports seem to drop power and the devices (Ubiquiti 5ac gen2 prism) then reconnect.
Switch info:
switch firmware: 1.5.1rc8
model: WS-12-250-AC
Before STP disable:
In the example from the logs below, you can see that the ports states are changed to "down" and STP sets to discard, yet power and link lights stay up and the web gui shows the ports as up
- Code: Select all
Dec 31 19:10:23 Port: link state changed to 'down' on port 11
Dec 31 19:10:23 STP: msti 0 set port 11 to discarding
Dec 31 19:10:24 Port: link state changed to 'down' on port 2
Dec 31 19:10:24 STP: msti 0 set port 2 to discarding
Dec 31 19:10:24 Port: link state changed to 'down' on port 3
Dec 31 19:10:24 STP: msti 0 set port 3 to discarding
Dec 31 19:10:24 Port: link state changed to 'down' on port 4
Dec 31 19:10:24 STP: msti 0 set port 4 to discarding
Dec 31 19:10:24 Port: link state changed to 'down' on port 5
Dec 31 19:10:24 STP: msti 0 set port 5 to discarding
Dec 31 19:10:24 Port: link state changed to 'down' on port 6
Dec 31 19:10:24 STP: msti 0 set port 6 to discarding
Dec 31 19:10:26 Port: link state changed to 'up' (100M-F) on port 4
Dec 31 19:10:26 STP: msti 0 set port 4 to discarding
Dec 31 19:10:27 Port: link state changed to 'up' (100M-F) on port 2
After STP disable:
after STP had been disabled, the switch does not lock up, but groups of ports seem to loose power and come back up. part of the time when this happens, the switch seems to be only half aware of the port changes.
- Code: Select all
Dec 31 17:24:37 Port: link state changed to 'down' on port 2
Dec 31 17:24:37 Port: link state changed to 'down' on port 3
Dec 31 17:24:37 Port: link state changed to 'down' on port 4
Dec 31 17:24:37 Port: link state changed to 'down' on port 5
Dec 31 17:24:37 Port: link state changed to 'down' on port 6
Dec 31 17:24:37 Port: link state changed to 'down' on port 7
Dec 31 17:24:37 Port: link state changed to 'down' on port 11
Dec 31 17:24:39 Port: link state changed to 'up' (100M-F) on port 4
Dec 31 17:24:39 Port: link state changed to 'up' (100M-F) on port 5
Dec 31 17:24:40 Port: link state changed to 'up' (100M-F) on port 6
Dec 31 17:24:40 Port: link state changed to 'up' (100M-F) on port 7
Dec 31 17:24:40 Port: link state changed to 'up' (100M-F) on port 3
Dec 31 17:24:40 Port: link state changed to 'up' (100M-F) on port 11
Dec 31 17:24:40 Port: link state changed to 'up' (100M-F) on port 2
Dec 31 17:24:59 Port: link state changed to 'down' on port 11
Dec 31 17:24:59 Port: link state changed to 'down' on port 4
Dec 31 17:24:59 Port: link state changed to 'down' on port 3
Dec 31 17:24:59 Port: link state changed to 'down' on port 6
Dec 31 17:24:59 Port: link state changed to 'down' on port 2
Dec 31 17:24:59 Port: link state changed to 'down' on port 7
Dec 31 17:24:59 Port: link state changed to 'down' on port 5
Dec 31 17:25:03 Port: link state changed to 'up' (1G) on port 11
Dec 31 17:25:03 Port: link state changed to 'up' (1G) on port 3
Dec 31 17:25:03 Port: link state changed to 'up' (1G) on port 6
Dec 31 17:25:03 Port: link state changed to 'up' (1G) on port 2
Dec 31 17:25:03 Port: link state changed to 'up' (1G) on port 4
Dec 31 17:25:03 Port: link state changed to 'up' (1G) on port 5
Dec 31 17:25:03 Port: link state changed to 'up' (1G) on port 7
Dec 31 20:57:38 Port: link state changed to 'down' on port 11
Dec 31 20:57:38 Port: link state changed to 'down' on port 2
Dec 31 20:57:38 Port: link state changed to 'down' on port 3
Dec 31 20:57:38 Port: link state changed to 'down' on port 4
Dec 31 20:57:38 Port: link state changed to 'down' on port 5
Dec 31 20:57:38 Port: link state changed to 'down' on port 6
Dec 31 20:57:38 Port: link state changed to 'down' on port 10
Dec 31 20:57:41 Port: link state changed to 'up' (100M-F) on port 11
Dec 31 20:57:41 Port: link state changed to 'up' (100M-F) on port 3
Dec 31 20:57:41 Port: link state changed to 'up' (100M-F) on port 4
Dec 31 20:57:41 Port: link state changed to 'up' (100M-F) on port 5
Dec 31 20:57:41 Port: link state changed to 'up' (100M-F) on port 10
Dec 31 20:57:41 Port: link state changed to 'up' (100M-F) on port 2
Dec 31 20:57:41 switch[3849]: unexpected link change on port 11 (170.60.pkr) from 1G to 100M-F
Dec 31 20:57:42 Port: link state changed to 'up' (100M-F) on port 6
Dec 31 20:57:42 switch[3851]: unexpected link change on port 3 (prkr.60.60) from 1G to 100M-F
Dec 31 20:57:43 switch[3853]: unexpected link change on port 4 (prkr.120.60) from 1G to 100M-F
Dec 31 20:57:43 switch[3855]: unexpected link change on port 5 (prkr.180.60) from 1G to 100M-F
Dec 31 20:57:44 switch[3857]: unexpected link change on port 2 (prkr.0.60) from 1G to 100M-F
Dec 31 20:57:44 switch[3859]: unexpected link change on port 6 (prkr.240.60) from 1G to 100M-F
Dec 31 20:58:01 Port: link state changed to 'down' on port 11
Dec 31 20:58:01 Port: link state changed to 'down' on port 4
Dec 31 20:58:01 Port: link state changed to 'down' on port 3
Dec 31 20:58:01 Port: link state changed to 'down' on port 6
Dec 31 20:58:01 Port: link state changed to 'down' on port 2
Dec 31 20:58:01 Port: link state changed to 'down' on port 10
Dec 31 20:58:01 Port: link state changed to 'down' on port 5
Dec 31 20:58:04 Port: link state changed to 'up' (1G) on port 6
Dec 31 20:58:04 Port: link state changed to 'up' (1G) on port 4
Dec 31 20:58:04 Port: link state changed to 'up' (1G) on port 10
Dec 31 20:58:05 Port: link state changed to 'up' (1G) on port 3
Dec 31 20:58:05 Port: link state changed to 'up' (1G) on port 5
Dec 31 20:58:05 Port: link state changed to 'up' (1G) on port 11
Dec 31 20:58:05 Port: link state changed to 'up' (1G) on port 2
If the ports were disconnecting and reconnecting at random times, I would suspect a configuration problem but with them dropping in unison I am not sure what to think. I had been reading about others problems with STP and FC, but I have not heard of the actual ports cycling power. Doing a cable diagnostic on the ports show as being good. Anyone have suggestions as to what could be going on?
