Constant "set port forwarding / learning / discarding"

[cwachs]

Last night, our tower WS-12-250 AC starting having an issue on one port and I can't figure out why. Every few minutes, I get this:


Aug 30 07:30:30 STP: set port 7 to discarding
Aug 30 07:30:31 STP: set port 7 to learning
Aug 30 07:30:31 STP: set port 7 to forwarding
Aug 30 07:31:23 STP: set port 7 to discarding
Aug 30 07:31:23 STP: set port 7 to learning
Aug 30 07:31:23 STP: set port 7 to forwarding
Aug 30 07:33:12 STP: set port 7 to discarding
Aug 30 07:33:12 STP: set port 7 to learning
Aug 30 07:33:12 STP: set port 7 to forwarding

Port 7 has been doing this since just after midnight. It filled our SYSLOG which is how we discovered it this morning. The radio attached to that port is working, power never goes down. I can ping the radio just fine but every now and again, a ping is lost or takes a long time to reply - which may be in line with the port changing states.

We turned off flow control on that port first. No change to behavior. We then set the port to 1G from Auto. No change in behavior. It is connected to and powering a Cambium ePMP 1000 GPS radio.

Switch is running firmware 1.4.2. Power and temps all look fine. It is in a climate controlled building. No power issues and is on a big UPS. The equipment is all well grounded. It's been running fine for months until last night with nothing new on our end.

What will cause these ports to do this?

[sirhc]

When you have an issue PLEASE always upgrade to "latest" code which is currently v1.4.3rc7 and see if issue still exists.

There are major BUGS in v1.4.2 - READ RELEASE NOTES

I am not saying that upgrading to v1.4.3rc7 will fix this issue but we do not investigate bugs in close firmware which v1.4.2 is closed and no longer being developed so what we need to know is if there is a problem in the current code that is being developed and if there is we can address that.

If upgrading to v1.4.3rc7 does not fix your issue we can then start investigating it but I would say the Ethernet link is dropping possibly from a lose connector or bad end or the ETherent port in the radio is going bad or the switch port which you can test by moving the cable to another port but first I would re-seat the connector in case it is a bad connection.

BUT PLEASE UPGRADE TO v1.4.3rc7 I DO NOT KNOW HOW MUCH BETTER I CAN EXPLAIN THAT THERE ARE VERY BAD BUGS IN v1.4.2

[cwachs]

Changed ports on the switch, the jumper between the switch and our Ethernet surge protector (I know you hate them) and upgraded to 1.4.3rc7. So far in 3 hours, no flaps. I'll keep my fingers crossed through the night. Always fun to wake up to 300 emails from your SYSLOG server...

[cwachs]

Well, it ran fine for about 18 hours and at 7:15 this morning, it started constantly going back to the forwarding/learning/discarding cycle. I changed out the port, the cable from the port to the POE surge and changed out the POE surge. I also upgraded to 1.4.3rc7.

[sirhc]

The log is indicating the link is dropping (repeatedly).

Aug 30 07:30:30 STP: set port 7 to discarding
Aug 30 07:30:31 STP: set port 7 to learning
Aug 30 07:30:31 STP: set port 7 to forwarding
Aug 30 07:31:23 STP: set port 7 to discarding
Aug 30 07:31:23 STP: set port 7 to learning
Aug 30 07:31:23 STP: set port 7 to forwarding
Aug 30 07:33:12 STP: set port 7 to discarding
Aug 30 07:33:12 STP: set port 7 to learning
Aug 30 07:33:12 STP: set port 7 to forwarding


Every time the link drops RSTP will cycle through those stages.

Possible causes:
End is not seated properly in jack or there is corrosion or dust in jack - Test by inspecting and re-seating RJ45 connector

Bad End - Test by replacing ends, end may have a poor connection

Cable is bad - Test by replacing cable, cable may have a break in it and when it moved a wire is disconnected

Radio is bad - Test by swapping radio

Ethernet port in switch is bad - Test by moving cable to another port but if that fixes it that may only mean the RJ45 jack you moved was not seated properly so re-test that port with another attempt by moving the cable back and inspect the socket for corrosion and or dirt.

Radio is rebooting, check the up time. However is the wire that is broken in pair 1 or 2 then the radio would not be rebooting.

[interceptwireless]

Recently switched from an 8port TS Pro (no problems with old switch, just wanted to try a Netonix switch), within a week we are seeing a very similar issue with a Rocket M2. We did replace the radio and wire. Set the Rocket from auto to 100 Full Duplex, no change. Nothing else on the tower is having this issue.

Sep 5 11:10:05 Port: link state changed to 'down' on port 8
Sep 5 11:10:05 STP: set port 8 to discarding
Sep 5 11:10:07 Port: link state changed to 'up' (10M-F) on port 8
Sep 5 11:10:07 STP: set port 8 to discarding
Sep 5 11:10:10 STP: set port 8 to learning
Sep 5 11:10:10 STP: set port 8 to forwarding
Sep 5 11:10:31 Port: link state changed to 'down' on port 8
Sep 5 11:10:31 STP: set port 8 to discarding
Sep 5 11:10:33 Port: link state changed to 'up' (100M-F) on port 8
Sep 5 11:10:33 STP: set port 8 to discarding
Sep 5 11:10:35 STP: set port 8 to learning
Sep 5 11:10:35 STP: set port 8 to forwarding

@ 20:44 the Rocket M2 went offline, nothing in the logs. Manually bounced the port and waiting for it to go offline again in the next 24-48 hours.

Let me know if you have any additional suggestions.

[cwachs]

I've got this on two separate switches now. Both were working for months.

[sirhc]

@interceptwireless
@cwachs

I would like to help so here is what I would like you to do so I can help:
1) Upgrade to v1.4.3
2) Post up a screen grab of the Stats Tab and the Device/Status Tab and the Ports Tab

To up load screen grabs of the Tab use the upload attachment under the BLUE SUBMIT BUTTON and then place in line

[intercept]

Attached.

Update -- I had to force the RocketM2 to 100FD and the port to 100FD to stop the errors.

[sirhc]

@intercept


Well holly crap man look at the errors on port 8, your link is dropping which is causing the RSTP issue you see in log.

One of these is your problem:
1) Port in switch is bad - Test by moving to port 7, if problem follows cable then not the port.
Although it could be the end on the cable in the port 8 and re-seating it fixes it.

2) Cable Ends = Replace ends

3) Cable is bad = replace cable

4) Radio is bad = replace radio

You could disable RSTP on port 8 but that just hides the problem.

Could also be ground current on this cable, are all these radios on the same tower and close to each other?
Is the tower grounds bonded to the electrical service grounds?

By the way v1.4.4 was release so upgrade!