Ports dropping and switch rebooting

[garry@hsbnv.net]

Can anyone tell me why the switch would restart? I have no ping watchdog on. Also shows ports bouncing No idea what that is either.



Dec 2 09:29:14 kernel: vtss_port: port event on port_no: 13 (link 0 link_down 1) Dec 2 09:29:16 kernel: vtss_port: port event on port_no: 13 (link 1 link_down 0) Dec 2 09:29:47 kernel: vtss_port: port event on port_no: 14 (link 1 link_down 0) Dec 2 09:30:11 kernel: vtss_port: port event on port_no: 14 (link 0 link_down 1) Dec 2 09:30:12 kernel: vtss_port: port event on port_no: 14 (link 1 link_down 0) Dec 2 09:30:35 kernel: vtss_port: port event on port_no: 24 (link 0 link_down 1) Dec 2 09:30:39 kernel: vtss_port: port event on port_no: 24 (link 1 link_down 0) Dec 2 09:32:25 kernel: vtss_port: port event on port_no: 5 (link 1 link_down 0) Dec 2 09:32:48 kernel: vtss_port: port event on port_no: 5 (link 0 link_down 1) Dec 2 09:32:50 kernel: vtss_port: port event on port_no: 5 (link 1 link_down 0) Dec 2 09:35:10 kernel: vtss_port: port event on port_no: 6 (link 1 link_down 0) Dec 2 09:35:33 kernel: vtss_port: port event on port_no: 6 (link 0 link_down 1) Dec 2 09:35:35 kernel: vtss_port: port event on port_no: 6 (link 1 link_down 0) Dec 2 09:36:19 kernel: vtss_port: port event on port_no: 7 (link 1 link_down 0) Dec 2 09:36:40 kernel: vtss_port: port event on port_no: 7 (link 0 link_down 1) Dec 2 09:36:42 kernel: vtss_port: port event on port_no: 7 (link 1 link_down 0) Dec 2 09:37:28 kernel: vtss_port: port event on port_no: 9 (link 1 link_down 0) Dec 2 09:37:52 kernel: vtss_port: port event on port_no: 9 (link 0 link_down 1) Dec 2 09:37:53 kernel: vtss_port: port event on port_no: 9 (link 1 link_down 0) Dec 2 09:39:20 kernel: vtss_port: port event on port_no: 9 (link 0 link_down 1) Dec 2 09:39:40 kernel: vtss_port: port event on port_no: 8 (link 1 link_down 0) Dec 2 09:39:44 kernel: vtss_port: port event on port_no: 9 (link 1 link_down 0) Dec 2 09:40:03 kernel: vtss_port: port event on port_no: 8 (link 0 link_down 1) Dec 2 09:40:05 kernel: vtss_port: port event on port_no: 8 (link 1 link_down 0) Dec 2 09:40:07 kernel: vtss_port: port event on port_no: 9 (link 0 link_down 1) Dec 2 09:40:09 kernel: vtss_port: port event on port_no: 9 (link 1 link_down 0) Dec 2 09:42:10 kernel: vtss_port: port event on port_no: 22 (link 1 link_down 0) Dec 2 09:45:13 kernel: vtss_port: port event on port_no: 18 (link 1 link_down 0) Dec 2 09:45:37 kernel: vtss_port: port event on port_no: 18 (link 0 link_down 1) Dec 2 09:45:38 kernel: vtss_port: port event on port_no: 18 (link 1 link_down 0) Dec 2 09:47:56 kernel: vtss_port: port event on port_no: 21 (link 1 link_down 0) Dec 2 09:48:20 kernel: vtss_port: port event on port_no: 21 (link 0 link_down 1) Dec 2 09:48:21 kernel: vtss_port: port event on port_no: 21 (link 1 link_down 0) Dec 2 09:51:06 kernel: vtss_port: port event on port_no: 10 (link 1 link_down 0) Dec 2 09:51:29 kernel: vtss_port: port event on port_no: 10 (link 0 link_down 1) Dec 2 09:51:31 kernel: vtss_port: port event on port_no: 10 (link 1 link_down 0) Dec 2 09:52:55 kernel: vtss_port: port event on port_no: 11 (link 1 link_down 0) Dec 2 09:53:00 kernel: vtss_port: port event on port_no: 11 (link 0 link_down 1) Dec 2 09:53:02 kernel: vtss_port: port event on port_no: 11 (link 1 link_down 0) Dec 2 09:53:18 kernel: vtss_port: port event on port_no: 11 (link 0 link_down 1) Dec 2 09:53:20 kernel: vtss_port: port event on port_no: 11 (link 1 link_down 0) Dec 2 09:56:42 kernel: vtss_port: port event on port_no: 24 (link 0 link_down 1) Dec 2 09:56:46 kernel: vtss_port: port event on port_no: 24 (link 1 link_down 0) Dec 2 09:58:53 kernel: vtss_port: port event on port_no: 3 (link 1 link_down 0) Dec 2 09:59:15 kernel: vtss_port: port event on port_no: 3 (link 0 link_down 1) Dec 2 09:59:17 kernel: vtss_port: port event on port_no: 3 (link 1 link_down 0) Dec 2 10:00:14 kernel: vtss_port: port event on port_no: 15 (link 1 link_down 0) Dec 2 10:00:37 kernel: vtss_port: port event on port_no: 15 (link 0 link_down 1) Dec 2 10:00:38 kernel: vtss_port: port event on port_no: 15 (link 1 link_down 0) Dec 2 10:01:29 kernel: vtss_port: port event on port_no: 16 (link 1 link_down 0) Dec 2 10:01:55 kernel: vtss_port: port event on port_no: 16 (link 0 link_down 1) Dec 2 10:01:57 kernel: vtss_port: port event on port_no: 16 (link 1 link_down 0) Dec 2 10:02:30 kernel: vtss_port: port event on port_no: 17 (link 1 link_down 0) Dec 2 10:02:34 kernel: vtss_port: port event on port_no: 17 (link 0 link_down 1) Dec 2 10:02:36 kernel: vtss_port: port event on port_no: 17 (link 1 link_down 0) Dec 2 10:02:52 kernel: vtss_port: port event on port_no: 17 (link 0 link_down 1) Dec 2 10:02:55 kernel: vtss_port: port event on port_no: 17 (link 1 link_down 0) Dec 2 10:04:29 monitor: restarting switch Dec 2 10:11:25 kernel: vtss_port: port event on port_no: 23 (link 0 link_down 1) Dec 2 10:11:27 kernel: vtss_port: port event on port_no: 24 (link 0 link_down 1) Dec 2 10:15:39 kernel: vtss_port: port event on port_no: 23 (link 1 link_down 0) Dec 2 10:16:09 kernel: vtss_port: port event on port_no: 21 (link 0 link_down 1) Dec 2 10:16:15 kernel: vtss_port: port event on port_no: 21 (link 1 link_down 0) Dec 2 10:16:39 kernel: vtss_port: port event on port_no: 21 (link 0 link_down 1) Dec 2 10:16:40 kernel: vtss_port: port event on port_no: 21 (link 1 link_down 0) Dec 2 10:19:21 kernel: vtss_port: port event on port_no: 2 (link 0 link_down 1) Dec 2 10:19:23 kernel: vtss_port: port event on port_no: 2 (link 1 link_down 0) Dec 2 10:23:03 kernel: vtss_port: port event on port_no: 2 (link 0 link_down 1) Dec 2 10:23:38 kernel: vtss_port: port event on port_no: 2 (link 1 link_down 0) Dec 2 10:24:01 kernel: vtss_port: port event on port_no: 2 (link 0 link_down 1) Dec 2 10:24:03 kernel: vtss_port: port event on port_no: 2 (link 1 link_down 0) Dec 2 10:39:59 kernel: vtss_port: port event on port_no: 23 (link 0 link_down 1) Dec 2 10:49:42 kernel: vtss_port: port event on port_no: 14 (link 0 link_down 1) Dec 2 10:49:52 kernel: vtss_port: port event on port_no: 14 (link 1 link_down 0) Dec 2 10:50:16 kernel: vtss_port: port event on port_no: 14 (link 0 link_down 1) Dec 2 10:50:18 kernel: vtss_port: port event on port_no: 14 (link 1 link_down 0) Dec 2 23:59:26 kernel: vtss_port: port event on port_no: 2 (link 0 link_down 1) Dec 2 23:59:28 kernel: vtss_port: port event on port_no: 2 (link 1 link_down 0) Dec 4 02:41:35 kernel: vtss_port: port event on port_no: 5 (link 0 link_down 1) Dec 4 02:41:38 kernel: vtss_port: port event on port_no: 5 (link 1 link_down 0) Dec 4 02:42:01 kernel: vtss_port: port event on port_no: 5 (link 0 link_down 1) Dec 4 02:42:03 kernel: vtss_port: port event on port_no: 5 (link 1 link_down 0) Dec 5 00:50:56 passwd: Password for admin changed by admin Dec 5 01:20:03 kernel: vtss_port: port event on port_no: 1 (link 0 link_down 1) Dec 5 01:26:04 kernel: vtss_port: port event on port_no: 2 (link 0 link_down 1) Dec 5 01:26:06 kernel: vtss_port: port event on port_no: 1 (link 1 link_down 0) Dec 5 01:32:33 kernel: vtss_port: port event on port_no: 2 (link 1 link_down 0) Dec 6 04:21:11 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 04:21:22 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 04:21:49 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 04:21:50 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 04:21:53 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 04:21:54 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 04:51:44 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 04:51:51 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 04:58:35 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 04:58:37 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 04:58:39 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 04:58:41 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 05:01:41 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 05:01:45 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 05:02:11 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 05:02:13 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 05:02:15 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 05:02:17 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 21:28:46 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 21:28:47 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 21:28:50 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 21:28:51 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 21:30:58 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 21:30:59 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 21:31:02 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 6 21:31:03 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 6 21:32:47 monitor: restarting switch Dec 7 14:52:22 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 14:52:25 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 14:52:51 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 14:52:53 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 14:52:55 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 14:52:57 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:25:52 monitor: restarting switch Dec 7 22:28:19 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:28:20 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:28:23 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:28:24 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:30:46 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:30:47 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:30:50 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:30:51 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:32:51 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:32:53 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:32:56 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:32:57 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:37:24 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:37:25 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:37:28 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:37:30 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:45:57 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:45:59 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0) Dec 7 22:46:01 kernel: vtss_port: port event on port_no: 12 (link 0 link_down 1) Dec 7 22:46:03 kernel: vtss_port: port event on port_no: 12 (link 1 link_down 0)

[sirhc]

Hi Garry,
I am not sure what you're saying is happening or what your asking?

Can anyone tell me why the switch would restart?

Is your switch rebooting?

I have no ping watchdog on.

OK

Also shows ports bouncing No idea what that is either.

Your screen grab of the log is all smashed together, hard to follow

Maybe provide us more information:
What firmware version
What is happening
How you are using it
Maybe some configuration screen shots

I would l love to help but your post is not very informative of the issue you want help with.

[garry@hsbnv.net]

Firmware 1.0.6rc3
1.When I remotely reboot I can not get back into it. I have to do a hard reboot with the power.
2. The log shows ports going up and down. Not sure why?
Example : link http://my.jetscreenshot.com/demo/20141208-kmeb-164kb

Dec 2 09:29:14 kernel: vtss_port: port event on port_no: 13 (link 0 link_down 1)
Dec 2 09:29:16 kernel: vtss_port: port event on port_no: 13 (link 1 link_down 0)
Dec 2 09:29:47 kernel: vtss_port: port event on port_no: 14 (link 1 link_down 0)
Dec 2 09:30:11 kernel: vtss_port: port event on port_no: 14 (link 0 link_down 1)
Dec 2 09:30:12 kernel: vtss_port: port event on port_no: 14 (link 1 link_down 0)

3. the log shows "restarting switch" Not understanding why?
Dec 7 22:25:52 monitor: restarting switch

The switch is at a Tower. port 1 is AF24 in and feeding APs and BH out.
http://my.jetscreenshot.com/demo/20141208-our3-88kb

[sirhc]

Firmware 1.0.6rc3

Please update your firmware to v1.0.8.

We ask people to upgrade to the newest version and if the problem is still there then please post screen grabs of your config tab and describe your problem and we jump right on it.

I know we are releasing versions often right now as we are working through the bugs any new product will expect.

If everything is working then upgrade as you like but if you encounter a problem please upgrade to the latest version then post your config tabs of any tab that is not default settings and describe the issue.

Well when the switch first boots we turn all the ports off until the firmware loads then it turns them back on after the switch core configuration is applied.

This is done to prevent loops and such as when the switch first boots it is just that, a plane switch without configuration to deal with RSTP, LAGs, and such.

After the switch is configured it turns the ports back up.

Well there is a hardware watchdog that can cause the switch CPU to restart but not restart the switch core itself or the POE ports if some weird things are happening such as loops and arp storms but that has been eliminated in v1.0.7 and newer.

However I noticed in your "partial" screen grab (see below) there are a LOT of Ethernet errors????
Are these CRC's or what are they from?
That many error would concern me.
CLICK ON IMAGE TO SEE FULL SIZE

garry.png (372.59 KiB) Viewed 7104 times

I notice that you have the first 3 ports configured as 48VH, but you only speak of 1 airFIBER, are all three of these airFIBERs becuase their power consumptions are vastly different. Notice that the 3 airFIBER links on my tower are all about the same current (see below).

Here is a screen grab of one of my towers, notice the errors:
CLICK ON IMAGE TO SEE FULL SIZE

QR.png (111.61 KiB) Viewed 7104 times

But please provide screen grabs of your config tabs like these:
CLICK ON IMAGE TO SEE FULL SIZE

qr ports.png (129.24 KiB) Viewed 7104 times

CLICK ON IMAGE TO SEE FULL SIZE

qr VLAN.png (100.26 KiB) Viewed 7104 times

CLICK ON IMAGE TO SEE FULL SIZE

qr LAG.png (121.58 KiB) Viewed 7104 times

[sirhc]

Also with that many port errors I am curious to see your ports configuration tab and to know if your hard coding port speed/duplex because if you are and your cable is bad then links would be going up and down.

And on a reboot if the port failed to negotiate then you would have to power cycle the switch to force it to re-negotiate or log into the switch with your laptop and see what the port status is, MY BET IS THE PORT YOU USE TO GET TO THE SWITCH IS DOWN BECAUSE IT FAILED TO NEGOTIATE AT THE HARD CODED SPEED/DUPLEX!

I strongly suggest against hard coding port speed/duplex on the switch or radios because that means it can not auto negotiate down which means the link drops if error occur.

But you definitely have some issues with the cabling to have so many errors which could cause links to go up and down especially if your hard coding speed duplex on bad cables.

[Rory]

I can add a little clarity to one of the questions posed here:

The log entry "Dec 7 22:25:52 monitor: restarting switch" is referring to a process (named switch) that is running in the OS of the switch, not the switch itself. I realize in explaining that entry it could use some clarity. As luck would have it, I am currently working with Eric on the logging system, so I will recommend that is changed in the next update.

[sirhc]

I can add a little clarity to one of the questions posed here:

The log entry "Dec 7 22:25:52 monitor: restarting switch" is referring to a process (named switch) that is running in the OS of the switch, not the switch itself. I realize in explaining that entry it could use some clarity. As luck would have it, I am currently working with Eric on the logging system, so I will recommend that is changed in the next update.

And when he says changed he is referring to the verbiage used in the logging which is what we are working on now, we had several discussions today on making log entries more human readable in clearer terms.

[adairw]

Shouldn't this really be moved out of this section and in to the hardware/software section?

[mhoppes]

Isn't that where it is?

[lligetfa]

It is now after being moved.