WS-8-250-DC rebooting

[jww]

We're having a problem that we're having a lot of trouble troubleshooting. I brought this up with Chris today, but no answers yet.

So we have a solar powered relay site that takes about an hour to get to. In the last few days the site has been going down intermittently. And it may have been longer than that, but maybe not long enough outages for AirControl 1 to send us an alarm.

Yesterday afternoon the old switch (running 1.3.2) went down and didn't come back up until about 1:30 am this morning. We could log into the backhaul and the backhaul showed link at 100 Mbps Full duplex, but we couldn't log into the switch or anything past it. The backhaul is a NanoBridge and we have three Rockets hooked to it as APs.

Some ants had gotten into the enclosure and we thought that was the problem with the switch. So we swapped in a new switch, (running 1.4.0rc25)
stayed around about an hour to watch it and saw no problems.

This afternoon the new switch started rebooting. And the backhaul radio would lose power when the switch rebooted.

I updated the firmware on the old switch and let it run on the bench for several hours with no problem.

There are no AirFibers connected to this switch, but we do have an AF connected to a different WS-8-250-DC at a different site.

I copied the log from the new switch below and maybe I can figure out how to insert a screen shot if that will help.



Help! :)








Dec 31 19:00:04 sysinit: killall: udhcpc: no process killed
Dec 31 19:00:05 netonix: 1.4.0rc25 on WS-8-250-DC
Dec 31 19:00:07 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
Dec 31 19:00:07 kernel: switch: 'Luton26' board detected
Dec 31 19:00:09 kernel: vtss_port: Loaded port module on board Luton26, type 5
Dec 31 19:00:09 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
Dec 31 19:00:10 kernel: i2c /dev entries driver
Dec 31 19:00:10 system: Setting MAC address from flash configuration: EC:13:B3:51:xx:xx
Dec 31 19:00:10 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19
Dec 31 19:00:11 sysinit: Loading defaults
Dec 31 19:00:11 sysinit: Adding custom chains
Dec 31 19:00:11 system: starting ntpclient
Dec 31 19:00:12 sysinit: Loading zones
Dec 31 19:00:12 sysinit: Loading forwarding
Dec 31 19:00:12 sysinit: Loading redirects
Dec 31 19:00:12 sysinit: Loading rules
Dec 31 19:00:12 sysinit: Loading includes
Dec 31 19:00:12 admin: adding lan (eth0) to firewall zone lan
Dec 31 19:00:17 Port: link state changed to 'up' (100M-F) on port 3
Dec 31 19:00:17 Port: link state changed to 'up' (10M-F) on port 1
Dec 31 19:00:18 sysinit: killall: telnetd: no process killed
Dec 31 19:00:18 Port: link state changed to 'up' (10M-F) on port 5
Dec 31 19:00:18 Port: link state changed to 'up' (10M-F) on port 7
Dec 31 19:00:19 sysinit: 1969-12-31 19:00:18: (log.c.97) server started
Dec 31 19:00:20 dropbear[741]: Running in background
Dec 31 19:00:21 kernel: eth0: no IPv6 routers present
Dec 31 19:00:22 Port: link state changed to 'down' on port 5
Dec 31 19:00:22 Port: link state changed to 'down' on port 1
Dec 31 19:00:23 Port: link state changed to 'down' on port 7
Dec 31 19:00:23 switch[769]: Detected cold boot
Dec 31 19:00:24 Port: link state changed to 'down' on port 3
Dec 31 19:00:25 switch[769]: PoE enabled on port 1, PoE Smart is starting cable check
Dec 31 19:00:25 switch[769]: PoE enabled on port 3, PoE Smart is starting cable check
Dec 31 19:00:25 switch[769]: PoE enabled on port 5, PoE Smart is starting cable check
Dec 31 19:00:25 switch[769]: PoE enabled on port 7, PoE Smart is starting cable check
Dec 31 19:00:25 system: starting ntpclient
Dec 31 19:00:30 switch[769]: Port 1 cable check results: Ok, Ok, Short, Short
Dec 31 19:00:30 switch[769]: Port 1 passed PoE Smart cable check, turning on power
Dec 31 19:00:30 switch[769]: Port 3 cable check results: Open, Open, Short, Short
Dec 31 19:00:30 switch[769]: Port 3 passed PoE Smart cable check, turning on power
Dec 31 19:00:30 switch[769]: Port 5 cable check results: Ok, Ok, Abnormal termination, Abnormal termination
Dec 31 19:00:30 switch[769]: Port 5 passed PoE Smart cable check, turning on power
Dec 31 19:00:30 switch[769]: Port 7 cable check results: Ok, Ok, Short, Short
Dec 31 19:00:30 switch[769]: Port 7 passed PoE Smart cable check, turning on power
Dec 31 19:00:32 Port: link state changed to 'up' (10M-F) on port 1
Dec 31 19:00:32 Port: link state changed to 'up' (10M-F) on port 5
Dec 31 19:00:33 Port: link state changed to 'up' (10M-F) on port 7
Dec 31 19:00:36 system: starting ntpclient
Dec 31 19:00:38 Port: link state changed to 'up' (100M-F) on port 3
Dec 31 19:00:47 system: starting ntpclient
Dec 31 19:00:55 Port: link state changed to 'down' on port 3
Dec 31 19:00:55 Port: link state changed to 'down' on port 1
Dec 31 19:00:56 Port: link state changed to 'down' on port 5
Dec 31 19:00:56 Port: link state changed to 'down' on port 7
Dec 31 19:00:57 Port: link state changed to 'up' (100M-F) on port 3
Dec 31 19:00:57 Port: link state changed to 'up' (100M-F) on port 1
Dec 31 19:00:58 Port: link state changed to 'up' (100M-F) on port 5
Dec 31 19:00:58 Port: link state changed to 'up' (100M-F) on port 7
Dec 31 19:00:59 system: starting ntpclient
Dec 31 19:01:10 system: starting ntpclient
Dec 31 19:01:21 system: starting ntpclient

[jww]

Some more information - this switch is connected directly to a large battery bank through a fuse. The battery bank stays around 24.2 - 28 volts.

Since I've tried to switches it's looking like the problem might not be the switches, but what could cause the switches to reboot?

[sirhc]

As I said I would lab the one you swapped out, if it performs as designed in the lab then something must be up at the site. If it fails in the lab then there is something wrong with it but I doubt this as the spare is doing the same thing.

Beyond that you can setup a syslog server and SNMP to monitor the switch because when it reboots we lose all the info we need.

I would monitor input voltages especially.

We have no idea what is happening.

Your voltage could get too low or spike to high when charging is occurring.
If voltage drops to low or high it may reboot.

How long does it go off line when this happens?

Post up your Tabs in this post so we can all see your config tabs to brain storm.
Primarily your Status, Power, Device/Configuration, and Device/Status as we are not interested in your network that is not the issue.

You can attach images to your post by clicking on the upload attachment link below the BLUE SUBMIT button

[jww]

Just updated this switch (the replacement switch) to 1.4.0 this morning. The old switch didn't have any fan warnings logged so I'm guessing this doesn't have anything to do with the old switch rebooting. I know that there were some fan logging issues with some previous firmware.

From looking at this log it looks as if I need to RMA this switch, correct?



Jan 1 00:00:06 sysinit: killall: udhcpc: no process killed
Jan 1 00:00:07 netonix: 1.4.0 on WS-8-250-DC
Jan 1 00:00:08 kernel: vtss_core: module license '(c) Vitesse Semiconductor Inc.' taints kernel.
Jan 1 00:00:08 kernel: switch: 'Luton26' board detected
Jan 1 00:00:10 kernel: vtss_port: Loaded port module on board Luton26, type 5
Jan 1 00:00:11 kernel: nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
Jan 1 00:00:11 system: Setting MAC address from flash configuration: EC:13:B3:xx:xx:xx
Jan 1 00:00:11 kernel: i2c /dev entries driver
Jan 1 00:00:11 kernel: i2c_vcoreiii i2c_vcoreiii: i2c bus driver on IRQ 19
Jan 1 00:00:12 sysinit: Loading defaults
Jan 1 00:00:13 sysinit: Loading synflood protection
Jan 1 00:00:13 sysinit: Adding custom chains
Jan 1 00:00:13 sysinit: Loading zones
Jan 1 00:00:13 sysinit: Loading forwarding
Jan 1 00:00:14 sysinit: Loading redirects
Jan 1 00:00:14 sysinit: Loading rules
Jan 1 00:00:14 sysinit: sh: bad number
Jan 1 00:00:14 sysinit: Loading includes
Jan 1 00:00:14 admin: adding lan (eth0) to firewall zone lan
Jan 1 00:00:22 kernel: eth0: no IPv6 routers present
Dec 31 19:00:24 syslogd exiting
Dec 31 19:00:25 syslogd started: BusyBox v1.19.4
Dec 31 19:00:26 admin: removing lan (eth0) from firewall zone lan
Dec 31 19:00:29 admin: adding lan (eth0) to firewall zone lan
Dec 31 19:00:33 admin: adding lan (eth0) to firewall zone lan
Dec 31 19:00:34 sysinit: killall: telnetd: no process killed
Dec 31 19:00:34 sysinit: killall: ntpclient: no process killed
Dec 31 19:00:35 sysinit: killall: ntpclient: no process killed
Dec 31 19:00:35 system: starting ntpclient
May 21 06:32:49 sysinit: 2016-05-21 06:32:48: (log.c.97) server started
May 21 06:32:50 dropbear[1331]: Running in background
May 21 06:32:54 switch[1356]: Detected warm boot
May 21 06:33:04 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:33:48 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:34:32 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:35:16 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:36:01 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:36:45 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:37:29 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:38:13 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:38:58 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:39:42 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:40:26 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:41:11 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:41:55 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:42:39 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:43:23 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:44:08 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:44:52 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:45:36 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:46:20 switch[1348]: Warning: expected 1 fans but found 0
May 21 06:47:05 switch[1348]: Warning: expected 1 fans but found 0

[Dave]

is this log a collection of different logs as the dates are all over the place...Jan1 , dec 31, may21

[jww]

After a reboot until it makes a connection with an NTP server it will show default dates.

[wvvamike]

We are also seeing this issue with a ws-12-250-dc. Running 1.4.1

We installed the unit a few days ago, its rebooted every night. Site has no FM, running two mimosa BH's, one Rocket BH and 2 AP's. Running on 24V and holding a steady 26.7V.

Have around 200Mbps total throughput on it.

Any ideas?

[Dave]

We are also seeing this issue with a ws-12-250-dc. Running 1.4.1

We installed the unit a few days ago, its rebooted every night. Site has no FM, running two mimosa BH's, one Rocket BH and 2 AP's. Running on 24V and holding a steady 26.7V.

Have around 200Mbps total throughput on it.

Any ideas?

was this also the same unit for which you made the other post saying the chassis was in direct sunlight & the chassis got so hot the metal warped & you needed to loosen fan screws a bit & fan then worked ok?

[wvvamike]

The chassis was not in direct sunlight. The outdoor enclosure was. It didn't warp the unit. The fan screws were so tight that once the metal of the unit got warm it put the fan into a bind. (metal expands at a different rate than plastic). Loosening the screws resolved the issue. Without the fan, the cpu temp increased to 120C, board temp was less.

Anyway, the unit was replaced with another. It has regularly rebooted every night since.

[sirhc]

Your going to have to dig a little and find out why.

Does it reboot at the same time every night?

If so login to the unit prior to the known time of reboot and watch the stats or be on site.

Setup a syslog server so you can review the logs prior to the reboot.

Maybe tell us a little about the site:
What is powering the switch?
How many volts?
Post up screen grabs of the Tabs.