Netonix WS-26-400-AC

[konspiracy]

Hey guys,

I was hoping I could pick your brains. I work for a wisp and we have over 40 Netonix Switches deployed, all of them are the WS-26-400-AC or WS-12-400-AC. We use them with the Ubiquiti 24v Airmax gear and some of the 60gig 48v stuff. The issues I have been having are sometimes the they will lock up and stop passing traffic to the AP's, you can still get to the web UI but it will typically lock up after making any change at all. Then we are stuck being able to ping and ssh into the device. Over the years I have tried everything from the reload cold command to restarting the http service (that will get you web access again till you make a change), I have even successfully used a dirty ssh command to get the device to lock up and reboot causing it to fix itself. 90% of the time though we will have to drive out and manually kill power for 30 seconds or so and they start working again. So my question is, can you give me a command that will allow me to fully reboot these switches if they lock up like this? I see in past responses you have said most people don't want to loose POE during a reload but I do want a last ditch effort to reboot these devices before having to drive out.

For instance this morning we rebooted every AP in our network using UISP, we had 4 switches all lock up in the same manner. They were all revision B boards but we had both the 26 port and 12 port models in that revision go out the same way. So can you give me anything else we can try before having to sending somebody on site in the future? Can you give me a command that would cause the switch to do a hard full reboot as a last ditch effort? I love your products and we will continue to use them in the future.

[sirhc]

What version of firmware are you using? If not v1.5.17rc2 upgrade it.

Are the switches accessible from the web?

Are you using the access control list?

Are you polling it with SNMP and if so don't do so more than once a minute?

Are you using our manager and if so don't use more than 1 manager to access the same switch.

[konspiracy]

What version of firmware are you using? If not v1.5.17rc2 upgrade it.

Are the switches accessible from the web?

Are you using the access control list?

Are you polling it with SNMP and if so don't do so more than once a minute?

Are you using our manager and if so don't use more than 1 manager to access the same switch.

1. Yes sir the switches are available from the web but they stop passing traffic from the AP's, we have a vlan set up for management and then one for public.
2. No we are not using the access control list.
3. Yes we do poll all of these with SNMP via a Zabbix server I will look into the polling.
4. No I have looked at the manager and even set everything up but it was shortly after deleted.

I just want to say what seems to cause this specific issue to happen is when we reboot the equipment for maintenence, random storms that cause surges ( we are behind large industrial battery backups), or in this last case rebooting 6-15 Ubiq AP's at the same time via UISP ( we have over 350 AP's so manually rebooting would take forever). One of these cases we had 2 switches in one cabinet and only the one locked up, the other 3 were all 20-30 min drive away from each other. A simple unplug and wait 30 seconds before plugging them back in and everything is back to normal.
When this happens the GUI is accessible and you can log in, making any changes will lock up the web server though and then you are stuck with ssh access only. I can ssh into the switch but no amount of reloads will allow the AP's to pass traffic or get the GUI working, only a physical power down brings them back. This only happens a handful of times a year so it would just be nice to have a last ditch effort way to reboot these switches remotely. My only other option is putting some sort of manageable device in front of the switch that will let me turn power on and off to it. I just don't want to add another point of failure.

Also these switches are on private IP's and behind Mikrotiks. They arent accessible to the outside world.

[sirhc]

What version of firmware are you using?

Are you using SFP modules?

This could be related to an SFP module. Next time this happens leave switch on, pull the SFP modules and then re-insert them one at a time see if traffic resumes.

Read these posts/threads:

viewtopic.php?f=6&t=293&p=11025&hilit=SFP+pull+insert#p11025

viewtopic.php?f=7&t=2849&p=19636&hilit=SFP+pull+insert#p19636

If this is the case replace the SFP module but test the new module by doing several software reboots. Some SFP modules have a tolerance +/- in timing issue even if other SFP modules of same type worked fine.

[konspiracy]

What version of firmware are you using?

Are you using SFP modules?

This could be related to an SFP module. Next time this happens leave switch on, pull the SFP modules and then re-insert them one at a time see if traffic resumes.

Read these posts/threads:

viewtopic.php?f=6&t=293&p=11025&hilit=SFP+pull+insert#p11025

viewtopic.php?f=7&t=2849&p=19636&hilit=SFP+pull+insert#p19636

If this is the case replace the SFP module but test the new module by doing several software reboots. Some SFP modules have a tolerance +/- in timing issue even if other SFP modules of same type worked fine.

While we do have some switches with SFP ports none of the 4 that went down due to Ubiq reboots have one installed. They all were running firmware 1.5.14, I still think our issue is caused due to a power surge or the fact that UISP is able to reboot 300+ AP's at the exact same time. Do you have a back door SSH command that can fully reboot these when they do lock up and I still have command line access? I'd rather have that in my back pocket then trying to chase down why 5-10 times a year these lock up due to a perfect set of events. Otherwise I'll probably just round robin my own custom reboot script so the AP's arent all hitting the switch at the same exact time.

[sirhc]

While we do have some switches with SFP ports none of the 4 that went down due to Ubiq reboots have one installed. They all were running firmware 1.5.14, I still think our issue is caused due to a power surge or the fact that UISP is able to reboot 300+ AP's at the exact same time. Do you have a back door SSH command that can fully reboot these when they do lock up and I still have command line access? I'd rather have that in my back pocket then trying to chase down why 5-10 times a year these lock up due to a perfect set of events. Otherwise I'll probably just round robin my own custom reboot script so the AP's arent all hitting the switch at the same exact time.

We don't have any type of back door.

Are you able to SSH into them?

If so there is a command to reboot the switch via SSH yes, "reload cold" but not the same as a power cycle.

There is help throughout SSH or console "?" [Enter]

I would suggest upgrading your firmware.
I would use v1.5.17rc2 and soon v1.5.17rc3 as there is a security hole patch being worked on.

[konspiracy]

While we do have some switches with SFP ports none of the 4 that went down due to Ubiq reboots have one installed. They all were running firmware 1.5.14, I still think our issue is caused due to a power surge or the fact that UISP is able to reboot 300+ AP's at the exact same time. Do you have a back door SSH command that can fully reboot these when they do lock up and I still have command line access? I'd rather have that in my back pocket then trying to chase down why 5-10 times a year these lock up due to a perfect set of events. Otherwise I'll probably just round robin my own custom reboot script so the AP's arent all hitting the switch at the same exact time.

We don't have any type of back door.

Are you able to SSH into them?

If so there is a command to reboot the switch via SSH yes, "reload cold" but not the same as a power cycle.

There is help throughout SSH or console "?" [Enter]

I would suggest upgrading your firmware.
I would use v1.5.17rc2 and soon v1.5.17rc3 as there is a security hole patch being worked on.

Yes sir I did see that, ya I have tried the reload cold but it's not enough to bring them back when they hang up in this fashion. I noticed that if you use cmdline and use reboot it seems to be removed or doesn't do anything. What I will probably end up doing next time is an SSH script to send the reboot command to the AP's with some sort of time delay so it doesn't hit the switch all at the same time, just a pain with 300+ of them in the field. I will try and push firmware upgrades for security and SNMP but unless they can get through our mikrotiks and on to the private network we are pretty safe. If they can get that far we are already in trouble lol. Thanks for taking the time to respond.