Wisp Switch Dead after Firmware Upgrade

[jportman]

I downloaded the 'latest' firmware from the download firmware page and applied it.
The switch never came back up. It's dead. No ping on the default IP.
Does not get an IP address from dhcp anymore.
Reset using reset button, no response on 192.168.1.20
I'll have to dig up a gender changer to hook to the serial port.

[sirhc]

Please answer these questions:

Which version did you upgrade to?

What version did you upgrade from?

We tested upgrading to 1.10rc15 from at least v1.0.6rc1 with no issues at all?

We had problems with 1.10rc12 but 1.1.0rc15 has been installed and tested by many people.

I upgraded all my WISP's switches with no problem and I have spoken to many people who have upgraded to RC15 with no issues?

Now there was a VLAN bug introduced after v1.1.0rc5, so if you were running a version with the VLANs broken and did not realize it and your VLAN config was incorrect when you upgraded and the switch applied the VLANs correct you may have an issue if your VLANs were set wrong?

We plan to make v1.1.0rc15 into a final release tomorrow.

Can you hook a console cable to it?

Feel free to call me on my cell 717-682-8692

I am VERY VERY confident that v1.1.0rc15 is correct and stable as we have been testing it for days.

[jportman]

Was on version 1.0.7rc6 I believe.
I upgraded to 1.1.0rc15 (latest one on the web page).
I have a serial console in there.
I have done this:
cmd
cat config.php | \
grep -v "<<<<<<< HEAD" | \
grep -v "=======" | \
grep -v "Config::save(\$config, false);" | \
grep -v ">>>>>>> remotes/origin/1.0.10" > config.php
reboot
echo Done

To no avail. The switch is in a 'boot loop' It runs for about 60 seconds and reboots. Again and again.

[jportman]

I have also held the reset button at the proper time ( a few times now).
I have held 'CTRL-C for failsafe mode. No effect.
I can get into the command line just fine.
After about 60 seconds, the unit reboots.

[jportman]

Very weird. I have access to the switch now. I logged in and saw the eth0 set to 192.168.1.20.
I have 192.168.1.1 on the router but was not able to ping.
I MANUALLY set the IP address and gateway and was able to log into the web interface.
It has quit rebooting itself as well.

[jportman]

I'm just glad I did not have this switch in production (yet). Would have been really bummed to drive to a remote tower etc.

[jportman]

Spoke too soon. Made some config changes and saved them then rebooted the switch.
Now it's back in a boot loop.

[sirhc]

Was on version 1.0.rc7 I believe.
I upgraded to 1.1.0rc15 (latest one on the web page).
I have a serial console in there.
I have done this:
cmd
cat config.php | \
grep -v "<<<<<<< HEAD" | \
grep -v "=======" | \
grep -v "Config::save(\$config, false);" | \
grep -v ">>>>>>> remotes/origin/1.0.10" > config.php
reboot
echo Done

To no avail. The switch is in a 'boot loop' It runs for about 60 seconds and reboots. Again and again.

This fix was SPECIFIC to a bug in 1.1.0rc12 that caused the switch to not boot up fully, it was not to be applied to ANY other version, not was it a fix for anything other then a BUG in v1.1.0rc12

[sirhc]

OK after working with Joe on the phone we determined that his problem is related to setting our switch to get a DHCP address.

If you use a static address there is no issue with v1.1.0rc15.

Now we DID test DHCP but to be honest we did not test this in-depth which we will tomorrow and Monday and release a fix if needed (probably is a bug).

The other thing is we only tested DHCP against a Cisco and or Microsoft DHCP server we did not test it against a Microtic DHCP server which is what Joe is using.

But to be honest the extent that we tested DHCP was probably "OK it got an address we are good" as we always statically assign all out equipment at my WISP.

So his issue was when he set it to DHCP, the switch would get into a reboot loop. The switch was getting an IP address but not a default gateway.
He statically assigned his switch and all was fine.

I promised him we would investigate this issue tomorrow and Monday and have a fix for him ASAP.

So if you as not using DHCP you probably will not have a problem

How he got it out of the reboot loop was to hook a console cable to the switch and when it boot you will see a line come up that says press the default button which he used a paper clip to do so then you have to wait about 2 minutes while the switch re-formats the JFF2 partition and then you are back to defaults.

A normal UI or CLI default does not reformat the partition it just erases it so it is faster.

He probably could have just pressed and held the default button for about 10 seconds until all the Ethernet lights looped green and then red before letting it up on the button but we did not bother checking.

There are a few strange configuration variables here which is that Joe does all his VLANs in the MT and sets all his switch ports to T,not sure how this effects things but we will test on Monday.

[sirhc]

It has been confirmed that there is a bug in v1.1.0rc15 where if you set the switch to get a DHCP address it neglects to get a gateway address.

We are not sure why yet this caused jportman's (Joe) switch to go into a reboot loop but so far all that has been done is Rory and I confirmed that the DHCP setting does indeed not get default gateway. We have also asked Eric to look at it and he said he will get a new release out today to fix this.

So Joe if you read this post please let me know if you did any other config change other than set the switch to DHCP and reboot.

In fact if you have the time give me a call on my cell.

Rory and I suspect that the hardware Watch Dog caused the reboot but we will leave it up to Eric to determine why.

So for now as long as you do not use a DHCP address but rather a static address this bug will NOT affect you in anyway.