Switches Not Passing Packets

[Stephen]

There has been at least seven different people reporting issue's with packets not being passed that I've seen so far on the forum's.
Based on the reports, it appears that several switches sent out with the preloaded firmware 1.5.5 may have been afflicted.

What would help us now would be if everyone posted their experience's on this thread so that we can consolidate all the various factor's that are presently spread across the forum's.

As stated above the working theory is that switches preloaded with 1.5.5 are the affected unit's. If there are any counter examples, please post them here.

If you are affected by this you should be able to solve it with one of the following:

• downgrading to 1.5.3 then back to 1.5.5 has been proven effective which to my knowledge was first reported by yahel here
• upgrading to the latest rc firmware 1.5.6rc1 should also fix it, as reported by t.beesley@centracom.com here
• upgrade to v1.5.5, even though it already is v1.5.5 this would show if the upgrade process itself clears the issue (possible config file issue)
  EDIT: jadkins@skynetky.com and mhoppes have both confirmed this method works as well
  
• potentially, defaulting the config on a new switch with 1.5.5 preloaded may solve the issue (no confirmation on this yet, please post here if this works or doesn't work for you)
  EDIT: jadkins@skynetky.com and Supernova have confirmed that unfortunately, this does NOT work

If anyone has been sent a switch preloaded with 1.5.5 and DOES NOT have this issue, it would also be helpful to find out who that is.

It will give us a better scope of the situation and potentially what caused it. So if you are reading this and you haven't seen this problem with a switch sent to you that was preloaded with 1.5.5 learning more about your setup or even just when it was sent might help.

What our current theory is on how this happened:

Around the time that me and Eric went to Philly to work directly with Chris to see if we could hammer out some serious problems that we had (the most notorious being the vtss_appl crashes - which was fixed, or at least GREATLY reduced) we were also working on a new software system to help technicians complete serializing switches faster. This also ties into the new RMA system that will hopefully make that entire process smoother for everyone. Many change's to the firmware were necessary during that time and development continues to this day. Potentially something may have happened there that could have caused this. That is what I will be investigating to prevent this from occurring again when 1.5.6 eventually comes out and replaces 1.5.5 as the preloaded firmware.

To that end, any information about your experience seeing this or not seeing this with switches with the preloaded 1.5.5 firmware will be helpful.
Thank you everyone for your reports, I have been watching and running test's in my lab to try and isolate what's going on. For now, please follow steps laid out by others above to prevent it from impacting your own operations as much as possible.

[sakita]

Out last batch of WS-12-250-AC have mid-March to early-April manufacturing dates (v1.5.5 Rev F).

The last one we deployed two weeks ago didn't have any issue out of the box.

This is how we typically deploy:

1. Upgrade firmware to current version (but this time it was already 1.5.5 so didn't have to).
2. Load a base configuration file into the unit (includes all VLANs, and other typical settings).
3. Adjust the config: Change IP address, set the first VLAN to the correct one, delete unneeded VLANs, etc.

Wondering if there is a specific date range and / or if the fact that we load a default config rather than hand config makes a difference?

[rwineteer]

I have a WS-26-400-AC that came pre loaded with 1.5.5, and I will try to explain the issue we experienced.

On this unit I changed the default vlan 1 to 22 (we do this on all units), and changed the default IP to one out of our management subnet.

This Netonix is connected to an existing WS-26-400-AC via rj45 that is running 1.4.9 and also has its default vlan as 22 for management.

The new unit with 1.5.5 was reachable on vlan 22, and has 6 Ubiquiti radios, 2 cameras, and a MikroTik connected to it. All in the same management subnet. The radios and MikroTik are tagging vlan 22, the cameras are untagged. Radios and cameras powered by the unit.

The new unit running 1.5.5 was reachable in our network remotely, but none of the equipment plugged into the the unit running 1.5.5 was reachable from our network. However, if I ssh into the unit with 1.5.5 I could ping and telnet/ssh to any of the devices directly connected. All of the devices plugged into the 1.5.5 unit is equipment that was on site previously and only moved to this new unit, they were in the old unit with 1.4.9 before.

The unit with 1.4.9 and 1.5.5 were both rebooted, and ARP was cleared for all IPs of the equipment that was in the new unit. Vlans were removed from the ports in the new unit, and re added, but we still could not reach those devices in the unit running 1.5.5 externally; only if we ssh into the unit with 1.5.5 first could we reach them.

We downgraded the 1.5.5 unit to 1.5.3, and then all devices were reachable remotely and passing traffic. We upgraded the unit back to 1.5.5 and the devices were still reachable now and passing traffic.

This is the first time we have ran into this issue with a unit that came pre loaded with 1.5.5. We have also never had the issue with any unit we've upgraded to 1.5.5.

[mhoppes]

Got a 12-400-AC with 5.5 on it. I can not talk to anything across the switch with default vlan setup. If I enable isolation in the port I’m trying to talk to, suddenly I can talk to that port.

[Stephen]

Do any of the methods mentioned help recover it?

[mhoppes]

We will find out tomorrow.

[jadkins@skynetky.com]

I just took delivery of (2) WS 26 400 AC. Both had this issue. The first one installed started working on it's own all the sudden so now I don't trust it. The second one would power on. I could login and my network could ping the switch but it would not pass packets across vlans. The solution for me was to reflash the firmware. This instantly fixed my problem. So far they are running. I could login and manage the switch and all. Just could not pass traffic. Very odd issues. We got some WS 8 models few weeks ago only one of them was affected. If I can provide any logs or something to help let me know.

[mhoppes]

You reflashed to the same 1.5.5 version?

[jadkins@skynetky.com]

Sure did. Downloaded via forum. Reflashed. It worked. Maybe corrupt file loaded from assembly? I thought maybe I had put the settings wrong in it. I tried default three times before reflash. I actually unracked it and gave up then thought to reflash.

[jadkins@skynetky.com]

My friend I bought the WS 8 units for said after flashing his defective one only a few ports are working not all.