v1.5.17rcX Bug Reports and Comments

[sirhc]

I think I know what happened here.

We had to shrink manufacturer lookup to fit newer packages.

I think Stephen isn't showing MAC's for unknown manufacturers.

Won't effect operation just does not display them.

I reached out to him on this.

Thanks

[sakita]

The best news is I haven't seen any issues with the switching and critical stuff.

Yeah, just looked and macvendors.txt slimmed down from 302KB to 37KB so definitely helped make some room!

Although if I connect my Nintendo to the industrial network it won't ID it now.

[Stephen]

Loaded 1.5.17rc2 on a WS-8-150-AC Board Rev F in my test rig. This is the switch connected to my laptop and 7 other devices (which includes devices that communicate with each other providing a little traffic).

The MAC Table page in the web UI wasn't showing all of the MAC addresses that were shown when issuing a "show mac table" command in the web UI Device Console. At one point there were no addresses on the MAC Table page but were on the Console. Flushing and refreshing didn't change anything... and then the list on the MAC Table page magically started displaying again but still not matching the full list shown by the Console.

I rolled it back to 1.5.17rc1 and the MAC Table page and Console now show the same list consistently.

Hi sakita, been looking into this after me and sirhc touched base. Just to clarify, are you saying that you are seeing different mac tables on the console and web ui? If so can you post screenshots showing the difference's? Or, if preferred, could you send me this information in a PM?

[sakita]

For example, with 8 devices plugged into the switch the console list would show all 8 but the web UI would have less than 8 devices in it.

Note that the time is not set as I don't have NTP available in my lab setup. I just re-updated the switch to 1.5.17rc2 and grabbed these two screenshots with 7 items plugged in.

Port 7 on the console is interesting as it added "unknown" as part of the mac address.

Note I won't have access to the lab until next Tuesday but will check in on this thread.

[Stephen]

Thanks,

No worries, looks like a parsing failure that's been exposed due to the required macvendor reduction.
When you get a chance, could you PM me the contents of /proc/switch/mactable as well as /tmp/mactable.json on the actively failing switch?
That will help me isolate the failure.

[CrackerRiley]

Got rc2 on a WS Mini with no issue.

Have not observed issue on ~200 switches we have on the network (mostly running 1.5.12 to 1.5.16). Haven't attempted to log into all of them since hearing of this hack but memory usage graphs via SNMP appear normal for all of them.

[RTGLW]

Hi Team,

We've been bench testing a WS-8-150-DC running 1.5.17rc2 for around 6 days now. The only notable observation we've made so far is that the RAM utilization is increasing slowly. After the FW was installed we were at 91MB of available memory and now it's at 82.5MB. This switch is sitting on our bench so there isn't any customer traffic transiting, only management and SNMP.

This is actually something we noticed and started investigating on some of our WS-8-150-DC hosts running 1.5.15rc3, before we paused our investigations due to newer firmwares being released (16 and now 17). Though a similar issue was mentioned by another user at the end of the 1.5.16 bug/issues thread: viewtopic.php?f=17&t=8011&start=10

The TLDR of what we discovered on 1.5.15rc3 was that some hosts which had the discovery tab enabled were increasing their RAM usage over time. Disabling the discovery tab on most of these hosts saw memory increases stop and plateau. Oddly, some hosts kept eating away at their memory even with the Discovery tab disabled. Other hosts which were never configured to have the discovery tab enabled from the start never saw any memory increases over time.

Something we noticed on hosts with low available memory was that the vtss_appl process had the highest memory usage (as per PS AUX), as compared to a freshly rebooted switch. EG: 101M vs 19564K. This is as far as we got though. We understand VSZ isn't the best gauge of RAM utilization but I believe it's the only tool available to measure memory usage per process?

The switch we're bench testing 1.5.17rc2 on has the discovery tab disabled and it's vtss_appl process has no more VSZ usage now than when it was rebooted for the FW upgrade, but RAM usage is increasing slowly, so we're seeing conflicting information. Let me know if there is any additional information I can provide or if there is anything specific to look into on our afflicted hosts that may help out.

[Stephen]

Hi Team,

We've been bench testing a WS-8-150-DC running 1.5.17rc2 for around 6 days now. The only notable observation we've made so far is that the RAM utilization is increasing slowly. After the FW was installed we were at 91MB of available memory and now it's at 82.5MB. This switch is sitting on our bench so there isn't any customer traffic transiting, only management and SNMP.

This is actually something we noticed and started investigating on some of our WS-8-150-DC hosts running 1.5.15rc3, before we paused our investigations due to newer firmwares being released (16 and now 17). Though a similar issue was mentioned by another user at the end of the 1.5.16 bug/issues thread: viewtopic.php?f=17&t=8011&start=10

The TLDR of what we discovered on 1.5.15rc3 was that some hosts which had the discovery tab enabled were increasing their RAM usage over time. Disabling the discovery tab on most of these hosts saw memory increases stop and plateau. Oddly, some hosts kept eating away at their memory even with the Discovery tab disabled. Other hosts which were never configured to have the discovery tab enabled from the start never saw any memory increases over time.

Something we noticed on hosts with low available memory was that the vtss_appl process had the highest memory usage (as per PS AUX), as compared to a freshly rebooted switch. EG: 101M vs 19564K. This is as far as we got though. We understand VSZ isn't the best gauge of RAM utilization but I believe it's the only tool available to measure memory usage per process?

The switch we're bench testing 1.5.17rc2 on has the discovery tab disabled and it's vtss_appl process has no more VSZ usage now than when it was rebooted for the FW upgrade, but RAM usage is increasing slowly, so we're seeing conflicting information. Let me know if there is any additional information I can provide or if there is anything specific to look into on our afflicted hosts that may help out.

Hello RTGLW,

Couple questions to see if we can figure out what's going on.

For the switch's that had discovery disabled and are still growing in memory, if you reboot them - does the memory growth stop?

For the same set of switch's that continued increasing in memory usage after Discovery was disabled, is there anything else different you can tell us about their configuration? Even if it seems small, such as, where SFP ports plugged in, where service's or ports enabled/disabled that differ from the other switch's? etc.

There were memory leaks found in SNMP and in the Watchdog functionality that where patched in 1.5.16. However, since the switch has so many different possible configurations, differing config's could expose others that haven't otherwise been caught, or wouldn't show up normally from an average deployment.

That being said, since you mentioned you saw that vtss_appl had a large memory consumption in a few instance's. For these switch's, did they have any of the following attributes?
- SFP ports plugged in? - if so, do you see any I2C error's in the switch log on these units?
- LAGs or LACP configured? - Is there anything in the logs related to these services?

For the switch's that continued showing memory loss but the offending process was not vtss_appl. Can you tell us which process it was? ps aux is OK for this type of test, on a production unit, this is the best that can be done. But knowing which process it is will definitely help find the issue.

Just in case it's still SNMP that's causing problems. If you have a linux box laying around anywhere. You can try running this in a loop to try and expose the problem more blatantly.

1. Install netsnmp, on Ubuntu, you can use snap to install it: https://snapcraft.io/install/net-snmp/ubuntu
- if on another distribution, I leave the details to you.
2. In a bash terminal (after net-snmp is installed) run the following:

Code: Select all

 
while true; do snmpwalk -v 2c -c public <ip-address-of-switch>; done
 

This will continuously query the entire snmp tree. I have tools that do similar things to try and expose issue's. You can launch this in a few terminals to try to increase the effect.
If SNMP is the cause, the memory loss should increase dramatically after running this script(s). If you notice this, please let us know here and any other details such as the switch configuration so we can try and replicate it. (I suppose this is obvious but I'll say it anyway, please don't do this on switch's in production)

Memory does go up over time during normal operation, but it should eventually level off and also clear itself, as there is caching that occurs in the kernel that is slightly different than an average one that is meant to keep the interface between frames going between the CPU and the switchcore moving smoothly. High traffic loads can trigger this, but as traffic fluctuates up and down, it should clean it up as it goes along. Depending on the load, that might be all it is.

On a similar vein, if enabled, try disabling pause frames to see if that helps.

[Flo]

Stephen,

No worries, looks like a parsing failure that's been exposed due to the required macvendor reduction.

are you going to release a v1.5.17rc3 fixing the mactable parsing issue?

[Stephen]

Hi Flo,

Me and sakita are still looking into it. If you are experiencing the same issue, the details I've asked for from him, /proc/switch/mactable and /tmp/mactable.json would help expedite the process.