HELP- Netonix Virus?

[Flo]

Eric,

is this a re-infection of a previously already infected Netonix switch or a new infection of a Netonix device running latest v1.5.17rc2?

Hacked again this morning right at midnight running 1.5.17rc2. Anyone have have a fix for this? same log
restarting lighttpd
Jan 2 09:31:14 monitor: restarting vtss_appl
...
Jan 2 09:35:01 monitor: restarting shellinaboxd


All time/date and power settings are also now wrong

[sirhc]

Eric,

is this a re-infection of a previously already infected Netonix switch or a new infection of a Netonix device running latest v1.5.17rc2?

Hacked again this morning right at midnight running 1.5.17rc2. Anyone have have a fix for this? same log
restarting lighttpd
Jan 2 09:31:14 monitor: restarting vtss_appl
...
Jan 2 09:35:01 monitor: restarting shellinaboxd


All time/date and power settings are also now wrong

This is NOT a re-infection.
v1.5.17rc2 does patch the hole we had.

We STRONGLY feel this is something different.
We have asked several questions I do not think have been answered.

QUESTIONS:
1) Is this unit on a valid IP? <- THIS IS NOT WISE

2) Are you using the Access Control List <- THIS WOULD BE REQUIRED IF SITTING ON VALID IP

3) Are you polling this unit with SNMP more frequently than once per minute? <- THIS IS BAD AS CAN OVERWHELM THE SMALL EMBEDDED CPU DISCUSSED IN OTHER MUCH OLDER THREADS

3) Please send config to Stephen

[ted.walsh]

At the risk of exposing myself to yet another proper telling off Id like to share the following.

I got hit with the original FBI page on a single device. at the time it was on V .12 and unknown to me the management interface was exposed via a tenuous 443 port forward on a legacy router. it took me a while to locate the route for the exploit, in the mean time,
I took the upgrade path to .16 and then last week swapped the hardware to a new device with RC2 installed on the same internal net IP. I ignored the good advice to setup an access control list as I felt this would make management problematic in certain circumstances (yes, I was wrong)

3 days ago the new device became 'problematic' with the previous 'issues' fingerprints, i.e. ping drops, poor client performance at high loads, high CPU reported in the Status screen, sluggish GUI, internal processes restarting in the log, warm boot being reported. I thought it may be just a process failing with RC2 so tried to reboot via the GUI but the switch would not restart despite several attempts. I took the plunge and reinstalled the RC2 upgrade as I knew this would initiate a restart and although it took over an hour to complete it did get there in the end, the upgrade restart took place and all became good.

This morning, while checking the specific switch I see high CPU, log entries etc again, get cheesed off and implement access control from the management PC only. the log entries stop at that point but the high CPU remains. running a 'top' via putty I see one process that restarts every 3 mins or so mopping up 75% of the CPU. I then go on an evangelist trail to locate the source of the external route and remove the reckless entry closing the port forward hole to the switch GUI via 443.

[The switch is bumping along ATM but after businesses close this evening Im going to re-run the RC2 upgrade again to flush the switch. This time I'm hoping it will stay that way.] EDIT: I re-ran the RC2 upgrade, it completed normally & in a reasonable timeframe, the switch seems to be stable

The observations I would like to put forward are
1) If exposed, the 'exploit' or a development of it appears to be still operational despite RC2, although it seems to work slightly differently (mebbe RC2 blocks some its mal-action)
2) you should not expose the GUI interface to the Internet in any way at any time. ( yes, I know...)
3) you should implement IP address access control on the switch as an immediate safeguard (the switch will even suggest the IP of the current station making it easy)

let the beatings begin...

[sirhc]

Ted,

Thank you for your feed back.

We are assuming you are correct and whereas we closed several holes in v 1.5.17rc2 that does not seem we got the one this person is using.

We upgraded lighttpd and openssl.

We are now in the process, based on your post, of upgrading and or removing other potentially vulnerable add in modules we are using. I am not going to tip my hand and say what at this time but I am sure you understand. I am sure eyes are upon us.

Hopefully soon we will release v1.5.17rc3 with those changes. We have moved all resources from the NEW WS3 coding back to this issue which sadly will delay the new WS3 models with 2.5G copper. The "NEW" WS3 code (not yet released) is a much newer base code provided from the SDK so should not be affected.

So as I gather from your post your switch was infected from a forgotten port mapping on port 443 to the switch?
Gets kind of fuzzy here but after removing the port forward you were safe?

But for sure after implementing the Access Control List you are safe?
This of course is assuming the leak into your network is not on the machine granted access from the Access Control List!

[Flo]

@ted.walsh can you setup a logging reverse proxy to track down the cause of this issue, like: https://hub.docker.com/r/jonaslejon/polarproxy ?

Having a .pcap file containing the cause of this issue should speed up resolving this security incident a lot I guess.

[ted.walsh]

So as I gather from your post your switch was infected from a forgotten port mapping on port 443 to the switch?
Gets kind of fuzzy here but after removing the port forward you were safe?

But for sure after implementing the Access Control List you are safe?
This of course is assuming the leak into your network is not on the machine granted access from the Access Control List!

Thats correct, since setting up the ACL and removing the open port forward Ive not had any further extra log activity. Reinstalling RC2 removed any remnants of the issue and the log file is now clean with no unusual activity, CPU usage and temps are all nominal.

Tally Ho!

[sirhc]

We hope to release v1.5.17rc3 soon which should close all holes.

[Flo]

We hope to release v1.5.17rc3 soon which should close all holes.

Did you succeed to isolate and debug the security issue causing unauthorized remote access via the WebUI application?

[sirhc]

We hope to release v1.5.17rc3 soon which should close all holes.

Did you succeed to isolate and debug the security issue causing unauthorized remote access via the WebUI application?

v1.5.17rc3 should be released VERY SOON

Yes confidence is HIGH that we have it resolved.

Thank You for your patience, once released and we explain what all we did it will become apparent why this took a bit of time.

Sad part is it delayed WS3-14E pre-release as we had to stop development to apply all resources to this. So altogether about 4 to 6 week delay.

For now running v1.5.17rc2 with the Access Control List (which you should use anyway) does protect you "unless" a device accessible from the web that is infected with some other malware that can be used as a spring board to the switch and that device is granted access to the switch in the Access Control List.

[boxer2t]

We hope to release v1.5.17rc3 soon which should close all holes.

Did you succeed to isolate and debug the security issue causing unauthorized remote access via the WebUI application?

v1.5.17rc3 should be released VERY SOON

Yes confidence is HIGH that we have it resolved.

Thank You for your patience, once released and we explain what all we did it will become apparent why this took a bit of time.

Sad part is it delayed WS3-14E pre-release as we had to stop development to apply all resources to this. So altogether about 4 to 6 week delay.

For now running v1.5.17rc2 with the Access Control List (which you should use anyway) does protect you "unless" a device accessible from the web that is infected with some other malware that can be used as a spring board to the switch and that device is granted access to the switch in the Access Control List.

Any update on the RC3 release date?