HELP- Netonix Virus?

[mhoppes]

Fair enough. I was just relaying what I had observed in case it shed any light on anything.

[sirhc]

OK UPDATE ON ANTI VIRUS

After checking I found that the reason only 1 computer in house needed Avast Disabled was other computers were protected but a recent AVAST upgrade needed a reboot to fully kick in.

So if you are running Avast AND IT IS FULLY UPGRADED AND SUGGESTED REBOOT DONE you need to disable all or 1 part of Avast as described above or you get a "can not connect to page" when trying to access switch UI.

Any further details on other anti virus effects would be helpful.

[mhoppes]

So should it be assumed that these devices were actually hacked and part of a DDoS network and the FBI shut them down legit then? As in someone hacked the lighttp system, installed something and the FBI command and controlled and disabled it?

[sirhc]

Fair enough. I was just relaying what I had observed in case it shed any light on anything.

Just a short note on this Stephen can answer more in detail

So If I login to switch see stats then just close tab without logging out on UI switch think session is active then on same machine you go to address and web browser cache opens connection the switch thinks it same session for a bit.

Or your local browser session tims out but your not moving around in UI but switch side server is still active then you move around and browser says hey im timed out but telemetry is still coming from switch and if browser page is inactive but graphs are a routine running and still getting data so make a move in the UI and the UI does a check and POP your terminated.

This is normal behavior. We could write little ack and nak handshakes to continuously check and fore page reload but why add overhead for no reason keep in mind embedded devices use very small CPUs to run UI/CLI and other needed routines but packet switching is handled in the switch CORE which is separate.

UBNT unifi devices that require a seperate site or device to configure have such a small embedded CPU to configure core and run routines they can not handle a little web server for a local UI for device config.

[rockhead]

FWIW I had two switches with this behavior that I pulled, on the bench only one of them persists.
I changed passwords on all switches on that L2 segment, just because.

One switch with virtually perfectly reliable power rebooted itself at ~1:10 AM PST today

[sirhc]

FWIW I had two switches with this behavior that I pulled, on the bench only one of them persists.
I changed passwords on all switches on that L2 segment, just because.

One switch with virtually perfectly reliable power rebooted itself at ~1:10 AM PST today

Read above it's easy to remove infestation.

Clear your browser cache.

Will only come back is it is pushed by hacker again but if on invalid ip then you have an infested device allowing access into private Lan. Could be a workstation, another device, or server as a bridge/springboard.

Note anti-virus issue. Guess is this vulnerability was recently discovered as latest avast refuses connection to lighttpd if vulnerable version regardless if infected. Not cool, checking how other anti viruses handle it.

[wtm]

Well after going into the field all day yesterday and upgrading all our switches to 1.5.16.

If the device has 1.5.12 in it, then you MUST first upgrade to 1.5.14, then to 1.5.16, you can not go directly to 1.5.16, it will fail.

Still wondering HOW this happened, did see the log entries regarding lightpd, and we did have some devices that were on Public IP's. But most devices that we physically had to go to them to upgrade( couldn't get to them remotely on network) were private IP's and could ONLY have picked up the bug if the switch contacted something outside of the internal network first. We also do not have any discover modes turned on.

This covered many different subnets, and we also had many different passwords on these switches, so how did the bug LOGIN to every one of them to do this?

SO is this a WORM, or Virus ? Does something in the Netonix firmware have "Call Home" programming, to deliver stats, or check for updates, or anything like that ?

Has Lightpd been updated in the firmware ?

This bug seems to only infected the main http screen, and seems to have overloaded the switch causing erratic problems. It seems to block out SSH, but does NOT change passwords on the switches.

[Stephen]

So a couple things.

But how was a new file uploaded?
...

REGARDLESS -- Vulnerability or otherwise.... this still says that the individual did not properly isolate their management network and has other infected machines on the network that infected the switch I would think.

This is our also guess presently.
We figure there are 2 possible attack vector's for this infection.

1. The infection was on a malicious site or they potentially created a malicious post on the forums that spread the exploiting code to browser's, stored as metadata, when the infected browser then connects to a switch, the exploit runs on the switch and the index.html file is created. (note after analysis this is unlikely)
2. The attacker utilized infected system's on victim's network's which actively searched for vulnerable system's on the private network connected to the infected machine (our switch's, and potentially other device's running lighttpd) which then attacked them.

We think it's likely scenario 2 right now because we found 2 switch's on our network that where infected, and checking the timestamp on these file's showed that the malicious index.html file showed up at exactly Aug 1 at 5:06AM simultaneously. On top of that there is only one person who maintains that switch and they would not have been in the office at this time. So there would have been no chance that the browser was connected to do the exploit from scenario 1.

... it *seems* to me like security is being handled by a Javascript entry???? Why is it not being handled by PHP sessions on the webserver side. Rather, it *seems* to be handled by the browser on the local side.

The webui use's session cookies for authentication. In short, when you login successfully, a cookie (which is just an entry in a list in your browser) is stored on your computer. When a request for status is sent to the webserver the cookie is sent in an http header which must be authenticated on the server before it will respond.

I just verified... I can log into a switch, destroy the session on the local browser side.... and still interact with the web interface and see real-time information updating (this shouldn't be able to happen if the session is destroyed and appears to be an attack vector).

Can you provide more detail about the verification? Note, that closing a tab or exiting the browser doesn't delete it's local memory. So a session cookie will be persistent until it either expires, you manually clear it (by for example in chrome -> settings -> clear browsing data -> and make sure "Cookies and other data" is clicked) or click "logout" in the webui.

...even things like the tarpitting seem to be handled on the browser side of things, rather than the webserver side of things.

The tarpit is handled in php, it checks the number of invalid logins attempts for the current unauthenticated session and if they exceed a number defined in the settings it blocks further attempts for a set period of time. This is executed on the server.

So it's important to know that even with an authenticated session, there is no normal mechanism for uploading file's into directories. With the exception of upgrading, but you cannot create a file called index.html in this case. Most likely the attacker exploited a vulnerability somewhere in the system in order to run malicious code, which then generated the file,

This might also means unfortunately, that having an authenticated session is not required. However, if the switch is theoretically using the firewall feature, these are configured via ip tables, so it would have prevented infected server's from reaching the webserver in the first place. So that is one method to prevent this while we work on a patch to remove the vulnerability. However, if the victim's machine is the same one primarily used to access the switch's webui, then it's still vulnerable.

[Stephen]

As far as the details of the infection itself. After a fairly extensive analysis. So far, it appears all it's really doing is adding this file. This file is loaded into the lighttpd service which utilize's a lot of extra memory due to the image data. This disrupts other process's and, as sirhc mentioned, if you check the lighttpd process's while the file is active it is using almost twice the memory it normally does which cause's issue's on other proc's. So far as I can tell, it doesn't appear to be trying to do anything else. Its just because our system does not use a powerful CPU and the RAM is relatively small. So when something like this waste's extra resource's it can affect the behavior. Clearing it has appeared to allow the switch to function normally for us.

[Stephen]

SO is this a WORM, or Virus ? Does something in the Netonix firmware have "Call Home" programming, to deliver stats, or check for updates, or anything like that ?

We think it's a worm presently, the switch does not phone home for anything. The webserver presently only responds when asked. This is how it works when using the webui or the manager.