Identifying torrents/p2p, more random thoughts

[rebelwireless]

I've been marking connections w/ L7 on mikrotik to catch torrents and a few other p2p apps, but I'm trying to think of ways to up the ante.

So I was thinking, what if I were to track connections and pull IP addresses. remove anything known from either firewall rules or previous analytics. Then reach out to each IP and and identify if it is a real web-page or email server etc, and if so add that IP to a higher priority address list and store the IP in a database for quick classification later. everything else sits in the 'bulk' or low priority queue.

maybe only analyse connections to customer IP with a high number of connections or something that might indicate a torrent download.

I'd do this either in bash script or python on a linux box.

[Rory]

There is a L7 package that can be installed in the OS of the switch. I'm almost positive it is not currently installed, as it would not be needed by any of our current features. But installing that package may provide the same capability that you have in your other gear. I"m honestly not familiar with the package, but AFAIK it is the same code you find in a few third-party firmware packages that helps classify traffic. If you are curious to play with it, go grab something you can install openwrt on, and check out the L7 packages (on the software page).

This is a neat idea, I would be curious to see it pan out.

[josh]

Procera is already doing this for us, but it's got some very nice custom hardware offloading to make it happen.

(the list shown is not all-inclusive, and it's broken down into 3 different sections)

[josh]

You could take a look at nDPI, but licensing is going to cost you a bit (and it may be x86/amd64 only).

[rebelwireless]

You could take a look at nDPI, but licensing is going to cost you a bit (and it may be x86/amd64 only).

nDPI identifies traffic well, but it doesn't have the ability to mark packets. There is an nDPI-netfilter module but it is unsupported and only builds against older nDPI releases, and it doesn't identify quite as well as nDPI. I have a test box marking packets with this and it does work. Identifies torrents in just a few kbs.

[rebelwireless]

I should add a few tidbits here as I've been digging in on nDPI pretty hard over the last 2 months.

nDPI *is* cross platform, it will build on arm, mips, ppc, x86, x86-64.

nDPI-netfilter allows you to use very simple iptables mangle rules to mark packets and/or connections based on nDPIs matching.

It's also pretty light weight, surprisingly. My test lab as pushed 300Mbps of bittorrent across an ALIX (amd geode 500Mhz) with moderate CPU usage. I can't build it for an edgerouter because of lack of kernel headers else I'd try it there.

I have been talking with the guys at nTOP and they may be interested adding the capability to mark packets w/ nTOP & nDPI. I'm thinking about starting a kickstarter to fund development.

nDPI offers some 'holy grail' type matching IMHO.

it's light, so it can be run on CPE allowing marking on both edges. Tower side shapers can handle last-mile ingress shaping thus keeping pesky UDP streams from consuming the network just to be dropped at the head end. It runs on linux and there is a lot of great hardware perfect for running this.

For a 'product', this could be very interesting. Products like procera are dinosaurs, they do have some great features but they are still in this TCP model of shaping with a casual effort put into UDP (or other connectionless stream) shaping, simply because you can't shape the traffic you haven't seen and there are no congestion techniques to employ with connectionless protocols. connectionless streams on the wire will eat up throughput until they hit the shaper. It's kind of crazy to transport those packets just to drop them at the shaper. The BEST shaper you could make, if you ignore a few technical hurdles, is one that identifies traffic at every ingress so it can be shaped before it eats up bandwidth. Your head end shaper, ideally, would never drop any outbound packets. Those would all have been dropped or queued at the CPE (again, ideally), then again on the tower and at each hop. Putting a procera box at every tower would be expensive, and putting one at the CPE would be ridiculous.

I don't expect to get nDPI into CPE, some people would be opposed to adding yet another task for the CPE and ubiquiti is unlikely to add this since they won't even consider OSPF which would be rather simple to add due to existing code. I do really like the idea of tower side shapers with nDPI. I already shape at the towers (just the backhaul, PCQ to maintain low latency) and at the headend (actual service level queues here). Be great to up the quality of those shapers :)

[josh]

The ntop-ng guys will sell you a working/new netfilter module. At least, that's the info they sent me about a month ago before we had decided on procera.