Network Managment

[LRL]

So I'm finally to the point where we're getting a /20 from ARIN and I've decided to review our IP management. I'm curious what some of you guys are using.

How do you keep track of IPs, do you just use DHCP and if so how do you handle keeping records of who had what IP at what time?

If you're willing to share I'd appreciate it and why you do things the way you do.

Thanks

[Ant]

Billing/radius software usually keeps the records of whom has what ip at what time. For us an x86 server running Mikrotik's RouterOs does the job very nicely. We make use of the usermanager and the Dude to keep logs of all traffic coming and going through our network. Comes in handy when you get those Copyright letters if you NAT or provide public ip addresses.

[rebelwireless]

I'm lame, I DHCP the client and then make it sticky. That way the IP is consistent and I don't really have to worry about tracking it. DHCP server for each POP, not centralized.

I do a /24 for each full POP and a /26 for micro pops.

I have a sql-express db with a lightswitch frontend that I track this in via script from a linux box that pulls the DHCP table and shoves it to the db. I've been planning to do a lot more but I don't really need it right now and I'm more focused on traffic shaping. Ideally I'll have a single cluster handling centralized DHCP, traffic queues, etc. tcng has been a bit more of a pain than expected in getting exactly what I want unfortunately.

[adairw]

Guys! It's almost 2015, get a billing system that will do all this for you. Personally love powercode, but there are lots of platforms out there also.

We have somewhat of a unique network setup that allows us not to have to subnet block of addresses out to towers and waste huge portions of them. We are 95% DHCP and powercode manages it all. So I can tell who had what address at any given time for DMCA requests, etc. happy to share more if people are interested.

[rebelwireless]

I'm interested to know how you do it adairw

[adairw]

I'll first say that there are many ways to skin this cat. This is just how we do it and it works really well. It ended up a lot longer than I expected. But I figured I could drop a little info and leave people wondering or a could give a pretty good overall layout. This isn't a typical wisp setup, that's for sure. But I get 253 subs per /24 and that's something most wisps can't say.

Here we go,

We have a fully OSPF routed network using mikrotik routers. All backhauls live in /29's and AP's are generally in /26's. We use MPLS to create LDP full mesh connectivity between all routers. This allows us to run VPLS tunnels to all our towers. I think of this as a route-able VLAN/full layer 2 tunnel, meaning, if OSPF re-directs traffic across another link the VPLS tunnel follow it.
At the core, we build a VLAN on our BMU (bandwidth management unit) which is just a powercode specific router/network monitor. Each VLAN gets a /24 of public address space. We trunk these VLAN's to a CCR1036 that we refer to as our "MPLS Router" This is basically where all the radio's in and out of the data center connect. In the router we bridge the incoming VLAN's from the BMU to all the individual VPLS tunnels to each tower.

For instance, if we had five towers and three /24 blocks I'd run 3 tunnels to each tower (15 tunnels total). These tunnels would be in a bridge with their respected VLAN from the BMU. This makes every block of address space available at every tower. We could move a customer from one end of the network to the other and never change their public IP.

Everything is consistent, I.E VLAN 1201 at the BMU is VLAN 1201 at every AP on every tower, etc..
We utilize split horizon bridging to restrict traffic in these tunnels from going anywhere but tower to core or core to tower. We do not allow any tower to tower comms inside the tunnels UNLESS you are routing to another public subnet. There are rules on the core router that further restrict and ensure that customer can’t get to ANYTHING except the internet.

Our network can be thought of in two pieces. The routed management network and the customer/delivery network. CPE's are managed with a private IP on the /26 of the sector it's connected to. Since the customer is either bridged in to the the public vlan's (1201, 1202, etc) there is no way for them to ever get to internal resources. they can only see down the tunnel and hit the BMU. Plus most of our CPE's are in router mode. This lets us use the WLAN mac of the CPE to put in Powercode so that we aren’t chasing the customer down to update their mac address every time they change their router around.

I like to think of this as network cables laying on top of a routed network. If you connected a network cable to a router with a /24 and plugged something in on the other side (assuming DHCP was running) you'd get an address and go. To add more cables (towers) you just plug them in to a bridge on one side which allows those towers to share the same address space. It's the same at the tower side with all the AP's, all the public VLAN's get connected to the bridge (think switch) with an UPLINK to the core where DHCP is running.

Powercode uses MAC auth to track the IP address. When you add the equipment to powercode a public up gets assigned and that user basically will have that address forever unless you tell powercode to assign another one.
We can also assign static IP's to people if we want. Whatever address powercode assigns to the user they can use to configure their router with. We just have to use a global matching mac address in powercode of 00:00:00:00:0A. This means that as long as something comes on the network with the matching IP, it'll be allowed online and tracked via their account. If I wanted to take it a step further and make the customer give me the mac address of their router, I could do that so that I know that user is using the IP I've assigned and not someone else.

This put's 95% of my IP address management in powercode. Other then some other smaller blocks I use inside my data center for servers which I just track on a google docs spreadsheet, powercode does the rest

There are some specifics I didn't go over. This obviously isn't a how-to on our network setup. But I wanted to be detailed enough that people had a good idea of how this works for us. It probably sounds more complicated then it really is, but it's actually very easy. We use config files for the CPE that are made for each AP and aircontrol tells us the next available IP address we can use for management of the CPE, which also goes in to powercode for monitoring. This allows us to pull up a customer account and see what their managment IP is, whether it's up or not and what their pubic IP address is.

[rkelly1]

Thanks to Adair and his guidance on this setup, we've been converting our network to this setup (minus powercode so far). It's pretty nice.

[rebelwireless]

Adair, do you have any backup or alternate paths for uplink? For instance, I have my primary backhaul into a datacenter and then on a different path I have a DSL (resellable!) as a backup. I push default route via OSPF and have this DSL with a very high path cost and then it NAT's. Additionally, I've been pushing out a couple low priority things like windows updates that way w/ static routes based on their ASN's perfixes.

so, that's a long winded way of saying I have 2 different paths to the internet. If I were to do this VPLS bridging back to the core, based on your example above, I would not be able to route anywere along the path.

Do you, or anyone else have a solution to this limitation?

btw, I really like your solution, very elegant, wasting very little IP space, and being OSPF+(BFD to I assume)+MPLS we can use TE tunnels.

I'm in the middle of a conversion, updating backhauls, expansion, and replacing some older 'tik routers (493) that don't have enough MTU for QinQ and/or MPLS. Giving the new 850x2 a shot at 2 towers w/ existing TSP8 and VLANs stubbed down to ports.

[adairw]

Adair, do you have any backup or alternate paths for uplink? For instance, I have my primary backhaul into a datacenter and then on a different path I have a DSL (resellable!) as a backup. I push default route via OSPF and have this DSL with a very high path cost and then it NAT's. Additionally, I've been pushing out a couple low priority things like windows updates that way w/ static routes based on their ASN's perfixes.

so, that's a long winded way of saying I have 2 different paths to the internet. If I were to do this VPLS bridging back to the core, based on your example above, I would not be able to route anywere along the path.

Do you, or anyone else have a solution to this limitation?

Can I assume these two connections are in different geographic locations?
The cool thing about our network is the tunnels follow you. So right now I have two upstreams in my data center. But no geo redundancy at this time. But if I did, I think (and I need to test this on the bench) as long as I have another router with an identical config somewhere else in the network advertise the same loopback address in to OSPF of other router the tunnels would establish a connection to that router and everything would keep on working.
It's really no different than normal routing. The tunnels just have to be terminated somewhere that has the IP space for them. It does mean that I pretty much have to have an exact copy of my hardware running elsewhere. MPLS router, edge/bgp, BMU, etc.
One thing I'm not sure about is how to control advertising the loopback address of the MPLS router. Sure if it's dead, it wont be there. but what if it's not and something upstream from it is? Details I'd like to test one of these days.

Since the building we are is basically a "carrier hotel" we have lots of options for upstream providers and unless the building blows up, geo redundancy isn't really a huge deal to me. If the network ever get's big enough to warrant that I think we'd just segment the network to a closer fiber pop and either backhaul it to the data center or just have that network stand alone.. dunno..

What I'm working on now is building out rings in the network so that if I lose any one piece of equipment I can still operate. I have 6 high capacity links that feed out from my data center and I should be able to build three distinct rings to return traffic to the data center. My goal is to have two wispswitches, two mpls routers, two edge routers, two bmu's etc. so that no matter what fails things will route around and keep on working. Again I'm about to start labing this to work out the kinks... I'm sure I've over looked something.

btw, I really like your solution, very elegant, wasting very little IP space, and being OSPF+(BFD to I assume)+MPLS we can use TE tunnels.

Thanks!

I'm in the middle of a conversion, updating backhauls, expansion, and replacing some older 'tik routers (493) that don't have enough MTU for QinQ and/or MPLS. Giving the new 850x2 a shot at 2 towers w/ existing TSP8 and VLANs stubbed down to ports.

Careful, I don't think 4xx boards support large enough MTU to do much outside of VLAN's. MPLS needs 1528 or larger MTU and I think they only support 1526.

[rebelwireless]

"Careful, I don't think 4xx boards support large enough MTU to do much outside of VLAN's. MPLS needs 1528 or larger MTU and I think they only support 1526."

493G actually only 1520. MPLS needs 1524 (IP=20,MPLS=4), adding a VLAN to the mix takes it to 1528.

I'm replacing these units with RB850Gx2. It's a dual 500MhzPPC board with 1580MTU on the ports. This seemed to be the best combo of price and performance and should be able to handle the AF5X backhaul that's on pre-order very nicely.

I started with the 493 and injectors, then I added the TSP8's to get away from injectors and am powering the RB493 via ETH1 from the TSP8 port1.
When I get these RB850Gx2 units, I'm going to swap the rb493 out and power the RB850Gx2 from the TSP8 also, but then I'll be able to stub VLANs down to the TSP8's individual ports.

I'm undecided on sticking with straight up OSPF routing, or adopting an MPLS setup like yours. Right now, I run a script on a linux box and create and address list and push it to my tower routers, that way I can tag+route those prefixes out the DSL.