2 Fiber providers in one local or separated geographically

[rkelly1]

Hey guys - We are moving quickly towards a second fiber provider and BGP. We have the option a) to have the second provider at our main tower (where the first provider is located), or b) 20 miles away at a tower on the other side of our network.

In option A, the redundancy/failover is easy to setup, etc. but provider #2 can't tell us the physical route until after we place the order and they figure it out - if the route is along the same path or poles as provider #1, then there is some risk if a car takes down a utility pole, etc.

Option B physically separates the paths but I think it gets more complicated on redundancy and how I set it up for failover, etc. I think either way we are going to have to have a high capacity wireless link running between the two locations - because in option #1, we have to backhaul hi capacity there for some growth (and for towers along the way), and in option #2, have to have have the high speed wireless connection in case of failover.

Can anybody provide insight or experience on which approach would be preferable? Thought I'd ask all the geniuses!

Rob

[sirhc]

Yea we are considering a second provider as well with the same concerns.

I think we are going to go with 2 locations for better fail safe.

[rebelwireless]

Is this second provider just for failover? BGP is much better or more suitable for multipathing that the 'failover'. With geographic distribution, you end up typically running a couple BGP sessions so you can logically separate subnets between your fiber links, that way you can prepend your ASN on routers+subnets that are less prefered for a specific subnet. FYI, many ISPs won't accept anything smaller than a /24 so double check with both your providers if you are going to try to prefer certainly links for certain subnets. If you don't do this, I'm sure you will have very unbalanced usage as one provider is likely a few hopes closer to the 'internet' at large than the other.

What I've done in the past is assign each pop a subnet(s) and then manipulate OSPF path costs so that they will pull the default route from the uplink I want. Additionally, advertise that subnet out that link with no extra ASN prefix. Advertise my entire subnet range with 3 prependeds so it acts as a failover failover. Then just watch your load on each side over time and move pops at the 'network divide' to one side or the other to balance. I had 180com at the time (now bought out by integra) and was lucky enough to be able to route /26s with them.

Now (second wISP), I cheat like a mad-man! I have my primary link w/ backup that all routed IP addresses go through. Then I purchase a couple of failover links and simply NAT clients out to handle redundancy. OSPF distributes the default routes with the backups having a cost of 200 so they never 'win' for shortest path unless the primary is down. Then just focus on keeping the primary route up with better routers, switches etc.

[rkelly1]

Thanks Rebel! The second provider is for a combination of redundancy and additional bandwidth. Second provider Centurylink is coming in 40% less.than our current provider. My thinking is that we can save on addition bandwidth, have redundancy, have leverage to lower pricing on primary provider , and if primary doesn't come.down maybe transition bulk of traffic to CL. we get GREAT service from our primary so it may be worth more..will see.

I'm still kinda torn between separating them geographically or not. Trying to decide which is more efficient and cost effective.

[rebelwireless]

It's not to hard to logically partition the subnets for links. Having far side customers on the near side link and vice versa is a waste of back/sidehaul radio links IMO. The hardest part of separating the traffic is how to slice up your subscribers to best utilize what's available. The plus side is that it won't matter which path data takes in or out so you can make these changes online.

Now, maybe the most annoying part of doing this is if your 1st link's backhaul link is down but the router is up. You'll need to be able to script tearing down the BGP session else it will keep advertising even though your network can't access that router.

Another annoyance is how to shape clients. If you are shaping at the edge, you'll have to find a way to replicate those queues between both head end routers etc. Easy enough with a little scripting in most routers -or- with a dedicated linux box on network to do that via SSH.

[rkelly1]

Thank you for input on this guys!

[rebelwireless]

Good luck, hope to hear what you do and how it works out.

[rkelly1]

Last week I signed the order for the second fiber link for a tower at the other end of the network. We have a learning curve ahead of us!

[TheHox]

I am just starting the same deal myself, but the opposite scenario you describe. I started out with bandwidth at a tower 9 miles away from what was my tiny core area(figured I would grow to provide coverage that direction anyway), but 2 weeks ago signed a 5 year fiber contract at my main tower right downtown where our office is. But with the same goal in mind, giving us a geographically redundant connection.

The new fiber is a full 1g, while the bandwidth at the tower 9 miles out is only half.
I was planning on doing something along the lines of what rebel said, BGP prepend to manipulate the load over the two. But depending on how the new fiber routes/latency looks, I may just cheat as you say and use it for a failover.

I'm sure I'll be posting questions soon. I meet with the fiber engineer Friday to go over the construction to the tower.

[rebelwireless]

TheHox, you would still probably want to do BGP and if you were favoring one connection over the other for all traffic, you just prepend your AS more.