our flow control testing - what to do?

[cyberbroadband]

I have read quite a bit about this in the last few weeks. I thought I had configured everything right and at times had better results but we are again getting some complaints. I've seen screen shots where people are reporting 3 down and 80 up. After first implementing flow control they'd get almost as good speeds down as up but not always. Here are my guesses and our configuration. I'm not an expert on network administration but, well, for now, we are what we are.

At our core we have several backhauls going different directions. Most are serving older canopy 900 radios that only do three meg so they never complain anyway (they are happy to get 3 meg). We have one backhaul (rocket m5) going east that i've seen spikes to 27 and 30 meg and a main backhaul going west (mimosa b5 / mimosa b5c) that is about 70 meg during the day and can push 200 at night.

Most segments are relatively short but there are a few multi hops. Here are some examples -

[east segment] core -> (rocket m5) -> east tower 1 (rocket m5) -> east tower 2

[west segment] core -> mimosa b5/b5c -> west tower 1 -> mimosa b5/b5c -> west tower 2 (which has feeds to west tower 3 south (rocket m5) and head end west 4 - west) - (mimosa b5 / b5c)

the core has a one gig fiber going into a mikrotik core router going into a powercode bmu going into a mikrotik could router switch.

The east segment is mostly dumb switches and packetflux timing equipment. I think the generic 5 port or 8 port dlink switches from officemax. There is some cambium epmp gear on this tower and those customers get high speeds, no problems.

The west segment (west tower 1) receives the bandwidth from the core over a mimosa and terminates into a netonix switch - the ws-12-250-ac

There is - as stated above - another mimosa going out from west tower one almost due south to west tower 2 - also powered by the ws-12-250-ac

There is a lot of gear at west tower 1 - it is a major hub site for us. There is also a 2nd ws-12-250-ac because our initial 24 port netonix switch would not fit into the cabinet we have

in this first ws-12-250-ac the ports are as follows:

port 1 - mimosa - our incoming backhaul
port 2 - mimosa - out going feed to west tower 2
port 3 - a rocket 5ac prism feeding two other towers (almost north / northwest)
port 4 - a rocket m5 feeding a tower just a hair north of west
port 5 - a nanobridge m5 feeding two towers (azimuth about 3 degrees off but 9 and 12 miles away)
port 6 - off - was an old radio the rocket 5ac in port 3 replaced
port 7 - an old 2.4 rocket feeding a tower 3 miles south
port 8 - epmp 2.4 radio #1
port 9 - epmp 2.4 raido #2
port 10 - epmp 2.4 radio #3
port 11 - epmp 2.4 radio $4
port 12 - downlink to switch #2

switch #2 has all of our 900 mhz canopy radios on it (There are eight on this tower)

we had received speed complaints on this tower and the faster upload than download problem but once enabling flow control on our core mikrotik back at the office and on the mimoas radios themselves we have not received any such complaints.

as many of you know (and i learned from reading), all the ubnt radios negotiated flow control (even the old bullet) - but the mimoas radios did not.
mimosa says their flow control works (and it does generate a few pause frames - not many). netonix reports (accurately) that it is not propertly negotiated with their switches (the icon indicates it is not on). All radios are in "auto" duplex.

I have read in the 80 mile long ubnt post that if you force a gig radio to 100 meg radio the problem will go away - - at least on toughswitches. We don't have any touchswitches.

Downstream from west tower 2 we have one of our customer service guys (west tower 2 is on his property).
again this tower feeds quite a bit and at this location is a mikrotik rb2011il.

flow control has been enabled on this switch as well. no way that i know of to view pause frames.

basically from west tower 2 several other head ends are fed - west tower 3 (which is a head end for a network) feeds different networks with ubnt ac gear. also has some 900 on it and also a m5 rocket feeding customers. at the end of the 2 hop ubnt ac gear backhaul shot is a network with lots of epmp 5 ghz reporting much faster upload than download.

at the end of the mimosa link from west tower 2 is another head end with many rocket 5acs feeding four different directions (and 200+ customers)
at this location we have a mikrotik x86 box and flow control has been enabled on all of those ports too.

i've read in some places it is best to not enable flow control on your gig backhaul ports - but enable it on everything else.
i believe i have read *here* it is best to enable it on everything (i may have misunderstood)
it is currently enabled on everything.

i have read mimosa's flow control does not negotiate propertly (here) but i have read it does work (on their forum)

fact: After implementing what i believe is the proper flow control implementation we're still seeing problems 2 hops down or more
on the east side of the network, again, no mikrotiks or netonix, no flow control, also no 1 gig uplink, no problems

i read on the ubnt side if you have 100 megabit links instead of 1 gig link there is no problem (i may have said that again)

happy to answer any questions - just curious what to do next.
obviously i could have posted here, one of the many mailing lists, one of the ubnt lists, i felt most people here had dealt with this the most.
appreciate any advice.

thanks!

Jay Fuller
Cyber Broadband Inc
Cullman Alabama

[mhoppes]

Hey Jay. A few things that totally made a HUGE difference for us:

Traffic shape at your head end if you don't already (this will keep huge amounts of traffic from trying to stuff themselves down your backhauls and into the APs).

Route at every site... route route route... every site. Even the ones too small for other's who's mouses...