where did my bandwidth go?
Posted: Tue Dec 06, 2016 9:01 pm
Howdy ho!
Very difficult to come up with a proper subject line here. Lots to share. Lots of individual facts to take into consideration leaving me wondering where the right place to go for answers is. I came here. yay me! first post too, I think!
Been using Netonix since about April. Very happy. They do great things and really clean up our tower cabinet...
Biggest regret? I purchased a 24 port without measuring our cabinet and wound up having to "temporarily balance" the 24 port unit on top of some cardboard light bulb packaging until i got two 12 port units. It is all better now ! :)
Ok - so my problem is this. It contains a lot of detail so read the whole post before replying please.
There was a fiber cut about a month ago. Our upstream apparently installed another customer and did not reassemble the splice case correctly disrupting our service (their service techs words). I am not stating this as an overall cause - just stating facts.
Sometime after this we got some random speed complaints on our network. Nothing major, just a few to look into. We're mostly a 900 mhz canopy network anyway (although we are looking at some newer gear) - but recently we have added some 2.4 ubnt and EPMP as well - so there are some customers with 10 and 15 meg packages.
Using a mikrotik btest server we determined we were not getting fully capacity incoming but were getting full capacity outcoming. At times it was about a 2:1 radio. If we got 900 meg sending data out we got 400 meg incoming. If we went further away (across the country for example) if we got 600 sending data out we only got 250 data coming in.
At peak usage we are only using about 250 meg. We mainly have the gig circuit in preparation for a fiber deployment
Upon some peer suggestions, we upgraded our mimosa backhaul radios (we have 6 - both b5 and b5c) from 1.4.1 to 1.4.4 . Immediately the GUI of the mimosas showed almost 2x the speed, showing it could do 300+ physical traffic. Indeed using a mikrotik bandwidth test I could send that much UDP. I couldn't send that much TCP....if i was lucky i could send 150 TCP. Running the same test with UDP I could get 305 to 320 meg. I know UDP does not require an acknowledgement packet - but it still amazes me the results are that different. (back to the fiber tests, I could UDP better speeds than I could TCP - but still results were similar)
On additional peer suggestion, I also upgraded our netonix switches (two of them, 12 port) to 1.4.6. They were on a release candidate that was out prior to my installing the switches (which was late summer).
Another peer redirected me to the flow control discussion happening here. I read through some of that. The master side of my mimosas are coming off a mikrotik cloud router switch. flow control is not enabled. my slave side (and 2nd hop master) is a netonix. Flow control is now on by default (I think I read in a firmware update here at Netonix?) The mimosas do NOT show the FLOW CONTROL icon on the status screen like my EPMP radios do. (and my UBNT radios for that matter). Does not matter if flow control is on or off in the radio - the mimoas do not show the flow control (two arrows left/right) icon for beeing enabled.
I am still able to send the above mentioned speedtests out our network - from our core router across to the remote side - so I don't think it is a speed issue there.
Is it flow control somehow on our epmp radios? or ubnt radios? I graph quite a bit using MRTG (and we have some internal graphing with powercode) and it looks like the ubnt radios are performing just fine.
Should I be expecting an overall reduction of 1/4th the bandwidth across our network due to our fiber feed being slower? ( trust me I have a ticket open with them).
We can still pull 400-500 meg from inside our office and our customers before any of this happened were only using 250-300 meg.
When enabling flow control, I know look under stats -> pause frames, but is it like a vlan where you need to enable it on every switch it touches? including mikrotik? does it play well with mikrotik? I have tweaked and tweaked but haven't really found a configuration that gets us back where we were before this began.
Even worse, I don't know what really broke it....
We have over 600 customers. I'd say we've heard from 50 of them. Especially the ones with higher packages.
Thanks in advance.
Jay Fuller
Cyber Broadband Inc
Cullman AL
Very difficult to come up with a proper subject line here. Lots to share. Lots of individual facts to take into consideration leaving me wondering where the right place to go for answers is. I came here. yay me! first post too, I think!
Been using Netonix since about April. Very happy. They do great things and really clean up our tower cabinet...
Biggest regret? I purchased a 24 port without measuring our cabinet and wound up having to "temporarily balance" the 24 port unit on top of some cardboard light bulb packaging until i got two 12 port units. It is all better now ! :)
Ok - so my problem is this. It contains a lot of detail so read the whole post before replying please.
There was a fiber cut about a month ago. Our upstream apparently installed another customer and did not reassemble the splice case correctly disrupting our service (their service techs words). I am not stating this as an overall cause - just stating facts.
Sometime after this we got some random speed complaints on our network. Nothing major, just a few to look into. We're mostly a 900 mhz canopy network anyway (although we are looking at some newer gear) - but recently we have added some 2.4 ubnt and EPMP as well - so there are some customers with 10 and 15 meg packages.
Using a mikrotik btest server we determined we were not getting fully capacity incoming but were getting full capacity outcoming. At times it was about a 2:1 radio. If we got 900 meg sending data out we got 400 meg incoming. If we went further away (across the country for example) if we got 600 sending data out we only got 250 data coming in.
At peak usage we are only using about 250 meg. We mainly have the gig circuit in preparation for a fiber deployment
Upon some peer suggestions, we upgraded our mimosa backhaul radios (we have 6 - both b5 and b5c) from 1.4.1 to 1.4.4 . Immediately the GUI of the mimosas showed almost 2x the speed, showing it could do 300+ physical traffic. Indeed using a mikrotik bandwidth test I could send that much UDP. I couldn't send that much TCP....if i was lucky i could send 150 TCP. Running the same test with UDP I could get 305 to 320 meg. I know UDP does not require an acknowledgement packet - but it still amazes me the results are that different. (back to the fiber tests, I could UDP better speeds than I could TCP - but still results were similar)
On additional peer suggestion, I also upgraded our netonix switches (two of them, 12 port) to 1.4.6. They were on a release candidate that was out prior to my installing the switches (which was late summer).
Another peer redirected me to the flow control discussion happening here. I read through some of that. The master side of my mimosas are coming off a mikrotik cloud router switch. flow control is not enabled. my slave side (and 2nd hop master) is a netonix. Flow control is now on by default (I think I read in a firmware update here at Netonix?) The mimosas do NOT show the FLOW CONTROL icon on the status screen like my EPMP radios do. (and my UBNT radios for that matter). Does not matter if flow control is on or off in the radio - the mimoas do not show the flow control (two arrows left/right) icon for beeing enabled.
I am still able to send the above mentioned speedtests out our network - from our core router across to the remote side - so I don't think it is a speed issue there.
Is it flow control somehow on our epmp radios? or ubnt radios? I graph quite a bit using MRTG (and we have some internal graphing with powercode) and it looks like the ubnt radios are performing just fine.
Should I be expecting an overall reduction of 1/4th the bandwidth across our network due to our fiber feed being slower? ( trust me I have a ticket open with them).
We can still pull 400-500 meg from inside our office and our customers before any of this happened were only using 250-300 meg.
When enabling flow control, I know look under stats -> pause frames, but is it like a vlan where you need to enable it on every switch it touches? including mikrotik? does it play well with mikrotik? I have tweaked and tweaked but haven't really found a configuration that gets us back where we were before this began.
Even worse, I don't know what really broke it....
We have over 600 customers. I'd say we've heard from 50 of them. Especially the ones with higher packages.
Thanks in advance.
Jay Fuller
Cyber Broadband Inc
Cullman AL
