FCS / CRC errors

[Omniflux]

I am having an issue with FCS / CRC errors on some new deployments.

WS-12-250-DC Port 12
80ft Shireen Cat5e/Cat6 shielded (I've tried both)
Mikrotik CCR1009-7G-1C-1S+PC Port 7

Everything works well if I power the Mikrotiks directly, but I get multiple errors per second when I power via PoE, either 24V or 48V.

I do not see the issue in deployments using short (6" - 5') patch cables.

I have many sites using this same equipment with no problem, but these four new sites are the only ones I have where the switch and the router are not in the same enclosure.


For grounding, I have a solid 8 gauge copper wire running the length of an 80' utility pole, ending 10' below ground. 1'ft below ground, it is crimped to a horizontal 8 gauge copper wire running 10' east and west, and attached at each end to a 10' 3/8" ground rod sunk 11' below ground.

At 65', the switch ground screw is connected with 2' of 12 gauge stranded to a busbar which is connected with 5' of 8 gauge solid which is crimped to the wire running the length of the utility pole.

At 6', the router ground screw is connected with 2' of 12 gauge stranded to a busbar which is connected with 10' of 8 gauge solid which is crimped to the wire running the length of the utility pole.



Is there something obvious I'm doing wrong? Is there something different I have to do with longer cable runs?

[Omniflux]

I've narrowed this down to only occurring when I have a Prism AP powered on.

Powered off but still connected it does not occur.

Powering an AF24 is not an issue...

[sirhc]

For grounding, I have a solid 8 gauge copper wire running the length of an 80' utility pole, ending 10' below ground. 1'ft below ground, it is crimped to a horizontal 8 gauge copper wire running 10' east and west, and attached at each end to a 10' 3/8" ground rod sunk 11' below ground.

At 65', the switch ground screw is connected with 2' of 12 gauge stranded to a busbar which is connected with 5' of 8 gauge solid which is crimped to the wire running the length of the utility pole.

At 6', the router ground screw is connected with 2' of 12 gauge stranded to a busbar which is connected with 10' of 8 gauge solid which is crimped to the wire running the length of the utility pole.

Does this site have electric from utility, and if so is the electrical service ground rods "bonded" to the pole/tower grounds?.

[Omniflux]

I just went to verify. The ground line coming out of the electrical service box is 8 gauge solid, about 4' long and connects to my ground wire 2' above ground level. It does not have an independent ground system.

I have also tried shutting off the service breaker and see the same issue when running off batteries.

[Omniflux]

I have simplified this as much as possible for testing.

Test setup

With the power to the rocket off, there is zero packet loss between the mikrotik and the netonix.

The moment power to the rocket turns on, I have 5% packet loss between the mikrotik and the netonix. This is before link is even established between the rocket and the netonix.

Should this test setup work?


All of the new sites I am having this issue with are using new netonix units with the square holes (ec:13:b2:83:*).

I did this test with a unit with circles holes (ec:13:b2:81:65:2c). No problem. Swapped out this netonix at one of the sites having this issue. Problem solved.


Were there any changes around this time? They are both board rev F, power supply board rev B.

[Dave]

my guess is RF interference from radio....what happens when you do same test (your picture test setup), but use a longer cable and move radio away from switch/router?

[Omniflux]

This is a good idea. I just tested by taking the radio 60' away with no difference, however I think that if this is the problem I need to be grounded for the test to be effective and I am out of daylight. I will try again tomorrow.

Later tonight I am going to go swap out one of these new switches at an existing site with no issues and bring its older switch back to see if it resolves the issue as the other older switch did, or if that was just a coincidence.

[Omniflux]

Tested with radio 60' away and grounded. Still getting FCS errors.

Took one of the new switches and swapped it at a site with an older switch that had zero errors. After 18 hours, the new switch has 57 errors out of 264 million received packets. Not zero, but low enough I'm not too concerned.

Used the old switch in this test setup. Zero errors.
Installed the old switch at one of the new sites. Zero errors. Once again this solved the issue.

I see this issue with these switches:
ec:13:b2:83:43:94
ec:13:b2:83:58:e8
ec:13:b2:83:5b:ce
ec:13:b2:83:75:90


I do not see this issue with these switches:
ec:13:b2:81:26:aa
ec:13:b2:81:65:2c



Any ideas what could be different with these newer switches?

[sirhc]

hmmmm?

[Omniflux]

Just received some new switches in with manufacture date 03/12/2020 and am still experiencing this issue.

To recap, I am experiencing a huge number of FCS / CRC errors between a WS-12-250-DC and a Mikrotik CCR1009-7G-1C-1S+PC when both the Mikrotik and a Ubiquiti Prism AP are powered via PoE from the switch.

I see this with all switches with square air vent holes and RMA 3642 which had a board replaced recently.

I do not see this with any of my older switches with round air vent holes.

I have taken six of these newer switches and swapped them with perfectly working old switches at six different sites and the problem occurs at all of them.


I have now used up my stock of old switches and so need to come up with a solution to continue new deployments (as well as have spares).


@Stephen @Dave Can one of you reproduce this?