As a general cynical view of the world, people take liberties when they think they can get away with it.
You can see that with the criminal families, elements of the travelling community etc.
So the only solution is to act tough, by that I mean show them that you know the law and are prepared to use it. Fair play to you for not fighting fire with fire (stooping to their level), because ultimately the guards and the courts should favour the one coming to court "with clean hands".
I'd agree with everyone else here, you need to get a few "scores on the board", talk to your solicitor. Have a chat with the local guards and give them your view - in my experience the majority of guards are reasonable types and probably have more sympathy with your plight than the point of view of your neighbours (especially if they've been out on fools errands before).
The only other thing you can do is make sure you are squeaky clean - overdo the fencing so sure no breakouts, be careful on pollution control, dates for spreading slurry, go handy on the local roads in tractors etc. etc.