Talk to your neighbours. You say they wouldn't be able to see it from their side, so it seems odd that they would go to the bother and expense of putting it up then. Perhaps the reason they put it up could be addressed in some other way, and so it could be taken down. (I'm thinking of perhaps a dog on your side that goes into their garden, in which case a low chicken-wire fence may do the same job while being less visible). Failing that you can either plant in front of your side of the fence, or paint it and make it more aesthetically pleasing.
(As an aside, it's amazing how many posts on AAM could be addressed in the first instance by "talk to your neighbours". It is possible, I've done it myself and it's all about a non-confrontational approach. And it's worth it, a good neighbour is worth their weight in fence-posts / parking spaces / skip bags / overhanging drainpipes)