AFAIK, you can do whatever you like on your side UP TO but not including the boundary. Altering a boundary itself (i.e. replacing a hedge with a wall can only be done with the consent of both parties.
The neigbours you describe have been a bit cheeky in assuming that the other wouldn't ever want to do the same as they have and my opinion would be that what they have done constitutes an invasion of your privacy. I assume they didn't go the full hog and actually USE the boundary to build the wall. This would be a complete invasion.
My solution would be to place a trellis or fence right up against your side of the boundary which would block their view into your garden. If they can do it you with glass, what's the difference?