The initial advice is to get up to date copies of the folios involved from the Land Registry to see what the current registered position is.
If these moved boundaries have been in place for 20 odd years, it may be well that you have lost your rights to the lands affected. Your brother may have acquired title by adverse possession, more commonly known as squatters rights.
I know it's not nice to hear, but if that is what happened, unfortunately it is your own fault as you acquiesced to the new boundaries for a long time.
If the above is not the case, you mention that a surveyor marked the sites out. Is there an agreed map? If there is, you can ask him to move the boundaries back to the agreed lines. If there's no map, you might have some difficulty proving your right to the land in question.
If you are thinking of going to court, really you need to see a solicitor to get an idea of how long it is likely to take (probably a couple of years at least) and how much it will cost. Make sure you get advice on the specific to the topic of adverse possession.