Well think if you where to sell in the future or him this could cause problems, The new buyer maybe could be in a position to ask them to take it down due to part of it been on their property, You could maybe sell the 4 inches for a small amount, with him having to look after the legal costs, this way in the future him or you selling up would not have this as an issue , even if his drains are on your land and not the building itself can cause problems, If it was me I would explain what could happen, also if u had a falling out over anything then you know what could happen if you do not go the legal way