The question is very specific, and I would suggest that you seek professional tax advice on the matter, from someone familiar with the topic. It's more complicated than a standard gifting of a site because of the relief you used. There are some specific provisions to navigate around claw back provisions, specifically when it comes to 'development land'. I don't know if there are any exemptions for gifting a site to a family member, usually any such relief would only be available to a child, but it's a very specific area. The claw back generally talks about reinvestment of the proceeds you receive, but with a gift there are no proceeds.
If one were to get over technical, it could be argued that the site with the unfinished house is not agricultural land and might not have qualified for relief in the first place. How it was all valued initially would have a bearing on the implications now.