The broker I used when I got my mortgage did say as a heads up that the bank get a clawback from them if the customer move within 3 years (presumably to hope that the customers stays with the bank for 3 years).. but I don't think they can force you to repay back.. does the T&C say anything specific about if you switch or end the mortgage within XYZ time frame, you would be liable to pay their fees? (or similar?).
I agree with Dr.Strangelove, if it is just an innocent query on why you switch, I'd say you are safe enough to go silent.