1. OP signed a contract to buy a property at a certain price. Thats a contract - a legally binding agreement.
2. How he finances that purchase is his problem. If he had cash he would pay cash. He does'nt so he is borrowing.
3. He cannot now borrow from the bank the amount he wants. That is his problem. Its not really the builders problem - except its going to make the whole thing more difficult all round.
4. Builder has a signed contract - he does not have to move away from that BUT he needs to be realistic also. He cannot want to drop it below a certain level - that would be madness when he has signed contracts. However, he equally may not be able to successfully recover the whole sum due from a reneging purchaser if he sues on the contract. So he is being pragmatic.
5. "Subject to loan approval" - this has not been acceptable to builders for about 10 years. It may start to come back into favour - but it is unlikely to be the case here.
By all means haggle - but OP needs to be aware of the real, true situation he is in. He is in trouble and he needs to find a workable solution which does not involve being sued.
mf