Just came across this thread.
Did the household insurance policy on the buildings have impact damage as one of the specific insured perils ?
If so, did the policy wording define the peril as including impact by road vehicles ?
If so, I cannot see any basis for refusing indemnity under the household policy.
The motor insurers might have declined to pay for the building damage as the OP cannot sue himself for his negligence.
However, the other interested parties to the ownership of the house would have a case against OP for their losses arising from damage to the building.
As is always the case in these matters insurance underwriters who refuse indemnity under a policy - a serious matter - should be made to explain their decision clearly and adequately in writing. The intellectual stress of this exercise sometimes causes a change of mind because they actually have to think about it properly as distinct from a cheap and casual wave off of the client :mad: