I would but not based on a particularly well thought out financial decision. I'd just like the idea of being debt free and not having that worry. It also gives you more options generally.
From a financial planning side of things all your investment eggs are in one basket. Using some of the cash you have freed up to plough money into your or your husbands pension should constitute less risk but your ages and the ages of your children will have a considerable bearing on what you should do next.
I'd be concentrating on the pensions as an investment strategy. I'd also use some money for the most valuable thing you will ever buy with it; great memories with your children. Go on some really nice holidays. They'll remember them forever.