It seems to be that as a couple you have never addressed how to deal with money, yet you have very opposing views, you a prudent saver, him a spender.
Other than NE you don't mention other debts, credit cards, car loans etc, so you must have saved as a family to pay for cars and holidays - how did that work? And tbh if you have managed to save a substantial sum whilst on benefits, it would indicate that he is paying most/all of the bills now.
It may be that you are carrying on as always - you the one saving for a rainy day, either out of your wages or benefits - but supported by him paying for other expenditure, but you are panicing now because of job loss, lack of security and NE.