You will probably find that it will be harder to move around between lenders. Obviously the current economic conditions have placed the "rate tarts" as they are apparently referred to as at a disadvantage.
If you move mortgage now to a new lender , won't they be suspicious about your motives, (rates aside).? Will the equity on your home be seriously devalued since the good old days? You'll be forced to revalue your home to avail of any new mortgage elsewhere, will will naturally right down values across the board.