Last time I cancelled a card I wrote to the bank concerned, cut up my card and enclosed it with the letter. They acknowledged, issued a final statement (including the Govt Stamp Duty) which I paid in full. That was the end of the matter and no reason why it should affect your credit rating. You may want to ask them for a letter confirming you paid the Stamp Duty so that you can avoid paying it a second time if you move to a new card provider.