Spot rates could vary wildly over the course of a day, but the rate he quoted you should be the rate you got. If the rate did change, you would have needed to be changing a lot of money to lose €80. Have you worked out the rate you got vs. the rate you were originally told? Could you post them here?
At this stage it's probably too late to do much about it given that you walked away and didn't ask for a receipt. It certainly sounds shady to me. No harm in going back and asking to see a manager and if you get no satisfaction that way, make a complaint to IFSRA. But this is a little after the fact.