Had similar situation myself - our site is 5ft up from the footpath. Dave28 is correct, you're not supposed to take as much as a carrier bag full of soil from your plot with out informing the Council!!
If I'm honest, we did it anyway - luckily man next door works for the waterboard and they were looking for a good few trailer fulls - paid a guy from the local quarry to take 2 loads to the tip (€70 disposal charge, €30 for the man with the truck), then got the rest taken to a man with a very large hole to fill about 10km away - digger man, tractor man and trailer for about 10 hours was €750. Levelled the site off lovely.
I think the main concern of the council is that this soil isn't just dumped on the side of the road. Depends how brave you're feeling - I'd do it again if I had to.