Ideally these are the steps that should be followed for a lasting finish
1) run a whacker plate across the freshly dug area to compact the disturbed soil, levelling as you go
2) Put down weedkiller and anti-weed membrane (lets water down and stops weeds soil and grass coming up)
3) Spread an inch or so of garden aggregate and run whacker-plate over this, leveling as you go
4) Repeat step 3 with balance of aggregate.
As it stands you may continue to get holes for a while especially in wet weather (is there any other kind?) and your only option is to keep back filling them.