Breaffy House is a little outside the town centre, but I would second it as a suggestion. Recently had a meal there as part of a large group (50 or so people) and every single meal was excellent (having worked in hotels in the past that's very rare for a large party).
A few nice restaurants but not sure how suitable they'd be for a Sunday lunch, An Carraig (on Chapel St.) is superb but might not be the right spot for a lunch (more of a Friday night dinner spot really).
A lot of the pubs in the town do pub grub (some of them even manage to do it well), Bosh (on Linenhall St.) seems one of the better ones at the moment (excellent variety on the menu for a pub and the meal I had was lovely - though I have only ever eaten there once so can't swear by the quality).