I can't speak for Addison Lodge being an issue with any degree of certainty. I have only been inside it a few times, but the clientele there seemed fairly sedate when I was in it.
I definitely don't think that being so close to the Botanic Gardens will be a problem. It has a large car park of its own, that is off a side street just past the main gates. It is well sign posted, so I can't see people parking on the street thinking that that is the only parking option. I have been in the Gardens in the height of summer when crowds are at their peak, and have never had a problem finding a spot in the car park.