I have to say, it would put me off, and I have moved because of the number of rented houses in my Estate (ended up with about 4 owner occupiers out of approx. 20 houses) Also, if I were buying again in the future, I would not buy a house where next door looked scruffy (rented or not)
Do you feel that the house is rented, or could it just be that it is someone has no interest/or is not able to look after the garden (perhaps elderly) People years ago had a habit of building "hacienda" type horrible extensions and perhaps this is just a throwback to this ?
Have you asked the Estate Agent about who lives next door. You could even be very direct and say, next door looks a bit unkempt, what is the situation?