To a large extent, no, you're not talking rubish but it DOES happen.
Petrol is not clean and there are gonna be contaminents in there from the tanker ship and then the truck and the underground tank at the station and finally the tank in your car. If the tank at the station is nearly empty then crud/sludge 'could' be sucked up and cause the fuel filter to become partially blocked. This will happen more in older cars as this crud/sludge accumulates over the years.
My car can run a little poorly if I'm 'running on empty' but I've never had the car conk out as a result of this.
Sensor failure and lack of oil changes, coolant changes, plug repacement and common or garden carbon build up in the throttle bodies are the most common causes of poor running/starting IMO.