I am assuming he might think the seeds came from the 17 ft elm tree
You need to understand this has no relevance whatsoever. While it's likely the seed came from your tree, it's not certain, and regardless, you have no liability under law for seeds that originate from plants on your property.
To be clear he could employ a worker to clean the gutter and remove the tree standing on the gable roof
The worker would need use safety gear like ropes strap to avoid falling off a sloping roof that 25 ft high
No. As Branz posted above, they can ask for access to carry out this maintenance via your property, that is the only reasonable option.