You need to have a serious chat with this guys boss,this is just not on,you are paying him a serious commission to sell your house,tell him if he wants to continue with this sort of lazy carry on that you will be dealing with another agency.
E/As are for the most part quite useless and given half a chance lazy,they will have you showing potential buyers around if they sense you are a push over,I know from a long number of years of dealing with this breed,who I consider one step above "carny folk"