It sounds like you are renting the place by room to individual tenants. In other words, this is not just 1 lease with multiple tenants/sublets.
If it is on a room basis then you could have John in Room A on a 1-year lease, Jane in Room B on a 6-month lease, and then Room C and Room D have new people every few weeks but no-one ever signs a lease for fewer than 14 days. Is that the gist?
If it is on a room-by-room basis then you should not need everyone's permission to enter the common areas, but you cannot enter private bedrooms.
This reads as though you are doing the short term lets in order to annoy or force out the long-term tenants.
It sounds like the RTB recognises that this arrangement is probably a nightmare for the long-term tenants. The fact that you only ever seek permission from the short-stay tenants does not look good.
You seem like you are focused on following the bare letter of the law and ignoring the spirit. If the RTB is within its rights to make a judgement on a tenant's enjoyment and peace in the property then I can see why you got the result you did.