Yes, that's correct.
You can only terminate during the fixed term in the event of a breach of the lease (e.g. non payment of rent) by the tenant.
The Part IV tenancy runs alongside the fixed term lease and therefore your tenant can stay in the property for the balance of 6 years, unless you terminate for certain specified reasons, having delivered the appropriate notice.