If you are wary of heavy borrowings, consider hiving off the apartments separately if that is physically possible -- apartment rentals in a small town can be more of a pain in the ass than anything else.
Also look at splitting the title of the three commercial units if not already done, so that all of them are not at risk in a worst case scenario. You can then refinance in a few years and load all the mortgage on one unit so as to cover your risk better., subject to best advice from your accountant and lawyer.
Agree that you need to keep ownership of the units either for yourself or a family member and rent them to tenants, including your own business.