You really need to get independent professional advice on this one.
The most affordable way to do the thing, if the company has the cash to support it, would be by way of a share buyback. This means the company buys the shares of whichever shareholder wants out, using its own money. These shares are then cancelled, leaving the other shareholder as the only remaining shareholder.
There are a number of requirements to be satisfied for this to be treated as a CGT transaction rather than a distribution liable to income tax - such as period of ownership of the shares (must exceed 5 years), and the "trade benefit test". A key element of the trade benefit test would be that the buyback must not adversely affect the working capital of the company, therefore my underlined statement about it only being an option if the company has sufficient cash available.
If the company borrows money to fund a buyback, this is viewed as a distribution for tax purposes and will result in an income tax liability (at up to 52% rather than CGT at 33%).
Similarly if the company loans you the money for you to buy the shares this will give rise to other messy income tax issues (this also includes you "borrowing money in the company's name"). So if borrowed money is needed to buy the shares then it will have to be borrowed personally as you were originally told.
A hybrid option might also suit you, whereby the company mightn't be able to afford the buyback in its entirety. So you would buy, say, 20% of your partner's 50% stake, and then the company buys back the other 30%.
The only way to avoid tax on the transaction would be if the departing director is over 55, and satisfies certain conditions as to period of ownership of shares and time spent working as a fulltime working director of the company, in order to claim retirement relief from CGT.
This is about as simple as this stuff gets, so if any of the above is even remotely unclear to you then you know you I'm right when I tell you to get proper professional advice - in these situations it ALWAYS saves you money in the long run if you pay for decent advice.