Investors badly burnt by misleading back-tests Business Post

[Duke of Marmalade]

 

[Duke of Marmalade]

Below is probably the most telling extract. The clearest indication yet from the Central Bank that multiple overlapping back-tests conducted against a background of strong markets could be grossly misleading. They claim to have addressed the gross abuse since Fagan/Woods complained about it 5 years ago - but that wasn't until last year. I fancy the FSPO are going to be busy.

 

[Brendan Burgess]

Was this one that you reported to the Central Bank?

 

[Colm Fagan]

Hi Brendan
Yes, Brian and I complained to the Central Bank, before the launch date. We told them that the index, which BNP Paribas had devised and which had outperformed the EuroStoxx50 by something like 4% a year on average for the previous 18 years (15 of them before the index went live) was going to underperform in future. It did - spectacularly so. The CBI didn’t act on our request that they force Broker Solutions to change the marketing literature or write to applicants before the launch date.

 

[Duke of Marmalade]

Was this one that you reported to the Central Bank?

Fagan/Woods submitted a complaint 5 years ago (see attached). I'm just a humble duke informing your parish. Fagan also predicted 5 years ago right here that the "Deep Value" index would undershoot the EuroStoxx 50 by at least 20% over the next 5 years despite the brochure showing it had beaten the pants of its benchmark over the previous 18 years, never mind the outrageous back-tests which the Business Post focussed on.
(Ah I see Fagan has beaten me to it)

 

[charlie007]

For investors to be guaranteed a minimum 85% return, does not close to this amount need to be put on deposit?
Remember in 2019 some deposit rates were in negative territory.
With just 15% to be 'invested' and charges to be applied, why the big surprise at a negative return?

 

[Duke of Marmalade]

For investors to be guaranteed a minimum 85% return, does not close to this amount need to be put on deposit?
Remember in 2019 some deposit rates were in negative territory.
With just 15% to be 'invested' and charges to be applied, why the big surprise at a negative return?

Absolutely! In the above complaint that Fagan/Woods submitted to the CBI, this is how they put it:

Returning to the situation at hand, this all begs the question as to what are the prospects for The Bond, based on open market quantitative analysis? We are hampered by the fact that there is no open market on the Select 50 Index as it is a proprietary BNP Paribas creation. However, Appendix A of the CPC provides a very helpful pointer to what might be involved. That was a good presentation of the financial anatomy of structured products but since it is not in the Brochure we presume it no longer applies.

Nonetheless let us try and construct that table. In this situation it is (B) The cost of the bonus (or the derivative providing the bonus) which we will get as the balancing item.

• Open market cost of the guarantee – 83% (85% discounted at a generous 0.5% p.a.)
• Open market cost of the bonus – 9.6% (equals 100% - A - C)
• Charges – 7.4% (page 18 of the Brochure)

And what is the bonus over the 85% guarantee? It is 55% (plus any excess) payable if the Select 50 Index is above its initial level in 5 years’ time.

In other words, crudely but not too far off if we presume that BNP Paribas have priced this rationally, this is being assessed as having a probability of about 1 in 6, the throw of a dice! Compare that to the 1,304 in 1,304 that is shown in the backtesting!

And yes you are right and I think many folk whilst not seeing it so clearly as you do just reckon it is too good to be true. But clearly not everybody has that protective common sense and if presented with the brochure by a regulated financial advisor they can be forgiven for jumping at what looks a sure thing.

 

[Colm Fagan]

Hi @charlie007. Of course we weren’t surprised but I’m sure that everyone who bought the product on the basis of 1,304 successes out of 1,304 back-tests was surprised. They were also surprised I’m sure that, while its benchmark, the EuroStoxx50, had gone up by something like 38% in the period, the index to which the product was linked fell over the same period. This was the same index as had outperformed the EuroStoxx50 spectacularly over the previous 18 years. That’s why we complained to the CBI about it back in 2019. We knew something like this was the more likely outcome - but the punters weren’t told.
Sorry! I only saw the Duke’s post after I had posted.

 

[joe sod]

Its truly terrible performance when you see how the markets have had outstanding performance since 2019 apart from the covid sell off, and value stocks have performed the best of all since the arrival of inflation. Its obvious that this fund like alot of these so called "low risk" funds had way too much money invested in government bonds the worst investments since covid and the ending of the negative interest rate era ended the government bond bonanza

 

[Duke of Marmalade]

Its truly terrible performance when you see how the markets have had outstanding performance since 2019 apart from the covid sell off, and value stocks have performed the best of all since the arrival of inflation. Its obvious that this fund like alot of these so called "low risk" funds had way too much money invested in government bonds the worst investments since covid and the ending of the negative interest rate era ended the government bond bonanza

No it wasn't bonds to blame. Below is how Colm satirically described this weird index 5 years ago: (I leave it to Colm to give his more colourful explanation which has attracted plaudits on LinkedIn for its heuristic value.)

Too good to be true
Colm Fagan Diary of a Private Investor Update 17 10 September 2019

A conversation with my friend Brian Woods almost four weeks ago started it all.

“Colm, I found an investment you might like. It’s called Accelerator Bond 4. Google it and see what you make of it.”

“OK, Brian, I’ll have a look. While I’m looking, tell me more.”

“It’s a five-year lump sum investment. There are two options; I’ll focus on the second. The return after five years is linked to a stock market index. If the index is at or above its starting level, the investment return is at least 40%. If the index is below its starting level, you lose 15% at most, less if the index has fallen by less than 15%.”

“So, I’m guaranteed at least 85% of my initial investment after five years, come what may, and I’ll get at least 140% if the index is above its current level. Is that what you’re saying?”

“Yes, assuming of course that the bank backing the product hasn’t gone bust.”

“And who are they?”

“BNP Paribas. They’re one of the world’s top investment banks.”

“Sounds great, but why do I get the feeling that you’re not convinced?”

“The fact that it looks so good is precisely the problem, Colm. It’s seems too good to be true, and you know what they say.”

“Indeed. If there is a catch, it must be in the index. The brochure says the bond is linked to an index called the Solactive European Deep Value Select 50 Index. I never heard of it”.

“It’s exclusively for BNP Paribas.”

“Brian, reading the brochure, this Solactive index is very peculiar. The most peculiar aspect is that up to 25 of the 50 stocks selected for inclusion each month are chosen specially because they’re due to go ex-dividend within the month. As you know, the price falls when a share goes ex-dividend, because the seller, not the buyer, is entitled to the dividend.”

“Colm, are you saying that stocks are chosen specially to depress the index?”

“Yes, that’s what I think. I’ll have to do some homework to estimate the extent of the drag. Talk to you tomorrow.”

***********​

“Brian, I’ve done the sums. I estimate that including in the index an above-average number of shares that are due to go ex-dividend cuts the return by around 2.3% a year. On top of that, the dividend yield on the Solactive index is at least 1.2% more than on the EURO STOXX 50, the main benchmark index for Eurozone stocks. So, if the shares in the two indices deliver identical total returns in future, the Solactive Index will lag the EURO STOXX (price only) index by at least 3.5% a year.”

“Are you sure, Colm? That’s a drag of almost 20% over five years. Putting it another way, are you saying that, if the EURO STOXX 50 Index increases by 20% over the next five years, the Solactive Index could still show a loss?”

“Yes, that’s what I’m saying.”

“Hold on now. If you’re right, how do you explain the chart in the brochure showing the Solactive Index outperforming the EURO STOXX 50 over the last 14 years?”

“Brian, it’s apples and oranges. The two indices are completely different, in terms of industry sectors, geographies, and even currencies. There is no logical reason for comparing them. The EURO STOXX 50 consists entirely of Eurozone stocks; the Solactive Index has a mishmash of currencies, including sterling, Swiss Franc and the three Scandinavian currencies. The UK and Switzerland have the highest weightings in the Solactive index. Neither is represented in the EURO STOXX. A cynic might claim that they compared the two indices simply because the comparison gave the “right” result (from their perspective), but I’m not a cynic.”

“Colm, I still find what you’re saying hard to believe, but it ties in with work I’ve been doing, based on the costs and charges on pages 18/19 of the brochure. The margins in the product indicate that there’s about a one in six chance of the investor getting a profit of 40% (or more) at the end of five years. That’s about the same as the chance of landing a six with one throw of a die. There’s a five in six chance that they’ll lose money. Those odds are reasonably consistent with your conclusion that the probability of making a profit is equivalent to the probability of the EURO STOXX Index (price only, excluding dividends) increasing by around 20% in the period.”

“For a mathematician like yourself, Brian, it’s nice to see the two approaches coming to similar conclusions. Not nice for investors, though.”

“Indeed. Our conclusion that there’s a small chance of investors making money also disagrees with the back-testing results on page 11 of the brochure. The Irish promoters (a company called MMPI Limited, trading as Broker Solutions) say that they back tested 1,304 5-year periods between 2 July 2009 and 1 July 2019 and that every single one of those 1,304 back-tests showed a profit. The worst return was +40% while the best was +81.27%.”

“That sounds very impressive, Brian. How did they get 1,304 simulated past returns?”

“Good question! They assumed that someone could have invested in a five-year product each working day between 2 July 2009 and 1 July 2014 and seen it mature. But there were only two independent five-year periods in that time – the first between July 2009 and July 2014 and the second between July 2014 to July 2019 - not 1,304.”

“Crazy. I’m surprised they didn’t aim even higher. They could have got more than 15,000 successful past simulations by assuming people invested every half-hour rather than just once a day. They undersold themselves!”

“That’s funny, Colm, but it’s no joke for the people who bought the product. We should do something”

“Like write about it in my investment diary and hope someone in authority will read it?”

www.colmfagan.ie

 

[Brendan Burgess]

No it wasn't bonds to blame. Below is how Colm satirically described this weird index 5 years ago: (I leave it to Colm to give his more colourful explanation which has attracted plaudits on LinkedIn for its heuristic value.)

Did they actually artificially pick the stocks which did well in retrospect, despite the huge drag, and create an index out of them?

So I could find the 20 best performing stocks over the last 10 years, take 5% a year off their performance, and it would still have been very good?

So I could look at the results of yesterday's race meeting. If I had backed the winner in each race, I would have had a return of 500%. Even if I knocked it back by 50%, it would still have been a great retrospective return. But the chances of the same system working in tomorrow's race meeting would be very small.

 

[joe sod]

I see it was constructed as a "deep value" european focussed index, in theory that should have way out performed as european value stocks were very undervalued in 2019. Therefore they were obviously investing in "value" stocks after they had out performed. Even the msci world etf has increased by 50% since 2019, but of course our stupid investment taxes are discouraging people from simply buying this etf because you have to give nearly half of that to government with no indexation for high inflation rates since 2019. The government themselves are partly to blame for this as they are indirectly channeling people into these inappropriate products rather than simple etfs

 

[Duke of Marmalade]

Did they actually artificially pick the stocks which did well in retrospect, despite the huge drag, and create an index out of them?

So I could find the 20 best performing stocks over the last 10 years, take 5% a year off their performance, and it would still have been very good?

So I could look at the results of yesterday's race meeting. If I had backed the winner in each race, I would have had a return of 500%. Even if I knocked it back by 50%, it would still have been a great retrospective return. But the chances of the same system working in tomorrow's race meeting would be very small.

I think something like that is what happened. They have the defence "what better way than to pick a system which did well in the past despite its obvious handicap?" The way they put it is the options on this strategy are mispriced based on past performance. Impossible to expose. But ask yourself if they had discovered a massive arbitrage why share it with jo blogs?