Tortured data - buyer beware

G

Gary

Registered User
Messages
13
For investment marketers, communications is key. But there’s a lengthy continuum from honesty, through “towing the party line,” to gilding the lily, and ultimately deceit.

Over the years, I have witnessed marketing/sales people from the various product providers operate on every point along this continuum. Some are very plausible. I should probably declare at this point that I am a former, or should I say reformed, investment marketer myself. I am certain that I towed the party line. But I’d like to think I exited with my integrity intact.

I have recently been through the glossy brochure for the latest guaranteed fund from Dolmen (safe harbour) and a chart in it stoked my ire.

I will admit to being cynical about the value in structured/ guaranteed products generally. But rarely to the point of opening a thread like this to vent my spleen.

The brochure refers to a BNP strategy and provides back testing as far back as 1995. It then goes on to say that the strategy being back tested was created in 2011 and the back testing data “includes estimates made by BNP Paribas in respect of unavailable data”.

Dolmen is not alone in the use of back testing of simulated strategies, so I am not pointing the finger solely at Dolmen, but there needs to be some debate around the use of such charts.

The financial markets have only one history. And when one knows what that history is, it is a trifling matter to find factors that ‘explain’ it (a statistical activity affectionately termed ‘torturing’ the data). After all, it is said that statistics is the science of producing unreliable facts from reliable data. Investors should view history with their eyes wide open.

In a famous tongue-in-cheek 'experiment' on data mining it was found that 75% of the variation in the S&P 500 could be explained by butter production in Bangladesh! By adding cheese production and sheep population in the US, correlation rose to 99%!

The link between butter production in Bangladesh and the S&P500 is obviously spurious, but if cloaked in terms of a complex strategy involving GDP forecasting, interest rates, volatility etc, it begins to appear plausible. And so when presented with a chart that ‘proves’ the strategy works, our greed takes over.

This product may turn out to be very successful (in terms of performance not in terms of sales – this dichotomy in how the provider and the investor would view success is important, but alas another rant all together). The point of this post however, is to highlight that if it does ‘work’, it will be as a result of sheer luck.

The irony about the famous experiment I refer to above, is that following its publication, the author received enquiries from people wanting to know where they could access data on butter production in Bangladesh. Can’t recall who said it, but it’s very apt; Regulation will not stop a fool from losing his money, but it should at least stop the average person from being made to look like a fool.
 
It then goes on to say that the strategy being back tested was created in 2011 and the back testing data “includes estimates made by BNP Paribas in respect of unavailable data”.
Click to expand...

That is the most extraordinary thing.

I had heard of "data mining" but "data torturing" is a new one for me Gary.
 
Interesting and disappointing observations. I guess when the incentive system is wrong to start with the customer has no real chance. We rely on the regulator to limit such instances but confidence that they are at hand to do is rightly non existent.
 
Dolmen product & ilk

More and more these products have at their core makey-uppy indices devised by investment bankers and mathematicians who have beaten up mountains of data.

It will be interesting to see what proportion of the current and recent 'vintages' of these products deliver a cent more than the guaranteed amount. My own guess is less than 25% will deliver bonuses and because of the various caps and volatility control mechanisms, the payouts will be small.

The idea that these are selling in large quantities is awful.
 
How to test statistical significance

The t-test was introduced in 1908 by William Sealy Gosset while working for the Guinness brewery in Dublin to evaluate the quality of the brewery’s ingredients.

In investing, the t-test can be used to determine whether any market beating returns are due to luck or skill. When we apply a t-test to the historical performance we get a statistical feel for how significant the results are.

A result from a t-test in excess of 2 is widely accepted to be a measure of statistical significance.

When we apply a t-test to the results of any particular fund or strategy we should always start with a "null hypothesis" that is to say that we don't believe a fund manager has any skill and we should set out to statistically attempt to disprove this proposition.

Here is the difficulty with this process. At any given time we have a limited data set perhaps the existing track record of any given fund manager or a backtested strategy, we also have the average excess performance achieved and we can measure the volatility of that excess performance compared to a risk appropriate benchmark. But the limiting factor here is the size of the data set such as the career of a fund manager. Typically we don’t get that many observations.

In another post, I recently pointed out that if I flick a coin 5 times and it comes up heads 5 times, there is a 3% chance that will happen just by luck. So, by the same token, 5 years of performance for a fund manager isn’t sufficient time for us to say with a high enough degree of confidence that here is the “Tiger Woods of fund management”.

We need at least 30 or 40 years of data to show statistically that, yes this fund manager or strategy has real skill above the market, allowing for the risks taken, and the persistency of the results is consistent enough to believe that they are really skilled.
 
I agree the wording is incredulous on the package etc !

However i find some of the displayed comments here even more incredulous.

"So, by the same token, 5 years of performance for a fund manager isn’t sufficient time for us to say with a high enough degree of confidence that here is the “Tiger Woods of fund management”.

" We need at least 30 or 40 years of data to show statistically that, yes this fund manager or strategy has real skill above the market, allowing for the risks taken, and the persistency of the results is consistent enough to believe that they are really skilled. "

This all depends on the time series used to perform the testing (if thats what i read into what you are saying (as most managers will be retired in 20-30 years)). Time series can be based on tick so you may have 1.5 millions ticks per second (so 1.5 million per second well you get the idea of how much data there is / could be to model the strategy).

To deem whether or not these methods are / can be succesfull through data mining i urge you to readup on 'Bill Dunn' | 'James Simons' . (If they are successful over a period they will dump there investors money and just run there own!)

Indeed if the dataset is purely backtested and then conditioned to be 'curve fitted' this is of no-use. You want too see what is used in relation to in-sample and out of sample data to proclaim whether a strategy is valid or not.

I guess this prospectus has been written badly / checked reviewed very badly.

If i was to tell you that if you bought wheat on 1st august and sold it at end of august every year since 1968 that prices ended up higher 83.33% of the time would you believe me ? Is this a tradeable concept ? In itself no its not as i havent stated the drawdown between 1st - end of august but it is a basis for further testing.

Does the market behave similiar over long historical periods (most definetly yes it does). Can i prove to you that market trade entries at best can be outperformed by random entries ? (yes i can as the exits are only thing that matters).

Can i prove to you that 93% of the time equity prices revert to previous prices ? Yes i can. Is it a tradeable concept in itself no with other things added into the mix yes.

Anyone read the 'hedge fund mirage' book recently released ?
In it you find compelling evidence that the average hedge fund return was less than the t-bond return over past 30 years or so (when fee's are taken into account).

Does that mean that hedge funds are bad not really in that some
'superstar's' produce genuine alpha. Does it mean that the average joe bloggs should just buy t-bills; yes it probably does.

A hedge fund manager once mentioned something smart to me; he said if i wanted to run a fund that looked great id just buy argentinian bonds and 1 every 35 years id blow the fund up (as they may default). For the other 34 years id look like a superstar with between 8-15% per annum return. Argentina defaulted on its debts 5 times in the last 175 years.

Any random walkers out there ?
 
So we seem to be mainly in agreement.

I am perfectly willing to accept that there will be some superstar hedge fund returns (no more than I would expect by pure chance before fee and considerably less after fees) but that I have no reliable way of deciding who the winners will be in advance and that therefore most people on average are better off with T bills. I think it was bill Sharpe who first made this observation.

So in conclusion since I don't have enough manager decisions in any reasonable period of manager tenure I can't determine with a high degree of statistical certainty if a manger is skilled or just lucky. I am therefore better off (on average) with a simple buy and hold strategy.
 
Buy and hold strategy for t-bills yes an average investor seems much better off with this (historically speaking). Then again who wants average or a C- grade on there paper. Average probably means just below inflation returns.

Buy and hold in terms of equity markets no way! we agree in some fascets but not in other fascets (think i spelt that wrong!).

If the fund has done correct back testing forward testing and in sample and out of sample then it probably has 'some' merit.

There is no 'superstar hedge fund' there by pure chance over a prolonged period of time. If they are betting every day on asset prices to move up or down they wont be in business long based on 'luck' if there making 10 trades a day or so.

There is little 'luck' associated with good professional trader's.

Take for example that markets are probably now run by 60% quantitive techniques this means in a way that mutual funds are at a disadvantage and indeed retail. Mutual funds not been able to short (apart from weighting a basket to mimic a short) (however why wouldnt i just buy an ETF in this case). Retail due to the actual quant machines and indeed due to them machines always watching the markets for a price level threshold that triggers a trade.

Check this out: http://www.trendfollowing.com/perf.html as an example of dunn capital. Is he just lucky ? I doubt it id say he has a statistical edge. How did he gain this edge from 'backtesting' his strategy.

On an aside note: personally i have created 5 strategies myself from an algorithmic perspective.

Of these 5 strategies 2 i dumped as they were crap and didnt work and 3 i run live. The data i analyse is probably around 4165516 minutes of actual market data in sample and 1/3 of that out of sample for each strategy.

Why did the 2 fail ? 1.) Illiquid market. 2.) Crap strategy. What did i learn 1.) test in demo live mode for 6 months after production too see which did and did not work. Do i need too see why 2.) was a crap strategy yes i do.

So do i believe a manager would do better if they backtested 'most definetly yes'!

Do i think backtest's can provide some evidence of future performance, to a degree i do (but it has to be well thought out etc etc and a lot of work done on it ). Do i believe a lot of backtests are invalid (yes i do) as id really like too see the out of sample data as much as the in sample tested data.

Do i think fundamentalists may be able to add value ? Maybe in some cases if the company is not cooking the books so too speak.

In sum if you want c- go for bills it as not everyone would go to the effort i have done to manage there own money . Its more of an interesting passionate hobby to me so dont go down this route!

You can never know the winners in advance but to be honest in investment terms 1 in every three (investments) may be a winner 1 in three breaks even and the other 1 loses money that could be enough depending on how big winners - losers are!

Losers and winners are part of life very successful corporations have plenty of losers added to there own balance sheet through Mergers but they have a few winners also which is what makes them winners!
 
A good stethescope when looking at a fund manager is what the sharpe / sortino and calmer ratios are.

Would anyone get funded based on not 2 consecutive years of live trading history, no they would'nt (even though in this case this fund has not been live for long).

If i was looking for a fund manager id actually look at fund managers based on poker / bridge tournament performances.

An interesting correlation maybe as to what table they get to at a major tournament in line with there fund performance. I wonder would there be a correlation ? Id never know unless i backtested it!
 
Some interesting misconceptions here.

Firstly the amount by which a winner wins is equal to the amount by which a loser loses since there are no orphan securities everything is held by someone.

Therefore in order for you to prosper from any trade you have to find someone willing to take the other side. For you to win they have to lose.

When you trade you pay costs and taxes. The more you trade the more you pay in costs compared to the poor average buy and hold investor. How do you know when you trade you are not going to be the loser on the wrong side of the trade? You don't and on average market participants are just as smart as each other so you need to find people less smart than you to exploit in order to consistently prosper. This in a market full of research depts, investment banks and of course private investors with their own systems.

It isn't credible that winning trade are down to anything other than taking on more risk or simply luck.
 
As for backtesting.

Fund managers use a process known as incubation. They design several strategies and run them privately for a period with their own money. They dump the strategies that don't work (sound familiar?) and launch the strategies that seem to make money. Naturally they are hardly going to launch a strategy which doesnt appear to work. The problem is that as the funds attract investors the persistency of performance tends to drop off and real investors tend to be left holding a dog.

Rinse and repeat until broke
 
If you did some research you would see that there are investors who make outsized returns. In fact the decent ones make / take all the money.

I think you dont understand what i meant when i termed winner and loser. If i bet 1K USD at what i believe is 50/50 to win 2K then my winner is twice my loser's size. So effectivly i can do this every day for the next 220 days (say the trade lasts 1 day) and on average ill have 110 winners and 110 losers.

110 losers = 110000
110 winners = 220000
220000 - 110000 = 110000 profit.

In another example just because i shorted something and someone else went long something doesnt mean theres 1 loser and 1 winner . The fact is we both may have different timeline horizons i may be in and out in a second and he may be in and out a few minutes later. At the end of the day price in a commodity term is essentially just a number that dictates in some facets supply and demand.

"When you trade you pay costs and taxes. The more you trade the more you pay in costs compared to the poor average buy and hold investor. " -

If i know that the typical standard deviation movement of a stock is X and it doubles the standard deviation movement in a downward wouldnt i be best selling out of that stock probably. Would the buy and hold be better off holding it, no they wouldnt as if they can buy it back cheaper than before its better than if they hold and hope.

Buy and hold equities havent a hope in todays markets as the bot's will pick them off. Anyone who thinks different is sadly mistaken the bots run the market and make most of the money.

In terms of "When you trade you pay costs and taxes. The more you trade the more you pay in costs compared to the poor average buy and hold investor. " - Well in fact this could be deemed wrong as big funds get paid too add liquidity to a market so if i was a huge fund id be getting rebates for adding liquidity so in fact i get paid to be in a market and defining it.

Theres different ways too look at things, certainly the product mentioned at start of this thread i wouldnt endorse.

If you look up historical figures off the top of my head there are 12 days of outstanding movement (major stdev movement) in the stock market every year if you miss them / are on the wrong side of them you basically miss out on the real fruits of been in the stock market. How can buy and hold pick those days by just buying on a particular day ? Impossible. How can they short it not possible either.

Still having said that if i was to be putting money towards retirement id be just holding high grade government bonds. (My reasons differ to others though as i have enough volatility in my portfolio to pick up outsized returns with my strategies ) or well at least i hope i do!

For others a see saw balancing portfolio may work a lot better.
 
they wont launch unless they have two years of live results (you can pick through these results by looking at what they have etc.)

much better a backtest than a buy and hope!
 
strategies sometimes just dont work thats why it was eddison who said he found a million ways not to make a lightbulb or something like that!

Look try this link for a list of funds that probably have outsized returns versus buy and hold maybe they are the poster children or maybe we should learn to follow them somewhat!

http://www.automated-trading-system....rds-july-2012/

http://abrahamtrading.com/performance

Id guess by looking at there career opportunites section in any of them you will notice they are looking for quants so its obvious they are willing to put money on non buy and hold strategies!
 
In fairness my 2 failing strategies were due to my own naievty there isnt enough volume on oats in 1 case and just not enough testing in the second scenario! We will see after two years live on the other 3 strategies. 14 months only at present.

If your interested on a book that shows some strategies etc and is probably worth the investment have a read of this:

http://www.amazon.com/Applied-Quantitative-Methods-Trading-Investment/dp/0470848855/ref=sr_1_1?ie=UTF8&qid=1346769521&sr=8-1&keywords=applied+quantitative+methods+for+trading+and+investment
 
Gary where have you been all my life? That is one of the best posts I have ever seen. I agree 100%. Let me add a few comments.

We are presented here with what looks like a very plausible and sophisticated strategy to beat the markets. It is like a system for winning at roulette. No such system exists and no such investment strategy exists. Of course it backtests well. That was the reason for its choice. No basis whatever in financial theory.

So now lets talk about backtesting. Backtesting is banned by the Consumer Protection Code for Tracker Bonds. This is a Tracker Bond. However, strictly speaking the backtesting is being applied to the underlying and not the Tracker Bond itself so technically not in breach. The CPC should ban all backtesting.

Now to my hobby horse. The CPC is a toothless sham. One of its key requirements on Tracker Bonds is to show clear disclosure of where the investor's money is spent and in particular to show the charges. The disclosure on this product says charges are zero. This is typical of Ulster Bank (it is they who approve the brochure). I have complained in the past to the CBI on this. No response.

They do make a reference to the commission costs but this is probably only half of the total costs on this product. The CPC will remain a toothless fig leaf until practitioners like me can make meaningful complaints about non compliance.
 
I find that interesting the CPC ban backtesting nice information! Thats interesting...

" We are presented here with what looks like a very plausible and sophisticated strategy to beat the markets. It is like a system for winning at roulette. No such system exists and no such investment strategy exists. Of course it backtests well. That was the reason for its choice. No basis whatever in financial theory."

However a roulette system would technically be feasible to work if you could martingale til infinity so as your point is sound on the surface its not sound when its mathematically evaluated!

Like a system for blackjack and poker also (obviously roulette systems do not work)? - There are systems for both blackjack and poker its only recently that poker has been recognised in america as a game of skill. Is it any wonder that over the last 30 years or so the same faces frequent the final table in poker tournaments.

Would this be skill or luck ?

By the way do you have a link to the underlying document at start of this thread.

Lets not forget backtesting is hypothetical its not actual!
 
[broken link removed] is the very brochure. Ecstatic I will wait till Poker is an official Olympic sport before agreeing your thesis:)
 
Well i looked at the brochure and believe some of it has merit. It has plenty of exposure for 1 thing!

However some major issue's in this that i see is that either they are using the VIX Volatility Index or the deviation of prices to control whether its risk on or risk off.

As we all know and appreciate the last 5 years have been very volatile within the stock markets however between 1999 - 2001 is less volatile. How was the volatility measured in fact as what was the way to measure volatility in 2000 is not the way now to measure volatility.

It doesnt show the maximum drawdown or the sharpe ratio so its very difficult to measure the performance against anything buy and hold included. Without knowing what risk is been taken you cannot decide whether alpha could be present.

From the prospectus those fees are unreal !
"the fee to Dolmen will be equivalent
to 4.18% of the Investment Amount in the case of Option A and 5.46% of the Investment Amount in the case of Option B."

Also it states the usage off bollinger bands to see the price etc which to be frank is a slightly dated notion to modern quantitive techniques.

Also without knowing the timeframe etc this product doesnt mention anything regarding slippage etc.

If the fund size affects the price etc for instance it has silver as a market to be in but thats not exactly a highly liquid market.

Also it doesnt state which treasury bills to invest in.

Also it doesnt state rollover for futures contracts.

Also it doesnt state any fx risk or is this been hedged out ?

More questions than answers but the underlying principle seems soundish, (once you expect germany not too default, us not too default and all those currencies to be still around in a few years well).

Based on your comment i am also waiting for hurling to be an Olympic sport also!
 
You must log in or register to reply here.
Back
Top