I covered this story on my blog in 2011.
[broken link removed]
This is a classic argument and the uncertainty around the "right" answer has been summed up by awarding the Noble Prize in Economics to two apparently different views of the markets.
Gene Fama is the father of the Efficient Market Hypothesis which says that prices reflect all available information and therefore in a strict view of the theory, its basically impossible to beat the market because you are always paying a "fair" price.
However, nowhere in this theory does it say that the market price is the right price. It says that it is the
best estimate of the right price. All prices could potentially be wrong. In fact, when tomorrows trades are placed and prices move we know for sure that today's prices were in fact "wrong" or the price would not move tomorrow.
Fama argues that you can't identify a bubble ahead of time and that;
“Irrational bubbles in stock prices are indistinguishable from rational time-varying expected returns.”
Eugene F. Fama. “Efficient Capital Markets: II Journal of Finance December 1991
Bob Shiller says no, markets are prone to forming bubbles and that these are predictable.
When I'm lecturing this stuff, I describe markets as being like custard...bear with me.
Custard is a non-newtonian liquid. This means that it typically has many of the characteristics of a liquid -except when it doesn't.
Because it contains starch, if you apply pressure to custard it forms a solid and in the extreme you can walk on a swimming pool full of custard.
Seriously, watch this: https://www.youtube.com/watch?v=BN2D5y-AxIY
So, markets are like custard, most of the time they work pretty well, except when they don't, and just like custard, they tend to break down when they are under pressure.
What is the recommendation for investors in markets? They should operate
as if they are efficient and both Fama and Shiller (and indeed Warren Buffett) would recommend that the best way to invest is through an index fund.
However, just because investors should operate as if markets are efficient doesn't mean that they are and that after the fact we will all know for sure if we got carried away with ourselves.
The real problem for an investor is to be sure beforehand that they are in the middle of a bubble and to get out.
“Market behavior often diverges from what we would expect in a rational efficient market, but these anomalies do not create such large profit opportunities that active fund managers as a group earn abnormal returns.”
Richard H. Thaler, “The End of Behavioral Finance” Financial Analysts Journal November/December 1999
"If you have money to invest, the only sensible place to start is with the assumption that the market is smarter than you are.”
"If you're picking somebody else to manage your money, the chances of finding a market-beating path are even harder.”
Justin Fox. The Myth of the rational Market Harper Collins, New York
“Markets are not so stupid. They’re actually quite efficient.”
“Most things are reasonably priced. It’s very difficult to outperform benchmarks.”
“Mispricings and Unrealistic Expectations” Dow Jones Asset Management March / April 1999