Here is why both Fama and Shiller are correct!

I just finished listening to interviews with Eugene Fama and Robert Schiller by Sweden's SVT.  Eugene Fama believes in rational economic participants and efficient markets.  Accordingly,  individual investors can neither outperform the market nor predict asset bubbles.  Robert Shiller, who has also done extensive work on asset prices most notably the construction of the Case-Shiller home price index, believes in economic participants who sometimes act irrationally driven by group-think and emotions such as fear or exuberance.  Accordingly, markets can be inefficient which explains asset booms and busts.  I believe that these seemingly irreconcilable positions can be explained by two economic forces.  First, rational individuals will not necessarily produce rational outcomes when acting as a group and second, since money is denominated in itself, changes in money demand affect the prices of other assets.

Let's explain the first force.  When people make asset decisions they do act rationally and consider all the available information in regards to their individual preferences and expectations for return, risk, liquidity and carrying costs.  There is a tremendous amount of uncertainty associated with each of those expectations, and individual decisions are not immune from error.  However, for each buyer there is a seller, which means that in total someone's losses will be offset by someone's gains.  If we stopped the analysis here, markets indeed would be perfectly efficient.  However, markets suffer from a particular flaw - imperfect information.  Specifically, market participants have no way of projecting the impact an individual decision can have on the macro-economy, which will ultimately exert an influence back on them. 
 
For example, if I buy and sell a home for a nice profit, my neighbor may attempt to do the same.  The inflow of new investors drives home prices higher pushing down the potential return.  The paradox of thrift offers another example.  If I am uncertain about my job prospects due to fear of a recession, I am likely to save more and spend less.  However, because I am not dining out as much, a local restaurant owner may cut down working hours causing his employees to worry about their jobs.  This initiates a cycle that turns fear into a self-fulfilling prophecy.   

Clearly, there is a link between our actions and the actions of the people around us; Because of this link, return is not independent of the decision whether to take a risk or not.  If you don't take the risk, you are leaving money on the table.  If you do take the risk, you're driving down returns for everyone.  However, this interdependency is unknowable to individual participants.  We cannot predict the ultimate chain of events that a specific decision can precipitate.  This information feedback loop simply does not exist.  As a result, risks do not offset perfectly but rather tend to compound while the associated returns tend to decline, which is how I would define a bubble.

 
Someone could argue that the information loop can be closed indirectly, especially in the present, media-saturated world.   For example, real estate investors could have heeded warnings of a housing bubble.  However, such warnings were pitted against powerful self-interests.  There were profits to be made in housing.  People and banks would convince themselves that if they didn't act, someone else would.  As housing and financial markets collapsed in 2008, consumers in the United States tightened their belts driving the savings rate to the highest level in decades.  It was a normal instinct – people wanted to make sure that they have sufficient resources should they lose retirement investments or even worse, become unemployed themselves.  However, this behavior multiplied across millions of consumers actually deepened the economic slump.  Whether individuals were aware that fear could be self-fulfilling is irrelevant.  Self-interest dictated that their individual survival and well-being came first.
 

There are numerous examples of rational individual decisions aggregating into irrational macro-economic outcomes.  On a more philosophical note, nobody has perfect knowledge of what prices should be at a particular point in time, not even the market.  Instead, markets find the optimum level by constant trial and error with each oscillation representing some compounding of risk.  This is where money comes into play.  Money exerts tremendous pro-cyclical influence on the economy as to exaggerate normal market fluctuations into full-blown asset booms and busts.
 

Money is created as a result of borrowing (more on the true meaning of money here).  When the economy is strong, individuals and businesses feel more comfortable taking on debt.  Banks expand the money supply to meet higher loan demand.  At the same time, people attempt to allocate more of their wealth away from cash into higher risk assets such as stocks, bonds and real estate.  The supply of money goes up just as demand goes down.  Since money is denominated in itself, changes in money supply and demand affect the prices of other assets.  As a result, an excess of money supply over demand causes an increase in asset prices.  The reverse occurs during an economic downturn.  People are fearful of the future and cut down on spending and debt.  Banks shrink money supply in response to declining loan demand and increasing losses.  At the same time, people attempt to allocate away from risky assets into cash.  Money supply declines just as the demand for money increases, which causes a dramatic drop in asset prices.

Money provides the fuel that can power a normal market fluctuation to the upside into an exuberant bubble.  On the flip-side, money can turn a fluctuation to the downside into a self-propelling depression.  Since monetary policy took center stage as the primary tool for managing the economy in the 1980's, the Federal Reserve has attempted to lean against the pro-cyclical influence of money.  As I explain at length in my post on stock market bubbles, a permanent side effect of an active Fed are asset booms and busts.  Due to a limited tool-set, specifically the lack of control over fiscal deficits and surpluses, the Fed can only control the supply of money, but not what people do with that money (a.k.a. the demand for money).  As a result, the demand for money has been an important driver of asset prices over the last 25 years.  I've copied the chart below from the same post to illustrate the relationship between the S&P 500 Index and changes in money demand  (for the most up-to-date version, please go to my Fed Watch page).

Source: St. Louis Fed; data as of Q3-2013

 
The chart clearly shows that since the 1980's there has been a strong correlation between the S&P 500 Index and changes in money demand.  The two peaks in the S&P 500 Index associated with the dot.com and housing booms can be explained with the declining demand for money.  The precipitous drop in asset prices caused by the crash in 2008 and subsequent recovery can be explained by the dramatic increase in money demand after the Lehman collapse and the efforts by the Federal Reserve to meet that demand through QE.
 
So going back to the opposing views of Fama and Shiller, did we experience a dot.com bubble in the late 1990's and a housing bubble in the mid-2000's?  If I am Eugene Fama, I would answer no - asset prices simply reflected the relative preference for stocks versus cash.  If I am Robert Shiller, I would definitely agree.  Rational decisions by individuals cumulated into irrational exuberance, which drove prices to unsustainable levels.  And, we would both be right!