Is the stock market predictable or inherently unpredictable? If you believe Nassim Taleb, author of The Black Swan
, trying to predict the market is not only impossible, it is also a very risky proposition. I recently finished reading this thought provoking book and a number of interesting points stuck with me. One of these key points is Taleb's claim that the stock market is "Mandelbrotian" by nature, i.e. daily returns in the market are not "normally distributed", they don't follow a neat Gauss-like distribution, where daily returns don't stray too much from the average daily return.
, trying to predict the market is not only impossible, it is also a very risky proposition. I recently finished reading this thought provoking book and a number of interesting points stuck with me. One of these key points is Taleb's claim that the stock market is "Mandelbrotian" by nature, i.e. daily returns in the market are not "normally distributed", they don't follow a neat Gauss-like distribution, where daily returns don't stray too much from the average daily return. Taleb's claim - which I tested for myself (but more about that in a second) - is that stock returns are "scale free". While on most days returns in the market remain in a relatively tight range, once in a while, a Black Swan strikes. A Black Swan is a completely unexpected event that greatly impacts the market in unforeseen ways, resulting in dramatic up or down days. These rare but dramatic events account for a large percentage of stock market returns over the long term.
I must admit that my assumption (even though I never really articulated it) was that while on a daily basis the market can swing up or down, these swings are largely confined to a pretty narrow range that would fall more or less neatly on a normal distribution curve. Well, Taleb claims (and I checked) that this is not the case.
Those that dislike statistics can skip this next paragraph, but for the rest of you, here goes: from Yahoo! Finance, I downloaded the daily returns for the S&P500 from January 3, 1950 to December 4, 2009. Almost 60 years of data. Through the miracle of Excel, I calculated the daily returns on the S&P500 (using closing prices in each case). From this population I calculated the average daily return (0.033%) and the standard deviation (0.966%). I then proceeded to calculate the z-score for each daily return figure. I won't bore you with all the results and analysis, but here are a few eye openers:
- I found a total of 90 days in which the z-score of daily returns exceeded 4 or -4. If stock market returns are normally distributed, we would expect to see one such event every approximately 143 year...
- I also found a total of 24 days in which the z-score of daily returns exceeded 6 or -6. Once again, if stock market returns were normally distributed we would expect to see one such event every approx. 4.6 million years...
- now here's a real doozy: on October 19, 1987 the S&P fell about 20.4% which translates to a z-score of -21.2 or one event every approximately... wait for it... 10 to the power of 93 years. For the sake of comparison, the age of the universe is estimated to be approximately 14.3 times 10 to the power of 17 seconds (or about 6000 years if you choose to get your information from certain unreliable sources). Another comparison point: the number of atoms in the universe is estimated to be approximately 10 to the power of 80...
Point spectacularly made. Stock market returns are NOT normally distributed.
Now, what are the implications of this discovery and what are we to do about it? Well, to be honest with you, I don't think I have good answers to this, but I will do my best to take a crack at some sort of answer over the next couple of days. Stay tuned.
13 comments:
Wow interesting concept. I've never actually considered how returns on the stock market as a whole work.
Fortunately, you've done the z testing and answered the normal distribution pretty promptly.
I don't know if mapping returns on the stock market as a whole really makes sense, I just don't see a reason for it.
Individual companies will do well or poorly based on an almost countless list of reason anywhere from management to technology and even luck.
Shadox:
I believe that stock returns are highly predictable. This is why I believe that stocks are a far less risky asset class than most today realize.
What throws people is that stocks are not even a little bit predictable in the short-term. They are somewhat predictable at 10 years out. And they are highly predictable at 20 years out.
The error that is made in stock investing analysis over and over again is that people look at the short-term and see what happens there and then presume that the same applies in the long term. This leads to all sorts of misunderstandings.
The other key to making effective predictions is including the effect of valuations in your analysis. It's not possible to make effective predictions without taking valuations into account.
Would you consider running the same analysis you described in this blog entry but looking at the long-term reality and including valuations? The idea would be to look at the P/E10 level that applies on any particular day and then going out 20 years and seeing what the annualized long-term return was over that period. I believe that you will find that stock returns have always been highly predictable in the long term for those willing to take valuations into consideration.
I believe that you will also find that a bell curve will apply for the results obtained.
Rob
Individual companies will do well or poorly based on an almost countless list of reason anywhere from management to technology and even luck.
You're right, Edwin.
The returns of individual companies cannot be effectively predicted even in the long term and even by those taking valuations into account, at least not by the average investor (Warren Buffett of course does this all the time -- so there are exceptions to the general rule).
What applies for individual companies does not apply for broad indexes, however. In the DOW or the S&P, there are always going to be some companies that do exceptionally well and some that do exceptionally poorly and it is all going to more or less balance out for the investor in index funds. So now that index funds are available to us, we all can effectively predict the return we will obtain from stocks before putting money on the table (or at least so Rob Bennett believes).
Rob
Guys - I think you may both be missing the point (or more likely I'm not doing a very good job of explaining my point).
Edwin - if the S&P - which is the average of 500 large caps - can be this volatile and unpredictable, the individual components of the average are expected to be even MORE volatile. The reason is that the stocks in the S&P are not precisely correlated, i.e. that don't move in exact lock-step, so some of the volatility is cancelled out. However, I have just shown you that the less volatile average is also subject to very, very extreme moves that are unpredictable by normal distribution... The point - each stock can be expected to behave in a MORE extreme way.
Rob - I am not taking the position I am about to explain, only explaining Taleb's point. I remain uncommitted. The problem with your argument of 20 year predictability is that there haven't been enough non-overlapping periods for us to run the analysis. The daily data I showed you, was based on over 15,000 trading days. If we look at 20 year non-overlapping periods we'll have maybe 5 or 7(?)
The point Taleb makes is that Black Swans are infrequent, unpredictable and massive. So while you may be lulled into a sense of safety by what appears to be 100 years (or 300 years) of consistent 20 year period returns, the next 20 year period can wipe you out completely... Taleb is considering such events as asteroid strikes, collapse of regimes etc.
The problem with your argument of 20 year predictability is that there haven't been enough non-overlapping periods for us to run the analysis. The daily data I showed you, was based on over 15,000 trading days. If we look at 20 year non-overlapping periods we'll have maybe 5 or 7(?)
But why does the entire historical record go against what you would expect from looking at your 15,000 trading days? What are the odds of that if the factors that affect short-term prices are the same as the factors that affect long-term prices? I think it would be fair to say that the odds need to be at least 10 million to one or something like that. This is just an incredible reality. The numbers are not a little off from what you would expect. They are the opposite of what you would expect for the entire historical record.
You have presented convincing evidence that short-term price changes are extremely unpredictable. Why? We have to make sense of that. The Buy-and-Hold Model says that short-term price changes are caused by economic developments. But economic developments are not so wildly unpredictable. What we are seeing with your exploration is that it is NOT economic developments that affect stock prices in the short term. If that is so, the entire conventional model is wrong because that is the fundamental building block.
Then we see that everything changes totally when we go long term. I agree that it would be nice to have more data. But we have to invest regardless, right? It seems to me that we have to come up with some explanation of what is going on.
It seems to me that the thing to do is to believe at least tentatively that there is something to what the entire historical record is telling us -- that long-term stock returns are highly predictable and that Buy-and-Hold is therefore highly dangerous. What alternative is there?
The only alternative is to say, that because we don't have as much historical data as we would like to have, we are going to invest in the way opposite to what the entire historical record says is what works. Why? This I do not get.
Caveats are helpful and sensible. I certainly think that we should be cautious in drawing definitive conclusions from the historical record. But I also think that we should be cautious about claims that it is safe to ignore the entire historical record. That's what the Buy-and-Hold Model does. I see that approach as being as dangerous as dangerous can be.
I don't think we know all the answers with certainty today. But I feel highly confident that we need to get about the business of asking a lot of hard questions about what has come to be the conventional wisdom in recent decades. My belief is that the conventional wisdom is built on sand and that the only way we are ever going to improve on it is to at least examine these questions in depth and in way that invites participation from people with all viewpoints.
I applaud you for the blog entry, Shadox. I believe that you are onto something big here. And I of course don't object even a tiny bit to the fact that your take is not in accord with my own. You are making great and helpful points coming at this from a different perspective. I just wish that all other personal finance blogs were looking at these questions today. I think we need a national debate on these questions aimed at resolving the holes in the Buy-and-Hold Model that have been brought to light in recent years and that have been acknowledged by a good number of big names in the field.
Great stuff here in any event!
Rob
the next 20 year period can wipe you out completely... Taleb is considering such events as asteroid strikes, collapse of regimes etc.
I certainly agree that there is always at least a small chance that we could see the collapse of the U.S. government and that that would render all investments in U.S. stocks worthless. That's true regardless of valuation levels. It's just a reality that we all need to accept.
What I am not able to swallow is the idea that we often hear put forward by advocates of Buy-and-Hold that the recent stock crash or the follow-up crashes we are likely going to see in coming days are somehow a "surprise." There were many, many people who predicted the stock crash in clear and public words long before it happened. Some of the names that come to mind are: (1) Robert Shiller; (2) Cliff Asness; (3) Rob Arnott; (4) John Walter Russell; (5) Andrew Smithers; (6) Ed Easterling; (7) John Mauldin; and (8) Jeremy Grantham. The only people that I know of who didn't see a huge crash coming were the Buy-and-Holders (unfortunately, this group always represents a majority of "experts" at a time of insanely high stock prices).
I believe that there is such a thing as a Black Swan event. But I don't think it is at all right to refer to a price crash taking place at a time of insanely high prices as a Black Swan. A price crash should be the expected thing in those circumstances. There has never been a time when we got to insanely high prices when we did not see a price crash. I am not even able to imagine a scenario in which a crash could be avoided in such circumstances. Once prices get to three times fair value (as they did at the top of the bubble), how else are they to return to reasonable levels except through a crash?
We need to separate out the things that are true surprises from the things that have come to be perceived as surprising only because a large number have come to be believers in the Buy-and-Hold Model in recent decades.
Rob
And they are highly predictable at 20 years out.
Rob, I went to your Stock Return Predictor. When I plug in the value of the S&P500 today (1,102.38 as I am writing this, P/E10=19.7) I get a range of 20 year stock returns of best possible=8.06,worst possible=0.06
Over 20 years this would turn $10,000 into $47,130.19 and $10,120.68 respectively.
How can you claim that stock returns "are highly predictable at 20 years out"
What a z test of the stock market as a whole does is show us the volatility of it as a whole and that it does not fit a normal distribution. I don't see this as telling me the stock market is totally unpredictable.
I do actually think if you had the time, resources, and knowledge (such as Warren Buffet), you can do well predicting individual companies (note - not the market).
Sahdox, I agree that individual stocks are more volatile because the outside factors required to affect them don't need to be as severe as for the market as a whole. For example a shock in copper prices may affect semiconductor manufacturing but it won't have a large effect on the market.
But, if you are a very savvy business person, know how to look at a business's financial AND physical operations AND know the industry very well such as things that affect it, I think you can be very successful in predicting individual companies (i.e. individual stocks). Of course this requires you have the knowledge, time, and even capital so companies will let you take a look at their operations.
I think buy and hold is a secure strategy when applied to the market as a whole to attempt and get a large enough portfolio. This means index funds. I don't think buy and hold is the best strategy if you have a less diverse portfolio, say... if you only hold 10 stocks.
I get a range of 20 year stock returns of best possible=8.06,worst possible=0.06
The point you are making (which is an important one) goes more to the precision of predictions than to the validity of them, Evidence.
If we were starting from the valuations level that applied at the top of the bubble, the range would be from 5 percent to a negative 3 percent.
The range in both cases is 8 percentage points of return. But the returns covered by that 8 percent range changes when the valuation level at which the purchase is made changes. It's the need to take valuations into consideration that is confirmed by the entire historical record.
We know that valuations matter. We know that short-term predictions never work. We know that long-term predictions always work. And we know that precise predictions are not possible, even in the long term (if you go out 60 years, you enjoy much more precision, however). All four things are so.
If you want a more narrow range of possibilities, you can obtain it by looking at what will happen in 80 percent of all possible scenarios. For a stock purchase taking place at today's valuations, you can say that there is only a 20 percent chance that your 20-year return will be worse than 2 percent annualized or better than 6 percent annualized.
It of course would be nice to have even better predictability than that. But knowing the numbers quoted above is no small matter. Shadox's examination of what happens in the short term showed no predictability whatsoever. I think it is fair to say that the story is very, very different in the long term.
It's better, by the way. Predictability is a good thing. It amazes me that there is any resistance at all to something that is such wonderfully good news. We should all want long-term returns to be highly predictable and it is good news indeed that the entire historical record show that they are. We all should be happy to have discovered this exciting reality of stock investing, in my view.
Rob
Been a while since I did much statistics, but I seem to recall that there are other distributions than Gaussian - so you might still have another distribution (e.g. Poisson) where the shape is not the bell curve but the distribution is still predicable - does your analysis rule that out?
Anon - sorry for the late response. I am on a family vacation in rainy, but awesome Costa Rica...
Nope, I did not test other distributions myself, however Taleb claims in his book (and I have seen other such claims in the past) that the distribution follows a power-law. Taleb also calls this a self-similar distribution, where at each level of "inequality", "inequality" remains the same. A brief explanation by way of example: if this distribution were true for personal wealth, if the top 1% of income earners earned 50% of all income in the nation, the top 1% of the top 1% would earn 50% of the income generated by the top 1% of the population (or 25% of income for the entire population). i.e. the super rich would be just as different from the rich, as the rich are different from the rest of us.
Hope this helps.
Quit asking money related questions. Money is not vital
Galileo, I believe the name of this blog is "Money and Such" and money tends to fall into one of those two categories.
Post a Comment