As part of our research on Morningstar’s new rating methodology, we sought to answer a question that is fundamentally important to advisors:  What is the probability that a randomly chosen fund with one rating will outperform a randomly chosen fund with another rating?  For example, what is the probability that a randomly chosen 5-star fund will outperform a randomly chosen 4-star fund?  With the help of John Rekenthaler, the VP of Research and Product Development at Morningstar, and his research team, we are able to answer this question.

The results reveal the following:

• With only minor exceptions, the results are monotonic, in that 5-star funds outperform 4-star funds, 4-star funds outperform 3-star funds, etc.  This supports the findings of Morey and Gottesman in their study.

• Ratings have the most predictive value for Balanced, Taxable Bond, and Muni Bond funds.  This is expected, since the performance in fixed income funds can be calibrated based on the fund’s credit rating and maturity characteristics, and is far less susceptible to individual security selection than in pure equity funds.

• Ratings have the least predictive value for International Equities.  In fact, the one instance where the predictive value of the ratings is not monotonic is in International Equities, where a randomly chosen 5-star fund outperforms a randomly chosen 4-star fund only 47.66% of the time (48.12% of the time using the risk-adjusted numbers).

• Ratings for US Equities have more predictive value than for International Equities, but are not as predictive as for funds with a fixed income component.

• Advisors should pay particular attention to the probabilities of funds with a given rating outperforming the funds with the next lower rating.  For example, within US Equities, a randomly chosen 5-star fund will outperform a randomly chosen 4-star fund 54.64% of the time (or 55.21% using the risk-adjusted numbers).

There are a number of important caveats that apply to these results:

• These results are based on only three years of data, 2004-2006, a period which is almost entirely an up market.  A more comprehensive test of the ratings methodology needs to include a full market cycle, including a down market.

• This analysis does not take into account survivorship.  As we know from the Morey and Gottesman study, approximately 20% of funds at beginning of the three year period will be merged or terminated by the end of the period, and are excluded from this study.  This is much more likely to occur with lower rated funds (specifically 1- and 2-star funds), since fund companies are less likely to merge or terminate a higher rated fund.  As a result, we can expect that the probabilities below understate the true probabilities for funds outperforming these lower rated funds.  But this is only conjecture and would need to be verified by additional data analysis.

• Morningstar’s assignment of ratings takes into account the effect of sales loads, but the rate of return calculations below (both total rate of return and risk-adjusted rate of return) do not reflect sales loads.  The universe of funds used includes load and no-load funds.  The inclusion of loads would reduce the performance of those funds.  If Morningstar had included the effect of star ratings on the total return and risk-adjusted return calculations, the results would have been better for the predictive value of the rating, as load funds have lower overall ratings than no-load funds. So in that sense, this study is a conservative look at the predictive value of the ratings because, to the extent that any loads are paid, the ratings would perform better. Ideally, from an advisor’s perspective, we would like to see this analysis done with a universe of no-load or load waived funds.

• This analysis does not utilize the risk-adjustment metrics employed by Morey and Gottesman (alpha and Sharpe Ratio).  Ideally, advisors would like to know the probabilities of fund outperformance based on alpha (i.e., what is the probability of a fund with rating x outperforming a fund with rating y on a risk-adjusted basis?)

Overall, we believe this study provides considerably more actionable information than the Morey and Gottesman study.  Advisors are often faced with the challenge of deciding between funds with different ratings (e.g., a 5-star versus a 4-star fund).  In the absence of other information, the probabilities below provide useful guidance as to the chances of one fund outperforming the other.  In a similar vein, these probabilities can be used by advisors in their discussions with investors, when explaining the chances of outperformance associated with funds of different ratings.

Methodology

Rekenthaler and his team looked at the ratings of actively managed funds as of the end of 2003 and measured the performance of these funds at the end of 2006 (i.e., three years of data).  Only funds that survived over this three year period were considered.  Index funds, ETFs, and other funds not rated by Morningstar were excluded from this analysis.  Each share class of a fund was treated separately (since different share classes can be rated differently).  The results for the fund universe are reported in five categories:  US Equity, International Equity, Balanced, Taxable Bond, and Muni Bond. 

To calculate the probability of a fund with a rating of ‘x’ outperforming a fund with a rating of ‘y,’ the performance of each fund with a rating of ‘x’ was compared against the performance of all funds with a rating of ‘y.’ This yielded the probability of the fund outperforming a randomly chosen fund with a rating of ‘y.’ The probabilities of all funds with a rating of ‘x’ were computed and averaged to arrive at the final answer.

The probabilities are presented in the two tables below.  The first table shows the probabilities based on the total rate of return, as reported by Morningstar.  The second table shows the probabilities based on the risk-adjusted rated of return.  This risk-adjusted rate of return uses Morningstar’s methodology.  This is the methodology Morningstar uses to compute performance when ranking funds, but it is not the same methodology used by Morey and Gottesman in their study. 

Table 1 –Probabilities based on Total Return

Table 2 – Probabilities based on Risk-Adjusted Return

Display article as PDF for printing.

Would you like to send this article to a friend?