Overwhelming academic evidence documents the difficulty in distinguishing skill from luck among actively managed mutual funds. Despite this fact, many vendors have attempted to identify those that will beat their benchmarks and deliver excess risk-adjusted returns. Noteworthy among those vendors is Morningstar, which offers forward-looking “analyst ratings.” We’ve evaluated the predictive ability of the first vintage of those ratings, which were published three years ago.
Our results affirm the academic research; it’s really tough to pick a winning mutual fund.
Morningstar’s methodology is documented here. The analyst rating reflects the “conviction in the fund's ability to outperform its peer group and/or relevant benchmark on a risk-adjusted basis over the long term.” Morningstar issues five ratings: gold (the highest), silver, bronze, neutral and negative (the lowest).
The first vintage of ratings, encompassing the 339 funds, is shown here and were the basis for our analysis. Our methodology and results are explained below.
The results of our study
Below is a summary of the 338 funds that survived for the three-year period, along with their analyst ratings and percentage distribution of those ratings:
-
|
Number of Funds |
% |
Gold |
149 |
44.1% |
Silver |
106 |
31.4% |
Bronze |
47 |
13.9% |
Neutral |
30 |
8.9% |
Negative |
6 |
1.8% |
TOTAL |
338 |
100.0% |
To create a baseline for our analysis, we divided the 338 funds into five groups based on their expense ratios2. These ranged from the 149 funds with the lowest expense ratios to the six funds with the highest expense ratios.
We used the single-factor alpha and the excess return to measure the performance of the funds. Morningstar provided all data used in our study directly to us.
Below is a summary of the alphas and excess returns for the funds, based on analyst rating and expense ratio. All averages are unweighted.
|
Average alpha |
Average excess return
|
Analyst rating |
Expense ratio |
Analyst rating |
Expense ratio |
Gold/lowest expense ratio |
0.74 |
0.78 |
0.37 |
0.79 |
Silver/2nd lowest expense ratio |
0.35 |
0.64 |
-0.10 |
0.15 |
Bronze/3rd lowest expense ratio |
-2.13 |
-0.16 |
-0.76 |
-0.72 |
Neutral/4th lowest expense ratio |
0.05 |
-3.43 |
1.42 |
-1.43 |
Negative/highest expense ratio |
0.15 |
-3.89 |
0.32 |
-0.63 |
The results look encouraging for the efficacy of the analyst ratings. They show that gold ratings had positive alphas and positive excess returns.
But there is a problem with the above analysis; the only way for advisors or clients to obtain the positive alphas and excess returns would be to purchase a portfolio of all the funds (i.e., buy an equal amount of all 149 gold funds). That is unrealistic.
One might therefore ask, “Is a randomly chosen fund based on analyst rating likely to have a greater alpha or excess return than a randomly chosen fund based on expense ratio?” In other words, how does a strategy of selecting funds based on analyst rating compare to the naïve strategy of selecting funds purely on the basis of expense ratio?
The results of that analysis are below:
|
Probability that alpha for a randomly selected fund based on analyst rating is greater than alpha for a randomly selected fund based on expense ratio |
Probability that excess return for a randomly selected fund based on analyst rating is greater than excess return for a randomly selected fund based on expense ratio |
Gold/lowest expense ratio |
47% |
44% |
Silver/2nd lowest expense ratio |
52% |
50% |
Bronze/3rd lowest expense ratio |
43% |
52% |
Neutral/4th lowest expense ratio |
61% |
65% |
If the analyst ratings had predictive power, the above percentages would be well above 50%. This was only the case for the neutral funds, which represented 8.9% of our sample3.
These results show that the analyst ratings had little predictive power. Indeed, an advisor would be better off selecting the lowest cost funds than selecting gold funds, based on either alpha or excess return.
A further statistical analysis
We also performed a statistical test on the data using the analysis of variance (ANOVA). This allowed us to compare the means (alpha and excess return) between the analyst ratings and expense ratios groups.
There was only a 0.03 (3%) chance (the P-value) that the differences among the mean alpha values for analyst ratings could have occurred by chance. This would be regarded as "significant at the 0.05 level." Since the average of the gold ratings outperformed the average of the silvers, which outperformed the other averages, this would seem to indicate that the ratings had significant predictive value. This effect would, however, be judged insignificant when measured on excess return instead of alpha since the P-value was 0.054.
However, the analyses of variance when performed on expense ratio alone (divided into categories in the same numbers as the analyst ratings) produced much more significant results. The P-value for alpha is 0.0004 or very highly significant. The P-value for excess return is 0.003, also highly significant.
In short, expense ratio alone is a highly reliable predictor of alpha and excess return, much higher—in fact—than the analyst rating. Since the analyst ratings correlate with expense ratio, the accuracy of the Morningstar predictions may be partly or wholly due to the expense ratio differences among the categories.
Conclusion
Our results are preliminary; they reflect a small portion of the overall mutual fund universe and they reflect three years of performance data. It may well be that if more funds were analyzed over a longer time frame, then the results would be more encouraging.
A half-century of academic research on mutual fund performance has failed to find any methodology to reliably predict risk-adjusted outperformance. When funds do outperform, research has shown, outperformance is rarely persistent, particularly over long time horizons.
Our results are consistent with this academic research. Advisors should be wary of any system that purports to identify funds that will deliver risk-adjusted outperformance. They should be equally wary of funds that promote a gold analyst rating as being predictive of future outperformance.
If you’re looking for a single metric to select better funds, it is a low expense ratio.
Read more articles by Robert Huebscher
