Providing hedge fund ratings and indexes presents significant challenges, as compared to comparable data for mutual funds.   Hedge funds are largely unregulated, invest in illiquid securities, employ (in many cases) significant amounts of leverage, and do not report security level holdings.  The cumulative impact of these challenges is that, when advisors utilize this data, they must clearly understand its applicability and limitations.  Our analysis focuses on the universe of funds in the Morningstar database, to understand any potential biases, as well as on the analytical techniques utilized.

We spoke with John Rekenthaler, Director of Research at Morningstar, in conjunction with this analysis.

Hedge fund databases are notoriously susceptible to two types of biases, survivorship bias and backfill bias.  Survivorship bias occurs when poorly performing funds stop reporting results, thereby artificially inflating the reported results of the remaining universe.  Backfill bias occurs when a fund does not report returns until they reach acceptable levels; poor results from a startup period are omitted.  This has a similar effect of artificially inflating the reported results of the remaining universe.

To address backfill bias, when a fund joins the index its returns are not included retroactively.  This removes the incentive for funds to wait until they have a history of successful returns before they begin reporting, but it does not eliminate backfill bias.  Survivorship bias is accounted for by “freezing” each month's index components, thereby including the performance of funds that become obsolete or are otherwise dropped from the index.  The issue still remains to the extent that poorly performing hedge funds may simply opt to stop reporting their numbers. This type of survivorship bias is inherent with any database that depends upon cooperation from the funds in reporting their data. Rekenthaler notes that 85-90% of funds report their results on time to be included in the index.

Another bias affecting hedge fund universes is the lack of incentive for funds which are closed to investors to report results.  Rekenthaler notes the database is not updated regularly with respect to whether funds are open or closed, and cannot say what percentage of funds fall into each category.  We would expect closed funds to be larger and better performing (higher returns and lower risk), and Rekenthaler confirms that larger funds tend to have less risk (although somewhat lower returns).  Rekenthaler does not have comprehensive performance data comparing open to closed funds.  We would expect the absence of closed funds in the universe to create a bias that artificially deflates the reported results of the remaining universe, but we cannot confirm this.  A primary purpose of the ratings and indexes is to facilitate comparisons among funds for investment purposes.  For this reason, a relative lack of closed funds is less problematic than backfill or survivorship bias, since the remaining universe would be skewed to funds that are open to investors.

Rekenthaler characterizes the index as a mid-sized fund index, primarily because it is equal weighted, but also since large funds are underrepresented (because many are closed or have no incentive to report) and smaller funds are excluded because they fall below the 90% asset inclusion threshold.  Morningstar provides the following breakdown of their fund universe (including those that are not in the index) by AUM, as of February 20, 2008:

As with the previous bias of over-representing open funds, the mid-sized fund bias is less problematic, since it is well-understood and skews the universe toward funds that are more likely to be candidates for investment.

The total AUM of single-strategy funds in the Morningstar universe is $840 billion, of which $627 billion is in the index.  The total size of the global hedge fund industry is a matter of debate, and is estimated to be between $1.5 and $2.5 trillion.  Morningstar’s universe is a sizeable percentage of this total, and an even greater percentage of the US hedge fund industry.

The “unsmoothing” of returns data (see our accompanying article for an explanation of the unsmoothing process) presents another opportunity for bias.  Funds with superior risk controls producing steady returns stand to be penalized by this procedure, even if they invest mostly in liquid securities.  Rekenthaler admits the potential for this problem, but notes that “if it were happening, we would be hearing complaints from the funds affected, and so far that is not occurring.”  On balance, Rekenthaler believes the unsmoothing of data is more of a service than a disservice.

For advisors, the ultimate question is whether the indexes and ratings offer predictive ability for future risk-adjusted performance.  We analyzed Morningstar’s mutual fund ratings in this context, and found that these ratings do offer measurable predictive ability, albeit with some important caveats.  Morningstar’s mutual fund ratings have evolved over last 15 years, undergoing a major revision in 2002.  It is likely hedge fund ratings will undergo a similar evolutionary process, and it is too soon to test whether the hedge fund ratings offer predictive ability.  For the ratings to succeed in this respect, a number of conditions must be met.  First, the classification scheme must accurately reflect industry dynamics and investor’s preferences for diversification.  Second, the process of categorizing funds according to this scheme must be highly reliable.  It is in this respect that Morningstar’s mutual fund ratings have received some of its most vocal criticism.  Third, it must be demonstrated that the biases in the data, noted above, do not mitigate the utility of subsequent data analysis.  Fourth, the degree to which the mathematical model used to determine risk-adjusted performance (i.e., the unsmoothing of data and the utility function) must be shown to have predictive ability vis-à-vis future risk-adjusted return.

The data challenges posed by the hedge fund industry are significantly more complex than with mutual funds.  We believe that Morningstar has taken a reasonable and thoughtful approach to tackling a very difficult problem, one which is extremely important to advisors considering hedge funds as an investment vehicle.  Rekenthaler notes that “with mutual funds, ratings are a starting point for analysis.  With hedge funds, ratings are a starting point for a starting point.”   

You cannot buy a five-star mutual fund and expect that it will do well.  The odds are in your favor, but there is no guarantee.  This is even more true with hedge funds.

Display article as PDF for printing.

Would you like to send this article to a friend?