The following letters are in response to our article Luck versus Skill in Active Mutual Funds, which appeared on August 5, 2008.

Dear Editor:

Our industry suffers from the misperception that statistical significance is synonymous with skill. We need to be aware that the evaluation process can only tell us who has succeeded and who has failed, and attribution analyses can tell us why. And yes it is possible to either succeed or fail in a short period of time, although regression analyses preclude this possibility. The assessment of skill can then proceed from this determination of success or failure.

Both evaluation and attribution are only clues that skill might exist, whatever “skill” means. “Skill” is presumably whatever it takes to deliver superior results into the future, and may in fact be mistaken for any number of unrelated factors, but a reasonable and traditional starting point for developing this expectation is a history of success. We can wait for decades to establish this history, as Professor Wermers has, or we can use what little data we have in a cross-sectional hypothesis test called Portfolio Opportunity Distributions. There are no other choices. Most who saw Tiger play amateur golf recognized his talent immediately. If we were using traditional financial tools to evaluate Tiger, namely regression analyses, we’d still be waiting to declare him a winner.

Professor Wermers’ work is legendary, but I fear that the desire for a quick answer has lead us to misinterpret apparent success, or lack thereof, as skill or chicanery.  

Wermers’ research addresses success or failure, not skill. Semantics maybe, but I think success still needs to be carefully examined before we declare a winner skillful.  Ditto a loser.  No set of numbers, including PODs or FDRs, should be relied upon exclusively as a skill identifier. What if the lead analyst has left the firm or the firm was acquired? And what if the “False Discovery Rate” has false positives?  And until someone checks out these unquantifiable aspects all we have are numbers.

I’ve put my thoughts on this in writing. See for example “A Fresh Look at Investment Performance Evaluation: Unifying Best Practices to Improve Timeliness and Reliability.”  Journal of Portfolio Management, Summer 2006. The introduction is as follows, with emphasis added. The point is that it’s the People, Process & Philosophy that deliver skill, as evidenced by the Performance, Relying exclusively on this 4th P is not a good thing, especially since we don’t yet have it right. Perhaps FDR is a better differentiator of the performance “P”, and for that I congratulate Professor Wermers, but someone needs to assess the viability of the other three Ps.

The jury is still out on the use of active investment managers, over the alternative of passive index funds. Research indicates that active managers don’t add value, but common practice suggests that investors don’t believe this research and/or that there are other benefits beyond performance that investors derive from active management. Consultants give clients what they want, so they try to find skillful investment managers, but this is a difficult task that is complicated by the use of performance evaluation tools that suffer from documented deficiencies. An integral part of the search for skill is getting the numbers to tell their most important stories, confirming subjective judgments about the talent of the people and the wisdom of their processes and philosophy.  This story is virtually always told by contrasting a manager’s investment return to an appropriate peer group and index. It’s time to recognize the deficiencies of these popular approaches so we can open up consideration to new contemporary methodologies. 

Ron Surz
PPCA Inc, TDA LLC, and RCG LLC

San Clemente, CA

Display article as PDF for printing.

Would you like to send this article to a friend?