Rating the Star Rating

Our study looks at how a revised Morningstar Rating has performed.

by Russel Kinnel | 12-05-05 | 06:00 AM | E-mail Article | Print Article | Permissions/Reprints

It has been more than three years since we changed the methodology we use to calculate the Morningstar Rating for funds. We now have enough data to examine in-depth whether our revamped star ratings can help steer investors toward funds that will produce strong, risk-adjusted, load-adjusted returns in the future.

But there's more to our study than that. We wanted to know how the star rating has fared at predicting the individual component parts of relative performance and volatility. We also wanted to see the details of the types of funds for which the rating system works best, as well as those groups for which it didn't work as well.

The Star Rating: Now and Then

When we revamped our rating system in 2002, we switched from rating funds within four broad asset classes to rating them within their specific categories . We overhauled the rating system to improve the chances that funds would earn high ratings when their managers added value rather than when their categories performed well.

We also wanted the star ratings to better reflect funds' risk levels, so we adopted a utility function measure of risk-adjusted returns that penalizes funds for volatility.

A lot of the fund rating system remained the same. As in the past, the star rating is still a quantitative measure that we update monthly. We also continue to grade funds on a curve, with 10% of portfolios getting 5 stars, 22.5% getting 4 stars, 33% getting 3 stars, 22.5% getting 2 stars, and 10% getting 1 star. We calculate separate ratings for the three-, five-, and 10-year periods (where applicable), then combine those ratings into an aggregate rating for each fund. We still adjust for loads.

How We Ran the Study

We recalculate new Morningstar Ratings for all the eligible funds in our database every month. That means we've issued about 40 sets of star ratings under the new methodology. Our study focuses on two of those sets of ratings: those from June 2002 and those from June 2003.

The 2002 ratings enable us to get our first view of how a rating predicts risk-adjusted performance over a three-year time period. We also studied the ratings from June 2003 in an effort to check how the rating system responded to a change in the market environment.

A Pretty Good Three Years

The ratings from June 2002 generally guided investors to funds that went on to generate superior returns in the ensuing three years, after adjusting for loads and risk. That is, 5-star funds performed better from mid-2002 through mid-2005 than did 4-star funds; 4-star funds outperformed 3-star funds, and so on. However, the amount of outperformance was often modest. Thus, the rating was directionally right but not exactly a means to getting rich quick.

Let's begin with how well a high star rating in June 2002 predicted a fund's three-year star rating in mid-2005. Among domestic-stock funds, those with 5-star ratings in 2002 earned an average rating of 3.1 stars in 2005. Funds with 4 stars in 2002 earned an average rating of 3.0 in 2005, funds with 3 stars in 2002 averaged 2.8 stars three years later, and 2- and 1-star funds earned average ratings of 2.7 each.

The star rating showed greater predictive ability when applied to funds in the other broad asset classes. Among international funds, 5-star funds in mid-2002 earned an average rating of 3.4 stars three years later, while 1-star funds went on to average 2.4 stars. Taxable-bond funds with 5 stars in 2002 earned an average rating of 3.4 stars in 2005, while 1-star funds averaged 2.6 stars. Muni funds with 5-star ratings went on to earn an average rating of 3.8 stars, whereas 1-star funds received ratings of just 2.3 stars, on average, three year later.

 Breaking Down the 2002 Ratings

Asset Class Morningstar

Rating 6-30-02 3-Yr.

Return % 3-Yr. Ret.

% Rank 3-Year

Std. Dev.** 3-Yr. Rating

6-30-05

Domestic Equity  10.1 43 14.1 3.1

   9.3 49 14.0 3.0

   9.2 50 14.5 2.8

   9.0 53 15.2 2.7

   8.1 54 17.3 2.7

International Equity  15.3 43 15.9 3.4

   15.1 47 17.2 3.0

   14.1 51 17.1 2.9

   12.5 59 17.2 2.8

   16.7 43 20.3 2.4

Taxable Bond  7.4 43 4.6 3.4

   6.9 48 4.4 3.3

   6.9 50 4.5 3.0

   5.9 56 4.3 2.5

   7.3 46 4.9 2.6

Muni Bond  8.3 35 7.9 3.8

   7.1 44 7.4 3.3

   6.1 50 7.2 3.0

   5.4 57 6.6 2.6

   5.7 58 7.3 2.3

* Period from June 30, 2002, to June 30, 2005

Predicting Pure Performance

We risk-adjust the star rating for the simple reason that high-risk funds are hard to handle. We've done a number of studies, including a recent analysis of dollar-weighted returns, that show investors capture much less of the official returns of volatile funds than they do of steadier funds. So, we're more interested in helping investors make good decisions than in predicting absolute returns. In a sense, we're trying to predict investors' returns rather than the official ones.

Nonetheless, you may also want to know how well the star rating predicts raw returns. The answer is that it generally did a decent job: 5-star funds outperformed lower-rated offerings across each asset class.

In domestic stocks, 5-star funds returned an average of 10.1%, compared with 8.1% for 1-star funds. Taxable-bond funds with 5 stars returned 7.4%, compared with 5.9% for 2-star funds and 7.3% for 1-star funds. (Funds with low-quality bonds enjoyed a strong rally after 2002 that lifted 1-star funds.) Five-star muni funds returned about 8.3%, compared with 5.7% for 1-star funds. Compared against their category averages, 5-star funds outperformed by 90 basis points in the two stock categories, by 70 basis points in taxable bonds, and by a hefty 200 basis points in muni bonds.

However, there was a notable quirk in the results. Five-star foreign funds did just fine, on average, producing returns that landed in the 43rd percentile within their peer group that existed as of June 2002. Likewise, foreign funds with 4 stars in mid-2002 landed slightly above average within their peer groups over the following three years. But in this case, 1-star funds did just as well as 5-star funds measured purely by returns. On average, international funds with 1 star in mid-2002 also landed in the top 43rd percentile of their peer groups in mid-2005, while 2-star funds produced the worst performance, with an average of 59th percentile. One-star funds didn't look as attractive after adjusting for loads and risk, however. They had an average three-year star rating of 2.4 stars for the next three years.

Why the odd twist? First, as noted, we hadn't broken up foreign-stock categories by style at that point. This made the star rating less accurate at locating manager skill because within the older, larger categories, asset mix could overwhelm skill.

In addition, 2002 was a turning point for emerging markets. Some foreign funds had minuscule weightings and others had 15%-20% of assets in emerging markets. The funds with big emerging-markets bets would have looked subpar in June 2002, but they've generally fared quite well since then. Thus, higher-risk strategies paid off. That's why the 1-star funds still received poor star ratings even though they had performance that matched those of 5-star funds.

However, if we look at returns beginning in June 2003, we see much better performance from the star rating. The average 5-star foreign fund produced performance in the 35th percentile for the period June 2003 to June 2005. One-star funds were in the 57th percentile on average. In addition, 5-star funds were significantly less volatile over the ensuing two years. The foreign style-box categories were introduced in October 2003, so we expect them to make for a more dependable rating.

 Testing the June 2003 Ratings

Asset Class Morningstar Rating 6-30-03 2-Yr Rtn*  2-Yr Rtn

% Rank 2-Yr Load Adj Rtn* 2-Yr Load Adj Rtn % Rank

Domestic Equity  15.66 42 15.53 42

   14.36 49 14.19 49

   14.12 50 13.92 50

   14.05 52 13.81 52

   13.47 54 13.24 54

International Equity  25.84 35 25.70 35

   23.70 43 23.49 43

   21.81 52 21.61 52

   21.25 56 20.99 56

   21.07 57 20.79 57

Taxable Bond  5.42 40 5.38 39

   4.75 46 4.69 45

   4.72 49 4.57 49

   3.76 56 3.52 58

   3.87 57 3.65 58

Muni Bond  4.34 33 4.31 28

   3.85 44 3.69 42

   3.63 49 3.38 49

   3.30 55 3.03 56

   3.03 64 2.82 64

* Period from June 30, 2003, to June 30, 2005

Characteristics of 5-Star Funds

To get an idea of what kinds of funds the star rating favors, we looked at overall average statistics for 5-star funds as of June 30, 2005. We found the typical 5-star fund is less expensive and managed by a longer-tenured manager than those with lower ratings. The typical 5-star domestic-stock fund has an expense ratio of 1.30%, compared with 1.87% for the typical 1-star fund. The expense gap between 5-star and 1-star foreign-stock funds is 71 basis points, 61 basis points for taxable bond funds, and a remarkable 84 basis points for muni funds.

The average domestic-stock fund with a rating of 5 stars has a manager with a tenure of 6.5 years, compared with 4.5 years on average for 1-star funds. The pattern holds up for foreign and taxable bonds. However, the trend reverses for munis, where managers have high experience levels across the board. The average 5-star muni fund manager has 6.7 years of experience, but the typical 1-star muni fund boasts 7.8 years.

Putting It to Work

We're pleased that the star rating generally points investors in the right direction. Not only do 5-star funds outperform lower-rated funds on average, but they also tend to be cheaper, less volatile, and run by more experienced managers at the time they earn their 5 stars.

Thus, the star rating can be a helpful tool in winnowing the field of potential investments or tracking your existing holdings. However, note that we said "generally" and "on average." Despite the star rating's merits, there are sure to be some duds among 5-star funds and some diamonds in the rough among lower-rated funds. In fact, we even have a couple of 2-star Analyst Picks, and one of our Analyst Pans has earned 5 stars. That's because we are big believers in fundamental analysis and recognize that the star rating is only a reflection of past risk-adjusted performance. Investing is much too complex for any single measure to sum up the entire merit of a security.

The Morningstar Rating for funds does a fine job of reflecting risk, loads, and past performance. However, the people behind a fund can change, as do other key characteristics that require in-depth analysis. In short, the star rating is a useful tool, but it's no substitute for doing your homework.

A version of this article will appear in the December 2005 issue of Morningstar FundInvestor