Substandard and Porous? A Belated Response to Nate Silver

After S&P downgraded the US last August, Nate Silver analyzed the agency’s record on sovereign debt and found it wanting (See Why S&P’s Ratings Are Substandard and Porous). Silver ran a number of statistical tests, and determined that S&P’s ratings were serially correlated, highly related to the Corruption Perceptions Index and less predictive of default than simple quantitative measures.

Silver’s analysis is impressive for an industry outsider, but it suffered some deficiencies. For example, he applied a linear scale (AAA = 9, AA = 8, A = 7, etc.) when mapping ratings to decimal values. Given the structure of historic default rates, some sort of geometric scaling would have been more appropriate.

Our criticism, however, should not compromise Silver’s core point: that statistical analysis can probably do a better job of telling us about sovereign default probabilities than the traditional rating agency approach.

An Argument for a Model-Based Approach

Certainly, an intensive statistical analysis avoids several of the pitfalls facing sovereign ratings, as currently implemented.

First, a rating methodology that relies heavily on qualitative techniques is vulnerable to bias. The serial correlation Silver found stems from a natural bias at rating agencies against extreme actions. Rather than imposing a large-scale, multiple-notch downgrade, rating committees may be predisposed to implementing a lesser, often single-notch change in the hope that subsequent events will obviate further action. While the biased rater might thus apply a rating inconsistent with the methodology, a computer model, lacking the capacity to “hope,” reverts directly to the honest, brutal truth.

There’s also a more fundamental objection to the rating agency model. Their qualitative approach, requiring the human touch, is labor intensive. But for-profit rating agencies are oddly notorious for understaffing their sovereign rating groups.

A model-based approach enables more frequent, more intensive analysis – as opposed to the infrequent reviews sovereigns now receive. New data can be loaded into the model at regular intervals and new results calculated. Analysts should still oversee the model parameters and check any results that may look suspicious.

The Ingredients of a Sovereign Debt Model

Model-based approaches to sovereign risk often involve credit default swap spreads, as the independent or dependent variable. A model can either extract default probabilities from CDS spreads, or attempt to predict those spreads on the grounds that they are a proxy for actual risk. Silver takes this latter approach in his piece.

The use of market inputs in credit models is fairly common. Such a modeling choice often implicitly or explicitly relies on the Efficient Market Hypothesis – the idea that market prices incorporate all relevant information and are thus the best available estimate of value.

Since the financial crisis, critics of EMH have sharpened their attacks. But whether or not you subscribe to rational expectations, the use of sovereign CDS is hard to defend. Most EMH advocates recognize that only liquid markets are efficient. Since liquid markets have numerous participants, their equilibrium prices incorporate substantial amounts of information.

This is not the case with sovereign CDS markets. Kamakura Corporation examined sovereign trading volumes reported by DTCC for late 2009 and 2010, and found that the vast majority of sovereign CDS contracts were traded fewer than five times per day (excluding inter-dealer trades). Five transactions per day falls well short of a liquid market, and thus the information content of sovereign CDS spreads is doubtful at best.

Absent meaningful CDS spread data, what else can a government credit model rely upon?

While one might look at Corruption Perception Indices, per capita GDP and/or terms of trade, it is not clear that these inputs will differentiate between advanced economy sovereigns and sub-sovereigns. Fortunately, government issuers produce reams of actual and projected fiscal data. This information, combined with demographic inputs and economic forecasts, can take us a long way.

When we suggest that budget forecasts can be employed in government credit modeling, skeptics point out accuracy issues with government forecasters.

The most famous forecasting error is attributed to the US CBO, which predicted trillions of surpluses for the first decade of the 21st century, instead of the trillions in deficits that actually appeared.

CBO forecasts are usually published in the form of point estimates. To be reliable, they have to reflect accurate forecasts of interest rates, GDP, tax levels and a host of other macroeconomic and policy variables. Given the number of variables and our (collective) limited capacity to predict, the point estimate is bound to be wrong. That notwithstanding, we can be pretty certain that these variables will fall within a given range. For example, it is almost certain that US GDP growth will be somewhere between -3% and +6% next year (2013). If we run a large number of scenarios with different GDP growth rates within this range, it is likely that some of the trials will closely approximate the ultimate fiscal outcome.

We can run a large number of budget scenarios by using a Monte Carlo simulation – in which scenarios are created by generating random numbers. Budget simulation forms the basis for PF2’s Public Sector Credit Framework that we will release next week. The tool allows the user to enter a default threshold in the form of a fiscal ratio; create macroeconomic series that vary with each trial through linkages to random numbers; and design fiscal series that rely on one or more of these macroeconomic elements. If you would like to learn more about this technology, please contact us at info@pf2se.com, or call +1 212-797-0215.
------------------------
Contributed by PF2 consultant Marc Joffe. Marc previously researched and co-authored Kroll Bond Rating Agency’s Municipal Default Study. This is the last of four blog posts introducing PF2’s Public Sector Credit Framework. Previous posts on this topic may be found here, here and here.

Pro Bono Finance

Lawyers fight to save death row inmates. Doctors provide charity care – treating the indigent, often without government reimbursement. In the financial services industry, volunteer work usually takes the form of pitching in at schools and cleaning parks. We finance folks lack a tradition of using our skills for public service. With our reputation in tatters, perhaps it is time to begin such a tradition.

Fears of sovereign and municipal debt crises offer a worthy volunteer opportunity. Government debt problems can easily become matters of life and death. Argentina’s sovereign debt crisis claimed 24 lives in December 2001. In Greece, crisis-related protests have claimed at least 5 lives and caused over 300 injuries. Annual suicide rates are up about 20 percent, as people despair over their diminished circumstances.

A US federal debt crisis could similarly lead to violent protests, fatalities and widespread psychological damage; it could also be accompanied by high levels of inflation and sudden, sharp cuts in benefits. Older people dependent on savings and social insurance payments would be especially hard hit.

Because hurricanes and tornados kill and injure, scientists have invested substantial time and effort in forecasting these natural disasters and helping members of the public avoid them. Fiscal crises – a type of human-made disaster – can also be anticipated and potentially curbed, or even avoided.

Last summer’s debt ceiling debate was a failed opportunity to avoid a US fiscal crisis. As we look back on the debate, it becomes evident that false and misleading rhetoric frequently crowded out accurate information. Among the myths that plagued last year’s discussion:

• Failing to raise the debt ceiling would have inevitably triggered a default.¹
  
  
• The US government has never defaulted in its entire history (see our earlier blog post for the facts).
  
  
• The nation’s long term budget imbalance can be resolved simply by controlling domestic discretionary spending or allowing the Bush tax cuts on high earners to expire (neither of these steps generate enough savings to avoid future problems).<
  
  
• A 90% Debt-to-GDP ratio will trigger a fiscal crisis (see Japan).
  
  
• We need to balance the budget to avoid a fiscal crisis (the Debt-to-GDP ratio will improve as long as the stock of debt grows more slowly than GDP).
  
  
• Rapid economic growth is impossible if the federal government spends more than 20% of GDP (see 1999 economic and fiscal statistics for a refutation of this contention).

The financial community can provide a useful community service by educating the public about sovereign, state and municipal credit issues. And when I say educate, I don’t mean pontificate. Many of us - this writer included - hold views about what should be done about taxes and spending. Mixing these opinions with facts is not an unambiguous public service. Just as we strive to dispassionately evaluate credit and select investment opportunities, we can and should separate fact from opinion when informing voters about their fiscal options.

Mary Meeker and her colleagues issued a free report entitled “USA, Inc.” that provided the type of service I am suggesting. The report, which received substantial publicity, analyzed the nation’s fiscal position in a manner similar to that of an equity investor analyzing a business – with hundreds of slides describing revenue and expense drivers.

The next challenge is to broaden the scope of analysis to provide a credit perspective with its focus on default risk and recovery. Also, rather than misapply corporate or structured debt analytics, we need a fresh approach that directly addresses the unique challenges of assessing government debt. We’ll also be better positioned if the analysis is ongoing, or even real-time, rather than a “snapshot” analysis provided in the Meeker report.

Ideally, having a well-structured, up-to-date model on which to base policy positions seems enviable. If would assist Congress and the Administration in defining the task at hand and dispelling myths surrounding the task – and it would encourage an environment in which commentators would be required to support their opinions with quantifiable data, not simply foggy criteria.

At PF2, we will kick-start the effort to dispel the fog of opinion by offering a free, open source Public Sector Credit Framework (PSCF). Our framework will be accompanied by a timely, transparent US federal budget simulation model, which we’ve designed to estimate the likelihood of a fiscal crisis in each of the next thirty years. Although we don’t know everything about this topic, we do know that many financial professionals are equipped to improve the software and the model. We encourage you all to join us in enhancing the analysis.

Few can afford to provide pro bono services exclusively – and we are no exception. If the framework generates interest, we may use it as a platform for valuing sovereign and municipal bonds – a service we hope to monetize. But that notwithstanding, the software and model are being supplied at no charge under GNU’s Lesser General Public License, for anyone to use and improve. Assuming interest is sufficient, we will regularly update the US federal model as a public service.

Many of us in the financial service industry have done quite well. Having reaped some of the rewards, we think we have found a great way to give back. We look forward to your collaboration.

------------------------------------
¹ Treasury could have avoided a default through some combination of asset sales and spending reductions. The President could have invoked a clause in the 14^th Amendment of the Constitution to mandate principal and interest payments that would have exceeded the debt ceiling. Congress would then have had to file a legal case to overturn the President’s order.

Contributed by PF2 consultant Marc Joffe. Marc previously researched and co-authored Kroll Bond Rating Agency’s Municipal Default Study. This posting is the third in a series of posts leading up to May 2nd. The prior pieces can be accessed by clicking here and here.

Credit Rating Agency Models and Open Source

When S&P downgraded the US from AAA to AA+, the US Treasury accused the rating agency of making a $2 trillion mathematical error. S&P initially denied this accusation, but adjusted some of its estimates in a subsequent press release. Economist John Taylor defended S&P, contending that its calculations were based on a defensible set of assumptions, and thus could not be categorized as a mistake. S&P’s model, which projected future debt-to-GDP ratios, has not been made public. As a result, it is difficult for outside observers to decide whom to believe: the rater or the rated.

There are at least three ways a model’s results can be wrong: if the model’s code itself doesn’t function as intended; if the known inputs are incorrectly entered, and if the assumptions are misapplied. In cases as important as the evaluation of US sovereign debt, we think rating agencies and the investing public would be better off if the relevant models were publicly available. Some may argue that the inputs to the models are proprietary or that they reflect qualitative assumptions valuable to the ratings agencies – i.e., that they are a “secret sauce.” But, even if rating agencies want to keep their assumptions proprietary, making the models themselves available would decrease the likelihood of rating errors arising from software defects.

Keeping one’s internal processes internal is the traditional way. Manufacturers assume that consumers don’t want to see how the sausages are made. In the internet era, it is now much easier to produce the intellectual equivalent of sausages in public – and, as it happens, many consumers are interested in the production process and even want to get involved. Wikipedia provides an excellent example of the open, collaborative production of intellectual content: articles are edited in public and the results are often subject to dispute. Writers get almost instantaneous peer review and the outcome is often rapid iteration moving toward the truth. In their books, Wikinomics and Macrowikinomics, Dan Tapscott and Anthony Williams suggest that Wikipedia’s mass collaboration style is the wave of the future for many industries – including computer software.

Many rating methodologies, especially in the area of structured finance, rely upon computer software. At the height of the last cycle, tools that implemented rating methodologies such as Moody’s CDOROM^TM, were popular with both issuers and investors wondering how agencies might look at a given transaction. While the algorithms used by these programs are often well documented, the computer source code is usually not released into the public domain.

Over the last two decades, the software industry has seen a growing trend toward open source technology, in which all of a system’s underlying program code is made public. The best known example of open source system is Linux, a computer operating system used by most servers on the internet. Other examples of popular open source programs include Mozilla’s Firefox web browser, the WordPress content management system and the MySQL database.

In financial services, the Quantlib project has created a comprehensive open source framework for quantitative finance. The library, which has been available for more than 11 years, includes a wide array of engines for pricing options and other derivatives.

Open source allows users to see how programs work and with the help of developers, fully customize software to meet their specific needs. Open source communities such as those hosted on GitHub and SourceForge, enable users and programmers from all over the world to participate in the process of debugging and enhancing the software.

So how about credit rating methodologies? Open source seems especially appropriate for rating models. Rating agencies realize relatively little revenue from selling rating models; they are more likely to be used to facilitate revenue generation through issuer-paid ratings.

Open source enables a larger community to identify and fix bugs. If rating model source code were in the public domain, investors and issuers would have a greater chance to spot issues. Rating agencies would be prevented from covering up modeling errors by surreptitiously changing their methodologies. In 2008, The Financial Times reported that Moody’s errantly awarded Aaa credit ratings to a number of Constant Proportion Debt Obligations (CPDOs) due to a software glitch. The error was fixed, but the incorrectly rated securities were not immediately downgraded according to the FT report. Had the rating software been open source, it would not have been much more difficult to conceal this error, and it would have offered the possibility for a positive feedback loop – an investor or other interested party could have found and fixed the bug on Moody’s behalf.

Not only do open source rating models promote quality, they may also reduce litigation. The SEC issued Moody’s a Wells Notice in respect of the above mentioned CPDO issue, and may well have brought suit. (A Wells Notice is a notification of intent to recommend that the US government pursue enforcement proceedings, and is sent by regulators to a company or a person.) Investors have brought suit against the rating agencies to the extent they felt the ratings were inappropriate, for model-related errors or otherwise. By unveiling the black box, the rating agencies would be taking an active approach in buffering against litigation, and enjoy the material defense that, “yes we may have erred, but you were afforded the opportunity to catch our error – and didn’t.”

Unlike the CPDO model employed by Moody’s, the S&P US sovereign "model" likely took the form of a simple spreadsheet containing adjusted forecasts from the Congressional Budget Office. In contrast to the structured and corporate sectors, there are relatively few computer models for estimating sovereign and municipal default probabilities. While little modeling software is available for this sector, accurate modeling of government credit can be seen as a public good. Bond investors, policy makers and citizens themselves could all benefit from more systematic analysis of government solvency.

Open source communities are a private response to public goods problems: individuals collaborate to provide tools that might otherwise appear in the realm of licensed software. Thus open source government default models populated with crowd-sourced data maybe the best way to fill an apparent gap in the bond analytics market.

On May 2^nd, PF2 will contribute an open source Public Sector Credit Framework, which is aimed at filling this analytical gap, while demonstrating how future rating models can be distributed and improved in an iterative, transparent manner. If you wish to participate in beta testing or learn more about this technology please contact us at info@pf2se.com, or call +1 212-797-0215.

--------------------------------------------
Contributed by PF2 consultant Marc Joffe. Marc previously researched and co-authored Kroll Bond Rating Agency’s Municipal Default Study. This posting is the second in a series of posts leading up to May 2nd. The prior piece can be accessed by clicking here.

Multiple Rating Scales: When A Isn’t A

Philosophers from Aristotle to Ayn Rand have contended that “A is A.” Apparently none of these thinkers worked at a credit rating agency - in which “A” in one department may actually mean AA or even BBB in another. While the uninitiated might naively assume that various types of bonds carrying the same rating have the same level of credit risk, history shows otherwise.

During the credit crisis, AAA RMBS and ABS CDO tranches experienced far higher default rates than similarly rated corporate and government securities. Less well known is the fact that municipal bonds have for decades experienced substantially lower default rates than identically rated corporate securities – and that the rating agencies never assumed that a single A-rated issuer ought to carry the same credit risk in both sectors. This discrepancy was noted in Fitch’s 1999 municipal bond study and confirmed by Moody’s executive Laura Levenstein in 2008 Congressional testimony on the topic. Later in 2008, the Connecticut attorney general sued the three major rating agencies for under-rating municipal bond issues relative to other asset categories. (The suit was recently settled for $900,000 in credits for future rating services, but without any admission of responsibility). Last year, three economists – Cornaggia, Cornaggia and Hund – reported that government credit ratings were harsher than those assigned to corporates, which, in turn, were more severe than those assigned to structured finance issues.

One might ask why it is important for ratings in different credit classes to carry the same expectation in terms of either default probability or expected loss? Perhaps we should accept the argument that ratings are intended to simply provide a relative measure of risk among bonds within a given asset class.

There are at least two problems with this approach. First, it is unnecessarily confusing to the majority of the population that is unaware of technical distinctions in the ratings world. Second, it creates counterproductive arbitrage opportunities.

If an insurer is rated AAA on a more lenient scale than insurable entities in another asset class, the insurer can profitably "sell" its AAA rating to those entities without creating any real value in the process.

Municipal bond insurance is a great example. Monoline bond insurers like AAA-rated Ambac, FGIC and MBIA insured bonds issued by states, cities, counties and other municipal issuers for three decades prior to the 2008 financial crisis. In some cases, the entities paying for insurance were of a stronger credit quality than the insurers. As it happened, the insurers often failed while the issuers survived, leaving one to wonder why the insurance was necessary.

During this period, general obligation bonds had very low overall default rates. According to Kroll Bond Rating Agency’s Municipal Default Study, estimated annual municipal bond default rates by issuer count have been consistently below 0.4% since 1941. Similar findings for the period 1970-2010 are reported in The Bloomberg Visual Guide to Municipal Bonds by Robert Doty. This 0.4% annual rate applies to all municipal debt issues, including unrated issues and revenue bonds. The annual default rate for rated, general obligation bonds is less than 0.1%.

Given this long period of excellent performance, one might reasonably expect that most states and other large municipal issuers with diversified revenue bases to be rated AAA. No state has defaulted on its general obligation issues since 1933, and most have relatively low debt burdens when compared to their tax base. Despite these facts, the modal rating for states is typically AA/Aa with several in the A range. (This remains the case despite certain rating agencies’ claims that they have recently scaled up their municipal bond ratings to place them on a par with corporate ratings).

The depressed ratings created an opportunity for municipal bond insurers to sell policies to states that did not really need them. For example, the State of California paid $102 million for municipal bond insurance between 2003 and 2007. Negative publicity notwithstanding, the facts are that single A rated California has a Debt to Gross State Product ratio of 5% (in contrast to a 70% Debt/GDP ratio for the federal government) and that interest costs represent less than 5% of the state’s overall expenditures. While pension costs are a concern, they are unlikely to consume more than 12.5% of the state’s budget over the long term – not nearly enough to crowd out debt service.

California provides but one example. The Connecticut lawsuit mentioned above also cited unnecessary bond insurance payments on the part of cities, towns, school districts, and sewer and water districts.

Meanwhile, AAA-rated municipal bond insurers carried substantial risks, evident to many not working at rating agencies. For example, Bill Ackman found in 2002 that MBIA was 139 times leveraged. As reported in Christine Richard’s book Confidence Game, Ackman repeatedly shared his research with rating agencies – to no avail.

This imbalance between the ratings of risky bond insurers and those of relatively safe municipal issuers essentially created the monoline insurance business – a business that largely disappeared with the mass bankruptcy and downgrading of insurers during the 2008 crisis.

Inconsistent ratings across asset classes thus do have real world costs. In the US, taxpayers across the country paid billions of dollars over three decades for unneeded bond insurance. Individual municipal bond investors, often directed by their advisors to focus on AAA securities only, missed opportunities to invest in tens of thousands of bonds that should credibly have carried AAA ratings, but were depressed by the raters’ inopportune choice of scale.

We believe that one reason for the persistent imbalance between municipal, corporate and structured ratings is the dearth of analytics directed at government securities. Rating agencies and analytic firms offer models (and attendant data sets) that estimate default probabilities and expected losses for corporate and structured bonds. Such tools are relatively rare for government bonds. Consequently, the market lacks independent, quantitatively-based analytics that compute credit risks for these instruments. This lack of alternative, rigorously researched opinions allows the incorrect rating of US municipal bonds to continue, without the alleviation of a positive feedback loop.

Next month, PF2 will do its part to address this gap in the marketplace with the release of a free, open source Public Sector Credit Framework, designed to enable users to estimate government default probabilities through the use of a multi-period budget simulation. The framework allows a wide range of parameterizations, so you may find it useful even if you disagree with the characterization of municipal bond risk offered above. If you wish to participate in beta testing or learn more about this technology please contact us at info@pf2se.com, or call +1 212-797-0215.

--------------------------------------------
Contributed by PF2 consultant Marc Joffe. Marc previously researched and co-authored Kroll Bond Rating Agency’s Municipal Default Study.