5.3 Calibrating the Rating Model

Calibration assigns a default probability estimate to each possible overall score. According to European Directive 2006/48/EC, the rating scale used by the credit institution to group debtors in classes with a reasonably small band of PD values. The directive allows the use of "direct estimates of risk parameters [which] may be seen as the outputs of grades on a continuous rating scale". (Directive 2006/48/EC, Annex VII, Part 4, 4)

There two distinct calibration procedures to be used, depending on the methodology used to build the scoring function.

• Logistic regression: already yields sample-dependent PD estimates, which need to be rescaled to each segment's average PD.
• Statistical and heuristic models (e.g. MDA): calibration assigns PD values to scores; rescaling may be necessary.

 Unless a full data survey is used to generate the data set, external data is necessary to calibrate the rating system. Specifically, for all segments it is necessary to know the a priori PD (the average default rate).

5.2 Developing the scoring function

The statistical analysis leading to the development of the scoring function occurs in two phases:

• Univariate analysis
• Multivariate analysis

Univariate analysis

Building a catalog of indicators

1. In the first step, the quantitative data items are combined to form indicator components which are meaningful in business and enable economic analysis. E.g. gross profit, EBITDA, EBIT, working capital...
   
2. Definition of relative indicators: constructional figures, relative figures, and index figures. E.g. gross margin, EBITDA margin, EBIT margin, acid test, current ratio...
   
3. Working hypothesis: Good > Bad or Bad > Good. The hypothesis has to be monotonic in order to apply either MDA or logistic regression. Non monotonic indicators need to be transformed in PDs to be used monotonically in the analysis.

Analyzing indicators for hypothesis violations
Two possible alternatives.

• Measure of discriminatory power: if positive, the hypothesis holds; if negative, the hypothesis must be rejected and the indicator cannot be used in the subsequent phases.
• Compare medians: reviewing of whether the indicator's median values differ significantly for the good and bad groups of cases and correspond to the working
  hypothesis.
  

5.1 Data Set Quality

The data collection process is central in establishing the quality of the data set and, hence, of the statistical rating model.

Data requirements and sources

The data collection process must include all data categories relevant to creditworthiness in the debtor segment to be studied (see 2. Best-Practice Data Requirements for Credit Assessment). It is necessary to

• specify the data to be collected more precisely on the basis of the defined data categories.
• quality assurance requirements for quantitative, qualitative and external data.

Quantitative data

Annual financial statements are usually standardized by commercial law. This makes them reliable indicators of a company's financial success. However, care must be taken when comparing data from companies abiding by different accounting standards (IAS/IFRS vs. domestic GAAP vs. foreign GAAP).

Other relevant quantitative data might not be available in standardized form (e.g. income and expense accounts, information from borrowers on assets/liabilities). Collecting such information meaningfully can be challenging.

5. Developing a Statistical Rating Model

Whereas heuristic and causal models allow considerable degrees of freedom in the development phase, the success of a statistical models depends heavily on the execution of the development stage, which must be carried out according to the following well-defined best practices.

1. Extraction of the sample data set
2. Development of the scoring function
3. Calibration of the the score values
4. Qualitative validation
5. Quantitative validation

(Chart 27: Procedure for Developing a Rating Model, from Rating Models and Validation: Guidelines on Credit Risk Management, Oesterreichische Nationalbank, 2004)

4. Model Suitability

Essential Requirements

Essential requirements for a credit assessment procedure:

1. PD: The target value of the model must be the probability of default (PD) of the borrower.
2. Completeness: The model must take into account all available information relevant to creditworthiness.
3. Objectivity: The results of the credit assessment procedure must be reproducible by different analysts working with the same initial data set.
4. Acceptance: In the eyes of the user, the credit rating model should assess the borrower's creditworthiness as accurately as possible.
5. Consistency: The model must not contradict accepted economic theories and methods.

Suitability of Individual Model Types

Heuristic methods
Statistical models yield a higher discriminatory power than heuristic methods. The former are generally preferrable when a sufficient data set is available. While the latter can be used, their discriminatory power and forecasting accuracy must be reviewed during validation.

Statistical methods
These methods require a sufficiently large data set for the development of the model, especially with regard to default cases. Not always is such a data set available for all borrower segments. Typical segments where these methods cannot be effectively applied are the following:

• Governments and public agencies
• Banks and other financial services providers
• Large corporate/public companies
• Project finance and other specialized finance

Multivariate Discriminant Analysis
MDA imposes the following requirements on input data:

• quantitative
• normally distributed
• same variance/covariance matrix for default and non-default groups

If these requirements are not met the method will not attain the maximum discriminatory power. If this method is used anyway, during validation the bank should review the effects of the unmet requirements on the discriminatory power and forecasting accuracy of the resulting model.

Regression models
Can be applied in all rating segments.

Neural networks
Can be applied in all rating segments.

3 Credit Assessment Models

Main categories of credit assessment models:

1. Heuristic models
2. Statistical models
3. Causal models

Heuristic models:
Heuristic models aim at documenting and leveraging previous experience shedding insight on the creditworthiness of a customer. The bases of this experience are the following:

• subjective experience and observations
• conjectured business interrelationships
• business theories

The main kinds of heuristic models are the following:

1. Rating questionnaire
2. Qualitative systems (e.g. BVR-I of the Federal Association of German Cooperative Banks)
3. Expert systems (e.g. CODEX, Commerzbank Debitoren Experten System)

Statistical models:
Statistical models aim at verifying hypotheses concerning potential creditworthiness criteria to using statistical methods on a sample of debtors representing a segment of the banks customers or potential customers. These hypotheses are statements as to whether on average a given recorded variable can be expected to be higher or lower for insolvent borrowers than for solvent borrowers.

The ability of a statistical model to correctly identify solvent and insolvent borrowers depends on the following:

• availability of a large enough data set to make statistically relevant inference,
• representativeness of the sample data set in relation to the target population.

The main kinds of statistical models are the following:

• Multivariate discriminant analysis (MDA)
• Regression models

2. Best-Practice Data Requirements for Credit Assessment

Types of Data

• Quantitative data: metric values
  
  1. Past/present: actual recorded values
  2. Future: forecasts (eg. cash flow, budget)
• Qualitative data: ordinal values
• External data:
  
  1. Public agencies (e.g. statistics offices providing macroeconomic data)
  2. Commercial data providers
  3. Other data sources (e.g. market prices)

1. Defining Segments for Credit Risk Assessment

Segment your borrowers and analyze each segment separately.

1. The factors relevant to creditworthiness depend on the type of borrower.
2. The available data source depend on the type of borrower.
3. The risk level depends on the type of borrower.

Business considerations lead to defining the following segments:

1. Government
2. Financial services providers
3. Corporate customers
4. Retail customers

Basel II defines the following categories of assets under the IRB approach:

1. Sovereigns and central governments
2. Banks and financial institutions
3. Corporate loans
4. Retail loans
5. Equity

Corporate loans/customers are split into the following sub-categories:

• Specialized lending
• Enterprises and business owners

Rating Models and Validation

In this series of posts you will find a summary of key concepts and guidelines on the development and validation of statistical credit rating models documented by the Oesterreichische Nationalbank in an omonymous publication.

Let me introduce myself

Let me introduce myself. I am a free lance IT and electrical Engineer. Although engineering is in my DNA, I'm quite tired of wiring diagrams and MOSFET amplifiers. I have taken interest in finance and banking, and am planning to steer my career in this direction. I am working on a Master's Degree in Corporate Finance and Banking at the SDA Bocconi School of Management in Milano, and will graduate in April 2010 Meanwhile, I will use this blog to jot down key concepts, ideas, news and whatnot that come to me around the subject of my current studies.

If you share the same interest in finance, read on: you might find something interesting.

And, by the way, you can find my resumé here.