5.2 Developing the scoring function

The statistical analysis leading to the development of the scoring function occurs in two phases:

• Univariate analysis
• Multivariate analysis

Univariate analysis

Building a catalog of indicators

1. In the first step, the quantitative data items are combined to form indicator components which are meaningful in business and enable economic analysis. E.g. gross profit, EBITDA, EBIT, working capital...
   
2. Definition of relative indicators: constructional figures, relative figures, and index figures. E.g. gross margin, EBITDA margin, EBIT margin, acid test, current ratio...
   
3. Working hypothesis: Good > Bad or Bad > Good. The hypothesis has to be monotonic in order to apply either MDA or logistic regression. Non monotonic indicators need to be transformed in PDs to be used monotonically in the analysis.

Analyzing indicators for hypothesis violations
Two possible alternatives.

• Measure of discriminatory power: if positive, the hypothesis holds; if negative, the hypothesis must be rejected and the indicator cannot be used in the subsequent phases.
• Compare medians: reviewing of whether the indicator's median values differ significantly for the good and bad groups of cases and correspond to the working
  hypothesis.
  

Analyzing the Indicators' Availability and Dealing with Missing Values
Kinds of missing values

• The information necessary to calculate the indicator is not available.
• The indicator cannot be calculated because the denominator is zero in a division calculation.

Four alternative methods to handle missing values:

1. Cases in which an indicator cannot be calculated are excluded from the
   development sample
   Pros: simple
   Cons: if too many records are removed the data set can become empirically invalid.
2. Indicators which do not attain a minimum level of availability are excluded
   from further analyses.
   Pros: If at least 80% of the data points are available for a given record, the indicator can be used.
   Cons: So, how do I handle records with missing values? N.d.R.
3. Missing indicator values are included as a separate category in the analysis.
   Pros: No arbitrary substitution.
   Cons: Inapplicable to quantitative data.
4. Missing values are replaced with estimates specific to each group. A minimum level of availability must be attained anyhow (80%). The estimate must be made independently for every group (non defaulters, defaulters, validation sample). The median rather than the mean is used as the group estimate in order to make the process robust with respect to outliers.
   Pros: valid statistical procedure for quantitative indicators with some missing values
   Cons: not applicable to qualitative data

Analysis of Univariate Discriminatory Power
An indicator should only be used for the multivariate analysis phase if it has a univariate discriminatory power significantly greater than zero. (If it is zero or less it ought to have been eliminated during the working hypothesis review).

Only those cases which return valid indicator values — not those with missing or invalid values — should be used in the univariate discriminatory power analyses. What if I have substituted the group median for a missing value? Does that count as a valid case? N.d.R.

Transformation of Indicators
In order to transform non monotonic indicators into monotonic indicators, to confine outliers within a manageable range, and to make different indicators comparable by setting them against a uniform scale, it is possible to transform each indicator into a PD estimate. This can be done by dividing the range of the original indicator into disjunct subranges. The average default rate in each subrange provides an estimate of the average PD therein. Interpolation yields the PD values corresponding to indicator values other than the centroid of each subrange. If the indicator is monotonic, it is possible to interpolate the PD values using a logistic function--otherwise stated, one can apply a logistic regression to a monotinic indicator. This results in a function estimating the probability of default for any given value of the indicator.

 
I is the actual value of the indicator, while T(I) is the transformed value, i.e. the PD estimate. The parameters of the logisitic function u, l, a, and b must be found via non-linear regression.

Note that the working hypothesis review must be done before the transformation of the indicator, as the PD values obtained through the logistic regression are inherently monotonic and satisfy the hypothesis that PD(non-defaulters) < PD(defaulters).

Analyzing Indicator Correlations
Pairs of indicators exhibiting linear correlation coefficients greater than 0.3 should not be included in the development of the scoring function. Hierarchical cluster analysis is a statistical procedure which can be used to identify clusters of indicators such that correlation within the cluster is high; whereas, correlation between indicators of different clusters is low. From each cluster only the indicator with the highest univariate discriminatory power is included in the scoring function.

All indicators which survive the hypothesis violations review, the availability review and the discriminatory power review are tested for correlation via the hierarchical cluster analysis. The indicators which are selected at this phase are the ones used in the multivariate analysis will ultimately appear in the scoring function.

Multivariate analysis
General requirements for scoring functions:

• Objective indicator selection based on the empirical procedure
• Attainment of high discriminatory power
• As few indicators as possible, for the sake of model stability and to avoid overfitting the sample, thereby losing generality.
• Inclusion of as many different information categories as possible (e.g. assets situation, financial situation, income situation)
• Explicit selection or explicit exclusion of certain indicators in order to enhance or allow statements which are meaningful in business terms

The scoring function is built through a statistical regression procedure (Multiple Discriminant Analysis or logistic regression). Various scoring functions can be built at this time. One must be selected by applying the following criteria.

• Checking the signs of coefficients
• Discriminatory power of the scoring function
• Stability of discriminatory power
• Significance of individual coefficients
• Coverage of relevant information categories

Checking the Signs of Coefficients
If the scoring function represents creditworthiness, indicators for which the working  hypothesis is G > B should be should appear with a positive coefficient; whereas, indicators for which the working hypothesis is G < B should be entered with negative signs. It the scoring function is to be interpreted as the PD of the borrower, then the opposite sign convention applies. If all indicators have been transformed into PDs via a logistic function, then all coefficients in the scoring function should bear the same sign.
Discriminatory Power of the Scoring Function
Frequently measured using the Gini coefficient or the accuracy ratio values. Other measures of discriminatory power are possible.

Stability of Discriminatory Power
The discriminatory power of the scoring function can be expected to be high on the analysis sample, but will it be equally high on other observations? Stability can be viewed in the context of

• the hold-out sample, and
• longer forecasting horizons.

Scoring functions for which discriminatory power turns out to be substantially lower for the validation sample than for the analysis sample are less suitable for use in rating models because they fail when applied to unknown data. A scoring function should be rejected if the Gini coefficient differs by 10% or more between the analysis sample and the validation sample. Furthermore, suitable scoring functions should show sound discriminatory power for forecasting horizons of 12 months as well as longer periods.
Significance of Individual Coefficients
An F-test should be run on each indicator to verify the hypothesis that its coefficient in the scoring function is not equal to zero.

Coverage of Relevant Information Categories
The scoring function should provide a holistic assessment of the borrowers economic situation. Therefore, at least one indicator from each information category should be included. When a choice must be made among several possible candidate scoring functions, to favor user acceptance, the one which contains the most easily understandable indicators should be chosen.

Combining Partial Scoring Functions
When several partial scoring functions have been developed (e.g. quantitative and qualitative), they must be combined via a linear combination. The weights of the partial scoring functions within the overall scoring function can be obtained by one of the following methods:

1. Optimization using multivariate discriminant analysis or a regression model
2. Purely heuristic (expert) weighting of partial scoring functions
3. Combined form: Heuristic definition of weights based on statistical results

The first alternative provides the highest discriminatory power. Because user acceptance is favored by including expert judgement, the most effective solution can be the application of statistical regression methods to determine a set of acceptable coefficient vectors and hence of overall scoring functions, such that the discriminatory power within it is within an acceptably small range from the optimum; experts are then allowed to choose the overall function coefficients within this set. The advantages of this hybrid method are the follwoing:

• It makes the influence of quantitative and qualitative data on the credit rating transparent due to the use of linear weighting.
• It ensures high user acceptance due to the inclusion of expert opinions.
• It also ensures high discriminatory power due to the inclusion of statistical methods.