5.1 Data Set Quality

The data collection process is central in establishing the quality of the data set and, hence, of the statistical rating model.

Data requirements and sources

The data collection process must include all data categories relevant to creditworthiness in the debtor segment to be studied (see 2. Best-Practice Data Requirements for Credit Assessment). It is necessary to

• specify the data to be collected more precisely on the basis of the defined data categories.
• quality assurance requirements for quantitative, qualitative and external data.

Quantitative data

Annual financial statements are usually standardized by commercial law. This makes them reliable indicators of a company's financial success. However, care must be taken when comparing data from companies abiding by different accounting standards (IAS/IFRS vs. domestic GAAP vs. foreign GAAP).

Other relevant quantitative data might not be available in standardized form (e.g. income and expense accounts, information from borrowers on assets/liabilities). Collecting such information meaningfully can be challenging.

Qualitative questions

Care must be taken to restrict the subjectivity of the analyst. Ideally, the questionnaire should be unambiguous and unmistakable, to the point that different analyst should be able to generate the same results.

External data

Various external data sources can be taken into consideration when developing a credit risk model, e.g. external ratings, credit reporting information, capital market information. It is necessary to monitor the quality, objectivity and credibility of the data source at all times. The availability of such external data must also be assessed.

International rating models

In order to develop a rating model applicable internationally it is necessary to constrain international discrepancies in

• the definition of the default event
• the GAAP of the countries within the scope of the model.
  

Data collection and cleansing

In order to ensure the statistical significance of the model to be developed, it is necessary to gather data regarding a sufficiently great number both of non defaulters and of defaulters. As defaults are generally rare, gathering enough data on defaulters is usually the main challenge in the data collection process.

Definition of default

Basel II defines the default event as follows. Either

• the credit institution considers that the obligor is unlikely to pay its credit obligations
  to the credit institution, the parent undertaking or any of its subsidiaries
  in full, without recourse by the credit institution to actions such as realising
  security (if held), or
  
• the obligor is past due more than 90 days on any material credit obligation to
  the credit institution, the parent undertaking or any of its subsidiaries. Overdrafts
  shall be considered as being past due once the customer has breached an
  advised limit or been advised of a limit smaller than current outstandings.

Data set generation strategies

• Full survey: most effective, most expensive. Uses all available historical data. Guarantees that the data set is representative of the credit institutions portfolio.
  
• Sampling: less effective, less expensive. Uses a random sample of records. It can be the alternative of choice when some of the relevant data (typically qualitative) is not stored in digital form in the bank's IT system but must be collected in paper form. The
  
• Data pooling: various banks pool their data to generate a wider data set. This can solve the problem of gathering enough data on defaulters. The back side is that the data set is less representative of each banks specific portfolio.

Representativeness

The sample must be built so that it is representative of the basic population (i.e. the set of all transactions in a given rating segment for the credit institution or credit institutions contributing to the data pool). This means that in each rating segment (corporate, retail, etc.) the distribution of the sample must match as closely as possible the distribution of the basic population with respect to any relevant structural characteristic (e.g. for the corporate segment the distribution across regions, industries, size classes, and legal forms of business organization; for the retail segment the distribution across regions, professional groups, and age).

Data quality assurance

Automated plausibility tests:

• Does the borrower entered belong to the relevant rating segment?
• Were the structural characteristics for verifying representativity entered?
• Was the data history created according to its original conception? 
• Have all required data fields been entered? 
• Are the data entered correctly? (including a review of defined relationships between positions in annual financial statements, e.g. assets = liabilities)

The results are recorded in a log. Any errors found must be corrected before proceeding.

Integrity checks:

• Were balance sheets entered which cannot be assigned to an existing borrower?
• Is there qualitative information which cannot be assigned to a cutoff date?
• Have different borrowers been entered with the same borrower number?
• Have different banks been entered under the same bank identifier code?
• Have any banks been entered for which no borrowers exist?

Records containing errors must be either deleted or, preferably, corrected.

Definition of the sample

The data gathered in the previous phase must serve both the purpose of developing the scoring function and validating it. In order to make the validation procedure meaningful, it is necessary to assess the performance of the scoring function on data other than that used for development. Hence, the data set is usually divided in two distinct subsets:

• the analysis sample
• the validation sample or hold-out sample

The analysis and validation samples thus have to be disjunct with regard to borrowers. In order to avoid bias, the subdivision of the sample should be done randomly; however, it is necessary to verify that the analysis sample is representative of the basic population (see Representativeness)

Criteria for weighting of good and bad cases in the analysis sample

The analysis sample can be created in such a way that the proportion of bad cases is representative of the rating segment to be analyzed.

Advantages:

• Easier model calibration
• Logistic regression values can be used directly as PD values

Disadvantages:

• Default cases might be too few in the analysis sample to allow the development of a statistically meaningful scoring function

Otherwise the analysis sample can be built with a higher proportion of default cases. This is especially necessary when the total number of cases which can be collected is limited. In practice it is necessary to have between one fourth and one third of defaulters in the analysis sample.

Advantages:

• More meaningful identification of differences between defaulters and non defaulters.

Disadvantages:

• Need for calibration and rescaling of default probabilities

Bootstrap method
If the available data is scarce, splitting it between the analysis and the validation subsamples might not leave enough data points for a statistically relevant analysis. The bootstrap method can be a viable alternative here.

• The scoring function is built on the entire sample.
• Validation is carried out by comparing the overall scoring function with several scoring functions built on random subdivisions of the original sample in an analysis and a validation subsample.

If the variation of the coefficients of the factors in the scoring function is small, the overall scoring function is acceptable. On the other hand, if the variations are large, and especially if the are sign reversals of the factor coefficients in the various subsample scoring functions, then there is a clear indication that the available data is too scarce to allow the development of a meaningful scoring function.

The discriminatory power of the overall sample is assessed by taking the mean and fluctuation margin (i.e. standard deviation) of the discriminatory power of each subsample scoring function; here, the hold out sample of each function is used to measure the discriminatory power. The higher the mean discriminatory power and the lower the standard deviation, the better the model.