Key Considerations for Validation of AI/Machine Learning Models In the AML Space

How are AI/ML Solutions Reshaping the Landscape in the Fight Against Financial Crime?

It is no secret that leveraging AI and Machine Learning (ML) models provides a wealth of benefits to financial institutions (FIs) in their fight against financial crime. In fact, 61% of FIs surveyed in our The Evolving Role of Machine Learning in Fighting Financial Crime study reported that the implementation of AI/ML solutions helped reduce risk.

Beyond general risk reduction, FIs often look to AI/ML solutions to specifically improve the detection and prevention of financial crime, particularly in the Anti-Money Laundering (AML) space. AML monitoring has historically been plagued by ineffective and/or inefficient transaction monitoring (TM) rules, compounded by increasingly sophisticated money laundering schemes. While AI/ML models can bolster financial crime detection and prevention efforts, institutions must ensure that they are built and validated correctly. Incorrectly or incompletely validated models expose companies to the risk of incorrect detection, inconsistent model performance, gaps in monitoring programs, and/or failure to meet regulator expectations. This may help to explain why only 51% of surveyed respondents indicated achieving efficiency gains through the implementation of AI/ML.

“Traditional” vs. AI/ML Model Validation: What’s the Difference?

Due to the added complexity over “traditional” AML transaction monitoring rules, AI/ML models require specific expertise during both the model development and validation stages. The addition of these types of models to a FI’s monitoring environment requires internal model validation and internal audit oversight. Unfortunately, many financial institutions are not adequately equipped in these areas, leaving them without the correct personnel that are familiar enough with AI/ML modeling in the AML space to develop and execute AI/ML-specific validations and perform internal audits.

While there are currently no AI/ML-specific regulatory requirements, these models will likely soon receive closer regulatory scrutiny. Historically, most model validation approaches have used the Federal Reserve’s SR Letter 11-72 as their basis, which outlines three key components of model validation:

1. Evaluation of conceptual soundness, including developmental evidence
2. Ongoing monitoring, including process verification and benchmarking
3. Outcome[s] analysis, including back-testing

During “traditional” model validation, each of these components receives roughly equal attention, as most rules have straightforward logic and code that lends itself to simple use-case evaluation and easy independent replication of both input and output. However, because of the statistical complexity, dearth of empirical data, and inherent randomness in AI/ML models, replication of code or alert/score output is typically not a valuable exercise. One of the key benefits of using AI/ML, in fact, is the ability to develop models based on average results over dozens or hundreds of iterations, something that is not feasible or even desirable to reproduce as part of validation.

Improving Model Validation Strategy for AI/ML

Most AI/ML models use well-tested and generally accepted statistical algorithms as their foundation, providing some inherent level of assurance in the model’s output, but only when the algorithm is appropriate for the identified use case and the inputs are correctly configured. AI/ML model validation should, therefore, place more emphasis on the first component listed above: conceptual soundness.

In particular, the feature engineering (the selection, transformation, and testing of included variables) and model tuning processes should receive substantially more attention than is usually given in validation of “traditional” rules. Validation of these areas of AI/ML models should, at minimum:

• Confirm that the selected algorithm(s) is/ are appropriate for the desired task. 
• Verify that selected variables are relevant from both a statistical and business perspective.
• Determine if variable transformations are logical, based on the specific type(s) of financial crime the model was built to address.
• Ensure the approach to tuning both parameters (variable-level settings) and hyperparameters (model-level settings) is both sensible and thorough.
• Review documentation for inclusion of all model development steps, including model design decisions, testing scope, final configurations, and results.

Backstopping sound design and development, FIs should also have clear model governance and controls in place to ensure that AI/ML models are being correctly reviewed/approved, deployed, monitored, and updated in a consistent, comprehensive manner. Model validation in this area should cover topics such as sampling and/or below-the-line analyses for events cleared by the model, performance monitoring to identify shifts in underlying data or model drift over time, and model review triggers (e.g., for ML alert scoring models, the tuning or calibration of underlying TM rules).

While there is no silver bullet in the fight against financial crime, FIs must continue to adapt their strategies to ensure their AML programs are both effective and efficient. As institutions look to AI/ML to solve existing issues or improve their monitoring environments, they must ensure that their model validation approaches similarly evolve, giving assurance that solutions are developed and implemented properly, adaptable to various changes over time, and provide maximum benefit.

While model validation is a key component of the overall monitoring environment, it is not the only consideration. Our next article in this series will discuss how AI/ML models should be developed to reflect the business or regulatory risks identified for coverage.

Authors: Tim Mueller, Sean McArdle, Nate Perry.