The Pros and Cons of Full Stack Risk Management

Motivation

An Open Risk Academy user recently asked if there is a real-world, end-to-end open-source project (in particular, written in the popular data science language Python) that walks through the entire (risk model) workflow: namely, collecting data, engineering features, building a risk model, iteratively optimizing it, and drawing actionable conclusions. It is a very good question. For many people learning theory in the context of real-world practice is essential and such an end-to-end example obviously helps connect all the dots of data driven risk management. But like with all good questions, the answer is not simple. In this blog post we explore some of the issues involved in a broader context.

The title of the blog post: the pros and cons of full stack risk management alludes to how we are going to tackle this: We are imaging professional risk management within an organization (could be a company or any other entity) that aims to organize around full stack professionals and tools, that is, people that are intimately familiar with the entire set of processes, and have “end-to-end”, multidisciplinary capabilities.

The prototype for such an organizational / team arrangement comes from a different and distinct domain, namely digital technology and more specifically the development of software tools and applications. Yet as the operations of all types of organizations rapidly digitize (and risk management can be on occasion be a very information-driven pursuit) it is not surprising that there these notions acquire relevance. This is related to the question of how to distribute different information technology skills, knowledge and responsibility within various teams. We will look into this into more detail in a credit risk context. But let us first examine how the notion of full stack came about in the technology sector.

The Full Stack Developer

The circumstances that necessitated the concept of a “full stack” developer in software development are quite interesting. Full Stack refers to the complete set of technologies and skills needed to develop both the front end and back end of modern networked (web) applications.

Front End is the part of a computer application that typical end-users interact with. It includes visual and functional design elements and determines the overall user experience (or UX) keeping in mind that end-users may have very different technical skills. In modern web applications the program used by the end-user is typically the web browser (also termed a client, not to be confused with other clients we will meet later).

Back End is, in contrast, the server-side business logic and potentially the database interactions that support the front end functionality. These typically run in the “cloud”, also known as somebody else’s Linux computer.

This orchestration of interaction between the front and the back of an application is not new. It is a staple feature of so-called client-server model of computing. It became an important consideration with the massive expansion of the internet and web applications. The widespread adoption highlighted the different needs and skills of front-end applications serving end-users versus backend applications that process business logic, store data etc.

The distinction between “front” and “back” is slightly less visible - but still there - in any traditional (desktop) applications that run on a single computer. These are typically written in a single programming language, feature a so-called GUI (graphical user interface) and associated logic that executes while the application runs.

In the early phases of the web the distinction between front and back-end was not stark, primarily because front-end programs where not very sophisticated. But over time specialization and increased complexity on the front-end created two essentially distinct technical development environments, where individuals working on one side would have very limited knowledge and ability to contribute on the other side of the client/server divide.

Yet this split and separation of duties was not without negative consequences and as a result various trends aim to reconnect the two realms. It is in this context where the context of a full-stack developer emerged. Crudely speaking, this is someone who straddles the predominant specialization paradigm and can work on both layers of a web application, from user-facing components to the server and database management.

Why would one want to do this? There are a number of advantages and disadvantages:

• When working across the two application layers (frontend, backend) one obviously has a more holistic understanding of the software architecture. In turn this means that functionality is implemented where it is more natural and efficient, problems can be tackled sooner or more comprehensively etc.
• One can switch easier between different development tasks as needed, which provides a more dynamic and flexible work experience.
• It involves reduced communication overhead as there are fewer distinct teams to collaborate and, of course, reduced costs if the organization employs fewer developers (although a full stack developer might command higher salary).
• On the other hand, full stack development requires mastering multiple technologies and programming languages. This can lead to breadth rather than depth of expertise (or the jack of all trades, master of none phenomenon). The list of required technologies in job vacancies, sometimes absurdly lengthy, is the source of perennial online fun.
• Finally the mental effort required from the broader responsibility menu is significant. Hence a higher potential for burnout, especially in times of organizational stress.

Needless to say, there is an endless discussion in technical circles about the merits and demerits of full stack development, which is not surprising as there are many different factors at play, and the outcomes may depend on the organizational context.

Full Stack Risk Management?

All right, we got a first glimpse of the conditions that generated the full stack role in technology. How do these considerations translate into the risk management domain?

For concreteness we will focus on a specific sector (finance). Within the financial sector (banking and similar entities) there is a well established layering or grouping of different business functions which is frequently reflected in organizational diagrams:

• Business units or front office roles (client facing staff, sales, origination, execution of contracts etc.), typically considered the revenue/profit generator (or PnL owner) of the firm.
• Support units or middle and back office (formal risk management teams, legal, compliance, IT infrastructure, accounting etc.). These are typically considered as cost centers and are being grouped in various combinations.

The essence of an organization’s risk management actions will in practice spread across different units reflecting that the entire organization is ultimately responsible. A popular framework to recognize this reality is the so-called three lines of defense model, or 3LOD. It is a Risk Framework that divides responsibilities for managing the risks faced by an organization into three distinct roles:

• the first line, which consists of business managers (front office personnel), who own (effectively generate in the course of business) and hence are firstly responsible for managing these risks
• the second line, which includes a so-called Independent Risk Management unit (simply Risk unit) and potentially other compliance functions. This is a specialized unit, skilled in various forms of risk management that oversees and supports the first line.
• the third line comprises internal and external auditors. Their role is to provide independent assurance on the effectiveness of the first two lines.

We see right away that in institutions with mature risk management arrangements (which may also be subject to stringent regulations) the concept of end-to-end risk management responsibility vested to single individuals / teams is not really applicable. There is formal separation of duties that helps organizations ensure accountability. It also recognizes that (especially large) entities face a multitude of complex risks, some external but also some internal. Hence multiple checks-and-balances are required and this automatically creates compartmentalized responsibilities.

Hence our starting point is the that the Risk unit is akin to a specialized backend developer, busy with data and algorithms but also interfacing with various other units, some of which are client facing. As we have seen above while discussing full stack development, there are some significant advantages when individuals have a holistic overview across the stack.

The equivalent of a developer being familiar with, and operating across both front and backend would be for Risk unit members to have capabilities, responsibilities and interactions with other organizational units beyond narrow technical skills.

Translated into risk management context the pros and cons of such an arrangement would read something like:

• More holistic understanding of the risks facing an organization. Risk mitigation happens where it is more natural and efficient, problems can be tackled sooner or more comprehensively etc.
• Risk managers have a more meaningful work experience which helps with better performance.
• Reduced costs, which, ceteris-paribus, help strengthen the organization.
• But also, risk of breadth rather than depth of risk management expertise which can expose the organization.
• Finally higher potential for burnout, especially in times of organizational stress.

Can the above pros and cons be reconciled, while in addition respecting the strong requirements for checks and balances? This is of course a very open ended question: the organizational context and the nature of the risk play significant role. Here we will only scratch the surface, by focusing on a specific process: the development and use of a credit risk model within a financial institution.

In a narrower context, one that is particularly intensive in data and algorithms, we can explore the match between benefits and challenges of full stack risk management. Concretely, the use case for this exercise is the fairly standard procedure for developing and using credit risk algorithms, such as scorecards. The linked Open Risk Manual article provides an overview of all the stages involved in designing, developing and deploying a Credit Scorecard in a business context. Here we will examine them from the perspective of how responsibilities can be partitioned between different individuals / units.

Partitioning the Risk Model Lifecycle

Credit Scorecard development activities can be grouped in six broad stages. Depending on the business and regulatory context some stages might be more extensive than others or potentially bundled as one stage. NB: These stages are not specific to credit scorecards but are common across a wider range of statistical and machine learning risk models.

Business Requirements

This stage defines the scope and objectives of the credit scorecard implementation and operation. As implied by the title, the first line (business) produces these requirements and the risk unit (model developer) aims to fulfill them. Business requirements crystallize diverse aspects of model development and usage, such as:

• The business use of the scorecard, e.g. for Risk Acceptance of new clients on the basis of, e.g., a Credit Score, Behavioral Scoring of existing clients, calculating Regulatory/Risk Capital etc.
• The applicable client population (Portfolio Segment) to which the credit scorecard will be applied.
• The type of Credit Risk / Credit Event (or events) that the scorecard will help to manage (e.g. the identification of a bad loan, the applicable Risk Horizon).
• The nature of the Credit Score produced, including whether and how it is linked to a Credit Rating System, a Probability of Default and related Risk Parameters.
• Any constraints on the available resources (data, algorithms, computational infrastructure) that can be used for the development and deployment of the scorecard.
• The desired performance characteristics for accepting the deployment and/or continuing use of the scorecard.

A full stack risk management setup must recognize the fact that the Risk unit does not own the business (for example it can not initiate a new business line in need of credit risk models). Yet in many of these above aspects the Risk unit can play a proactive role, e.g. by defining best practice standards the business must consult when formulating requirements.

Model Development

This stage captures the main technical activities of risk model development. While the role of the Risk unit is here dominant, it does not operate in a vacuum and there are important interfaces to consider. In particular the diversity of possible data sources (e.g. from production systems operated by the business, external data etc.) means a reliance on business or external parties for critical information. At high level the activities involved here are:

• Data Collection
• Data Cleansing
• Exploratory Data Analysis / Risk Data Review
• Risk Model Development
• Expert Analysis
• Risk Model Documentation

A good example where the full stack mentality pays divided are the Data Cleansing activities. Intimate knowledge of business systems and potential Data Quality issues is essential to ensure that the model development is not biased.

Model Validation

The Model Validation stage provides a more or less formal review of the Stage 2 developed scorecard. This stage is a good example of purposeful separation (and to some degree duplication) of duties for the purpose of reducing risk. Namely most if not all of the concrete quantitative techniques used in model validation are (or can) be used during model development.

Here we have a very concrete example where the validation must be performed by an independent party. The purpose of a separate model validation process is to reduce the possibility of accidental error or poor practice. For large entities there is also assurance of comparable standards across different business units.

Model Deployment

After a model has been developed and successfully validated and approved the Model Deployment stage deploys the model in production. This stage includes a range of tasks that may or may not be under the remit of the Risk unit. The outcome is one or more operating (live) model instances that are processing actual client data, and might further be embedded in Credit Rating Systems and used in related risk and portfolio management processes. Activities at this stage include:

• Production Implementation (if the model has been developed in dedicated, non-production oriented systems)
• Acceptance Testing, Parallel Runs and other such arrangements
• User Training and
• Supporting the ongoing operation of the scorecard.

While under some circumstances the platform for model development may also enable model deployment, in general this will not be the case. This might be for a combination of reasons such as security and performance. Other tasks such as supporting model operations might be also be far removed from the expertise of the Risk unit.

Model Monitoring

For any deployed model, scorecard performance must be monitored throughout its active (operating) life to identify potential pathologies such as Model Decay due to a number of different possible causes. To address this typically there will be a frequent Model Monitoring Report or equivalent that captures the essential performance indicators (including also historical development).

Here we have another example where the close involvement of the Risk unit in the design and production of such reporting can increase its insight and overview of the risk landscape.

Model Adjustment or Decommissioning

Finally, Model Adjustment or Decommissioning are potential end-stages in the model life cycle. If the monitoring report or other relevant information suggests that the model does no longer meet requirements, the scorecard might need to be redeveloped using e.g., additional data or maybe decommissioned altogether. In case of re-estimation / redevelopment the six stages sketched above must repeated but now on the basis of the pre-existing implementation.

Conclusion

We saw that the separation of duties that is frequently an institutional requirement limits the extend to which individuals with risk management tasks can be “full stack”, having skills, oversight and responsibility across organizational boundaries. Yet the development and active use of quantitative risk models such as credit scorecards has many potential pitfalls and individuals with rounded capabilities that can bridge organizational silos can have significant impact:

• Executive (Senior Management) buy-in and understanding of key issues around algorithms and models may be limited and ability to converse on their interplay with business objectives is valuable.
• Model development is resource intensive (time, money, expertise, project management support etc.) and optimal use of organizational resources help secure the right outcomes.
• Model quality may suffer from data scarcity or other data quality issues and deep understanding of the data production processes and monitoring can help mitigate this risk.

The notion of a full stack risk manager is not tenable given the many internal and external constraints of typical institutions. But this is not the end of the story. As risk management (and in particular financial risk management) keeps getting more digital, the concept of holistic overview of information flows and the growing role of algorithms requires individuals with a broader set of skills, indeed a better handle on the entire "stack"!