Risk Manage System for AI Safety and Alignment
Risk Management System is an integral part of a Quality Management System (QMS) commonly used and required in Medical Device, Healthcare industry. However, guidelines in developing and maintaining a risk management system are not limited to just medical product, they can be used in other jurisdictions as well. For Large Language Model (LLM) based products, given the paradoxical combination of predictable component manifested in the “Scaling Laws” as well as the unpredictable capabilities and output, the trade-off between the beneficial impact of LLM and the residual risks, negative society impacts can be managed and regulated by such a risk management system.
Definition of Risk
It is generally accepted that the concept of risk has two key components: the probability of occurrence of harm and the consequences of this particular harm, that is, how severe it might be. Putting this under the context of LLM deployment, risks such as misinformation, disinformation, bias, discrimination, breaches of privacy, mis-use for malicious intentions, uncontrolled proliferation, malicious replications, so on and so forth, as well as their associated probabilities need to be analyzed using approaches including expert elicitation and survey, attack scenario or task design and modeling, data-driven metrics report on past incidents, etc. Utilizing a well established risk assessment and management framework can help structure the estimation process and ensure that all relevant factors are considered. For example, frameworks such as Failure Modes and Effects Analysis (FMEA), Probabilistic Risk Assessment (PRA), Monte Carlo simulation methods can all be of great help to identify, evaluate, and mitigate potential failures and adverse outcomes, ensuring safer and more reliable AI deployments.
Risk Management Process
All LLM practitioners, providers shall establish, implement, document and maintain an ongoing risk management process that identifies hazards, hazardous situations associated with a LLM deployment, estimates and evaluates the risk level for each hazard, controls the various risks, monitors the effectiveness of the planned risk control measures. This process shall be incorporated in all existing, LLM based, product realization and documentations throughout the life cycle of the LLM. A typical risk management process is shown below where individual elements can have varying emphasis depending on the specific phase of the LLM development and deployment.
Risk Analysis
-
Identify hazards and hazardous situations associated with a model
-
Identify characteristics related to safety
-
Document Intended use of the deployed LLM and reasonably foreseeable mis-use
Risk Evaluation
-
Use the criteria for risk acceptability defined in risk management plan
-
For each hazardous situation, the LLM practitioners shall evaluate the estimated risks and determine the acceptability
-
Result of risk evaluation shall be recorded in risk management file
Risk Controls
-
The LLM practitioner shall determine risk control measures that are appropriate for reducing risks to an acceptance level
-
The implementation of each risk control measure shall be verified and verification shall be recorded in the risk management file.
How to Build Risk Management on Large Language Models
Risk Management System provides a framework within which experience, insights and judgement can be applied in a systematic manner to manage the risks associated with the use of Large Generative Models. Residual risk evaluation, benefit-risk analysis should always be performed for LLM based product before deployment. There can be particular hazardous situations for which the risk exceeds the LLM practitioner’s criteria for risk acceptability. This scenario enables the practitioner to provide a high-risk LLM based product for which they have done a careful evaluation and can show that the benefits of the product outweighs the risk, especially for use cases in healthcare settings. However this cannot be used to weigh residual risks against economic advantages or business advantages such as business decision making.
There quite a few benchmarks that has been used to evaluate different aspects of LLM such as multitask language understanding (MMLU), natural text generation (BLEU), text summarization (ROGUE)
Case Studies
A Title to Turn the Visitor Into a Lead
FAQ
What is the definition of “Risk”?
Risk is defined by two components: the probability or likelihood for a hazardous situation to occur and the severity level of that particular harm