|

Computerized System Risk Assessment and Test Strategy

Computerized-system risk assessment determines what could fail, why the failure matters, which controls are needed, and what evidence is required before the system is released.

It should connect the system’s intended use and requirements with credible failure scenarios affecting patient safety, product quality, data integrity, or regulatory compliance. The result should be a focused test strategy—not merely a numerical risk score.

Purpose of Risk-Based Testing

Risk-based testing concentrates validation effort where failure could have the greatest effect or where control effectiveness is uncertain.

The assessment should help determine:

  • which requirements and functions require verification;
  • which failure conditions must be challenged;
  • the appropriate combination of scripted and unscripted testing;
  • the need for positive, negative, boundary, interface, and security testing;
  • what supplier evidence may be used;
  • the required regression-testing scope;
  • the evidence needed to demonstrate acceptable performance; and
  • who may accept the residual risk.

Risk-based testing does not mean testing only high-risk functions. Lower-risk functions may still require verification when they support intended use, provide a necessary control, or affect another critical function.

Begin With Intended Use

Risk assessment should begin with the approved intended use established during Computerized System Validation Planning and Strategy.

The intended use identifies:

  • the regulated business process;
  • the users and operating environment;
  • the functions relied upon;
  • the records created or maintained;
  • the calculations and decisions supported;
  • the connected systems;
  • the regulated requirements; and
  • important limitations or exclusions.

An assessment cannot reliably determine risk when the intended use is vague. For example, “the application manages laboratory data” does not identify which data, calculations, approvals, reports, or interfaces are relied upon.

A more useful statement identifies that the system receives analytical results, applies approved specifications, records review and approval, maintains audit trails, and transfers approved results to the batch-release process.

Connect Requirements, Functions, and Data

Risk assessment should be traceable to the approved User Requirements Specification and the relevant functional, design, and configuration specifications.

The assessment may evaluate:

  • an individual requirement;
  • a business function or workflow;
  • a calculation;
  • a data element or regulated record;
  • a configuration setting;
  • an interface;
  • a report;
  • a security control;
  • an audit-trail function;
  • an electronic signature;
  • an automated decision;
  • or a group of closely related controls.

Assessment at too high a level—such as assigning one risk rating to the entire application—can conceal critical differences between functions.

For example, a training-management system may contain:

  • a low-impact dashboard color preference;
  • a report used for administrative planning;
  • an automated calculation of training status;
  • a restriction preventing untrained personnel from performing a GMP task; and
  • an electronic record demonstrating personnel qualification.

These functions should not automatically receive the same risk treatment.

Develop Credible Failure Scenarios

A failure scenario should describe how a requirement, function, control, or data process could fail and what the resulting effect could be. Useful failure scenarios include:

  • an incorrect result is calculated;
  • required data are not recorded;
  • a record is associated with the wrong batch or sample;
  • an unauthorized user changes approved information;
  • a workflow bypasses a required review;
  • an electronic signature is applied incorrectly;
  • an audit trail does not record a critical change;
  • an interface loses, duplicates, delays, or transforms data incorrectly;
  • a report omits relevant records;
  • an alarm or notification is not generated;
  • a backup cannot be restored;
  • a configuration change alters approved system behavior; or
  • a system failure prevents completion of a required GMP control.

A useful failure statement is specific enough to support a test objective.

Weak failure statement:

The report does not work.

Improved failure statement:

The batch-status report omits rejected lots when a user selects multiple manufacturing sites, potentially providing incomplete information during product-disposition review.

The improved statement identifies the condition, failure, affected information, and possible consequence.

Evaluate the Consequences

Each credible failure scenario should be evaluated for its possible effect on:

Patient Safety

Could the failure contribute to an unsafe, ineffective, contaminated, incorrectly labeled, or otherwise unsuitable product reaching a patient?

Product Quality

Could the failure affect identity, strength, purity, quality, sterility, process control, specifications, material status, or product disposition?

Data Integrity

Could the failure make regulated data incomplete, inaccurate, unauthorised, unavailable, untraceable, or associated with the wrong activity? Relevant considerations include the ALCOA+ expectations for attributable, legible, contemporaneous, original, accurate, complete, consistent, enduring, and available records.

Regulatory Compliance

Could the failure prevent the organization from meeting a regulatory, filing, recordkeeping, retention, review, or reporting requirement?

For US drug manufacturing, relevant controls may include 21 CFR 211.68 and, where applicable, 21 CFR Part 11.

Process Availability

Could system unavailability prevent manufacturing, laboratory testing, product release, deviation management, or another required GMP activity? Availability becomes particularly important when no reliable manual or alternate process exists.

Consider Detectability

Detectability describes whether a failure is likely to be identified before it causes an unacceptable consequence. The assessment should determine:

  • what control detects the failure;
  • whether that control is independent;
  • when detection occurs;
  • whether detection occurs before the regulated decision;
  • who reviews the exception;
  • what information is available to the reviewer; and
  • whether performance of the control is documented.

A control should not be credited merely because a person might notice an error.

For example, manual review may provide meaningful detection only when:

  • the reviewer has access to the necessary source information;
  • the failure is visible;
  • acceptance criteria are defined;
  • the review occurs before approval or release; and
  • completion of the review is recorded.

Displaying the same incorrect calculated value on two screens does not provide independent detection.

Existing and Supplier Controls

Existing technical and procedural controls may reduce risk when their effectiveness is understood and supported by evidence. Controls may include:

  • input validation;
  • range and format checks;
  • workflow restrictions;
  • segregation of duties;
  • independent calculations;
  • audit trails;
  • electronic signatures;
  • reconciliation;
  • alarms and exception reports;
  • interface acknowledgments;
  • automated monitoring;
  • backup and recovery;
  • periodic review; and
  • supplier development and testing controls.

Supplier evidence may support the assessment when it has been evaluated through Computerized System Supplier Assessment and Evidence Leverage.

The organization should understand:

  • what the supplier tested;
  • which version and configuration were tested;
  • the test environment;
  • the acceptance criteria;
  • the results and unresolved defects;
  • whether the evidence applies to the organization’s intended use; and
  • what site-specific verification remains necessary.

Supplier testing cannot replace verification of the implemented configuration, interfaces, data, security roles, procedures, and business processes.

Complexity and Configuration Risk

Risk assessment should consider how the system is constructed and controlled. Relevant factors include:

  • software novelty and maturity;
  • configured workflows;
  • complex business rules;
  • custom code;
  • scripts and macros;
  • custom calculations;
  • low-code components;
  • custom reports;
  • interfaces;
  • data transformations;
  • infrastructure dependencies;
  • supplier release frequency;
  • undocumented configuration;
  • and difficulty detecting unintended behavior.

A standard product can create significant risk when it is extensively configured. A custom component can present limited risk when its purpose is narrow, its logic is simple, and its operation is independently verified.

Software category and complexity inform the type and depth of evidence needed. They do not determine GxP impact by themselves. This distinction is explained further in GAMP 5 Software Categories and Validation Strategy.

Do Not Rely on the Risk Score Alone

Numerical scoring can help prioritize assessment and testing, but the score should not be the sole justification for excluding verification.

A total score can conceal:

  • a severe but unlikely consequence;
  • weak or unproven detectability;
  • uncertainty in supplier controls;
  • a critical regulatory requirement;
  • cumulative failure across several functions;
  • or a low-scoring function required for intended use.

The documented rationale should explain:

  • the failure being considered;
  • its consequence;
  • the controls relied upon;
  • the uncertainty in those controls;
  • the selected test method;
  • and why the resulting evidence is sufficient.

Professional judgment and critical thinking remain necessary even when a formal scoring model is used.

Risk-to-test strategy framework connecting intended use, requirements, functions, data, failure scenarios, consequences, detectability, controls, uncertainty, test strategy, and residual-risk acceptance.
Risk assessment converts credible failure scenarios and control uncertainty into focused test objectives, challenge methods, evidence, and residual-risk decisions.

Define Test Objectives

Each test should have a clear objective connected to a requirement, risk, control, or failure scenario. Test objectives may include confirming that:

  • a calculation produces the correct result;
  • a user cannot bypass a required approval;
  • unauthorized roles cannot perform a restricted action;
  • invalid data are rejected;
  • boundary values are processed correctly;
  • an audit trail records the required information;
  • an interface transfers complete and accurate data;
  • a failed transaction is detected and recovered;
  • a report includes the correct population;
  • an electronic signature is linked to the correct record;
  • a backup can be restored;
  • or the system supports the complete intended business process.

Testing should demonstrate both correct operation and the effectiveness of controls intended to prevent or detect failure.

Select the Appropriate Test Method

No single test method is suitable for every objective. A risk-based strategy may combine scripted testing, unscripted testing, challenge testing, supplier evidence, automated testing, and review of configuration or code.

Risk-based test method selection comparing scripted testing, unscripted testing, challenge testing, supplier or automated evidence, and the complete validation evidence set.
Test methods should be selected by objective and risk; several methods may be combined to provide sufficient evidence.

Scripted Testing

Scripted tests are appropriate when consistent execution and detailed objective evidence are important. A scripted test normally defines:

  • preconditions;
  • test data;
  • user role;
  • test steps;
  • expected results;
  • actual results;
  • evidence requirements;
  • tester identity;
  • execution date;
  • and deviation handling.

Scripted testing is particularly useful for critical calculations, regulated workflows, security controls, signatures, audit trails, interfaces, and repeatable acceptance criteria.

Scripts should provide sufficient direction without forcing the tester to record obvious or meaningless actions.

Unscripted and Exploratory Testing

Unscripted testing allows a knowledgeable tester to explore system behavior beyond predefined steps. It can help identify:

  • unexpected workflow combinations;
  • confusing user behavior;
  • inconsistent error handling;
  • unanticipated data conditions;
  • navigation problems;
  • or interactions not covered by scripted tests.

Unscripted testing should still be controlled. The test charter, scope, tester, observations, evidence, and discovered defects should be recorded.

It should supplement—not automatically replace—scripted testing where precise verification and repeatable evidence are required.

Positive, Negative, and Boundary Testing

Positive Testing

Positive tests confirm that the system performs an intended function when valid information and authorized actions are used. Examples include:

  • completing an approved workflow;
  • calculating a result from valid inputs;
  • transferring a valid record;
  • or generating an authorized report.

Negative Testing

Negative tests challenge the controls intended to prevent unacceptable actions or data. Examples include attempting to:

  • enter an invalid value;
  • omit a required field;
  • bypass an approval;
  • perform an action without authorization;
  • apply an invalid status transition;
  • reuse an electronic signature improperly;
  • transfer a duplicate record;
  • or modify protected information.

Boundary Testing

Boundary tests evaluate values at and around defined limits. For a field accepting values from 1.0 through 10.0, meaningful tests may include:

  • the minimum accepted value;
  • the maximum accepted value;
  • a value below the minimum;
  • a value above the maximum;
  • the permitted decimal precision;
  • a blank value;
  • and an invalid format.

Boundary testing is particularly important for calculations, specifications, dates, capacities, field lengths, and processing thresholds.

Interface Testing

Interfaces should be tested end to end rather than only confirming that a message was sent. Testing should consider:

  • source and destination records;
  • field mapping;
  • units and precision;
  • timestamps;
  • status values;
  • transformations;
  • required metadata;
  • rejected messages;
  • duplicates;
  • retries;
  • partial transfers;
  • interrupted connections;
  • reconciliation;
  • and recovery.

The test should confirm that the receiving system contains the correct, complete, and usable information.

Further controls are addressed in Computerized System Interfaces and Data-Transfer Controls.

Security and Access-Control Testing

Security testing should be proportionate to the system’s intended use and risk. It may include verification of:

  • unique user accounts;
  • authentication;
  • role permissions;
  • least privilege;
  • segregation of duties;
  • privileged access;
  • account suspension;
  • session controls;
  • unauthorized-action prevention;
  • electronic-signature permissions;
  • audit-trail protection;
  • and secure interface accounts.

Testing should verify both authorized and prohibited activities. Confirming that an administrator can change a configuration is incomplete without verifying that an ordinary user cannot make the same change.

Regression-Testing Scope

Changes can affect functions beyond the component directly modified. Regression scope should consider:

  • the changed requirement or function;
  • shared calculations;
  • common workflows;
  • configuration dependencies;
  • reports;
  • interfaces;
  • security roles;
  • master data;
  • audit trails;
  • electronic signatures;
  • infrastructure;
  • data migration;
  • and previously resolved defects.

A supplier statement that a release contains only minor changes should not automatically determine regression scope. The organization should assess the release against its own configuration and intended use.

Regression tests may be selected from previously approved tests, automated tests, focused challenge tests, supplier evidence, or new tests developed for the change.

Validation Evidence

Evidence should be sufficient to demonstrate what was tested and whether the acceptance criteria were met. Depending on the test objective, evidence may include:

  • completed test records;
  • screenshots;
  • electronic test logs;
  • database or interface records;
  • audit-trail entries;
  • calculations;
  • report outputs;
  • configuration records;
  • automated-test results;
  • supplier test reports;
  • code-review records;
  • and deviation or defect records.

Screenshots should be used selectively. They should show meaningful results, system identity where necessary, and enough context to support review.

Evidence should be attributable to the tester, associated with the applicable test and system version, protected from uncontrolled change, and retained with the validation record.

Deviations and Failed Tests

Unexpected results should be recorded and evaluated rather than altered to make the test appear successful. The evaluation should determine:

  • what occurred;
  • whether the script or expected result was incorrect;
  • whether a system defect exists;
  • the root cause, where appropriate;
  • whether other tests or requirements are affected;
  • whether correction is required;
  • the required retest and regression scope;
  • and whether any residual risk remains.

Repeating a failed test without explaining the original failure does not provide adequate resolution.

Residual-Risk Acceptance

Residual risk is the risk remaining after controls, testing, correction, and other mitigation have been applied. Acceptance should consider:

  • the severity of the possible consequence;
  • demonstrated control effectiveness;
  • unresolved defects;
  • test coverage;
  • supplier evidence;
  • procedural controls;
  • detectability;
  • operational monitoring;
  • and uncertainty.

Significant unresolved risks should identify:

  • the affected requirement or process;
  • the reason immediate correction is not feasible;
  • interim controls;
  • operational restrictions;
  • monitoring requirements;
  • responsible owners;
  • target dates;
  • and formal approval authority.

Residual risk should be accepted by individuals with appropriate process, technical, and Quality authority. A risk score alone is not evidence that the risk is acceptable.

Maintain the Assessment Through the Lifecycle

Risk assessment and test strategy should be reviewed when changes affect:

  • intended use;
  • requirements;
  • configuration;
  • custom code;
  • calculations;
  • workflows;
  • reports;
  • interfaces;
  • data;
  • security;
  • infrastructure;
  • suppliers;
  • or applicable regulations.

Risk information should remain traceable to requirements, specifications, tests, deviations, defects, and release decisions.

The same information should support change control, regression testing, periodic review, incident assessment, and eventual system retirement.

Practical Outcome

An effective computerized-system risk assessment should clearly establish:

  • what can fail;
  • why the failure matters;
  • which controls prevent or detect it;
  • how confident the organization is in those controls;
  • what test objective addresses the risk;
  • which test method is appropriate;
  • what evidence must be retained;
  • what regression testing is necessary;
  • and who may accept the remaining risk.

The goal is not to produce the lowest possible risk score or the smallest number of tests. The goal is to obtain sufficient, reliable evidence that the configured computerized system supports its intended GxP use and that significant residual risks are understood and accepted.