Pharmaceutical, biotechnology and medical device manufacturers generate considerable amounts of quality information.
Deviation records, laboratory results, process-monitoring data, complaints, supplier findings and CAPA histories all contain information about how manufacturing and quality systems are performing.
Yet many organizations review these data sources separately, often after a problem has already occurred.
Digital quality metrics can help reveal relationships between quality events and support earlier investigation of emerging risks.
Artificial intelligence and machine learning can extend these capabilities, but only when the underlying data, analytical methods and decision-making processes are appropriately controlled.
What makes a quality metric useful?
A useful quality metric helps an organization answer a meaningful operational question.
For example:
- Are repeat deviations increasing in a particular manufacturing area?
- Is a critical process parameter becoming less stable?
- Are supplier quality issues concentrated in a specific material category?
- Are investigations repeatedly exceeding their planned completion times?
- Are corrective actions preventing the same failures from recurring?
Each question should lead to a defined metric and a planned response when the results indicate concern.
A dashboard that displays dozens of indicators without clear ownership may offer little practical value.
FDA describes quality metrics as a way of measuring and monitoring pharmaceutical product and process performance in its Quality Metrics for Drug Manufacturing resources.
It is important not to confuse an organization's internal monitoring practices with a universal FDA requirement to submit a particular dashboard or metric set.
FDA's previously published Submission of Quality Metrics Data guidance was draft guidance, not a generally binding final reporting requirement.
Distinguish leading and lagging indicators
Traditional quality reporting often focuses on outcomes that have already occurred.
Examples include product rejections, complaint totals, confirmed deviations and batch failures.
These are generally lagging indicators.
Leading indicators attempt to identify conditions that may increase the probability of future problems.
Examples might include repeated minor process drift, overdue maintenance on critical equipment or an increasing number of recurring investigation themes.
Neither category is inherently superior.
A leading indicator is useful only if it is supported by a credible connection to the risk being monitored.
For example, a rising training completion percentage does not necessarily demonstrate improved operator performance. The organization may need a separate measure of task execution or error recurrence.
A balanced quality framework combines information about process stability, outcomes, controls and the effectiveness of corrective actions.
Establish data integrity before introducing AI
Machine learning cannot repair inaccurate, incomplete or poorly governed source records.
Suppose a manufacturer trains an anomaly-detection model using historical deviation records.
If different departments use inconsistent deviation categories, the model may identify apparent patterns that reflect documentation practices rather than actual manufacturing problems.
Before applying advanced analytics, organizations should establish:
Data ownership: Who is responsible for the quality and meaning of each data source?
Consistent definitions: Do departments use the same terminology for similar quality events?
Traceability: Can an analytical result be connected to its underlying source records?
Data completeness: Are missing or excluded records understood?
Change management: Are transformations, model changes and configuration adjustments controlled?
Access management: Can unauthorized personnel alter source data or calculated results?
FDA's Data Integrity and Compliance With Drug CGMP guidance explains the importance of reliable and accurate data in regulated operations.
That guidance should be read alongside applicable binding CGMP regulations.
Use AI for defined questions, not vague automation goals
AI can support activities such as identifying anomalies, grouping related deviation descriptions, recognizing trends and prioritizing records for further review.
However, the value of an AI system depends on its intended use.
Consider two hypothetical applications.
Application A: A model groups similar deviation descriptions to help investigators identify possible recurring themes.
Application B: A model independently determines whether a batch should be released.
The second application has substantially greater potential impact on product quality and patient safety.
It therefore raises different requirements for assurance, governance, evidence and human decision authority.
Before introducing AI, define:
- The precise question the system will answer.
- The data it will use.
- The consequences of an incorrect result.
- The performance that is acceptable for the intended use.
- How humans will review and act on the output.
- How performance will be monitored over time.
These decisions should precede software procurement or model development.
Validate analytics in proportion to risk
A quality dashboard may calculate straightforward descriptive statistics or incorporate a complex predictive model.
Its assurance strategy should reflect the intended use and potential consequences of failure.
For medical device production and quality management system software, the FDA's Computer Software Assurance guidance, issued in February 2026, describes a risk-based approach to establishing confidence in software.
It should not automatically be treated as a regulation governing every AI model used by every life sciences organization.
For drug and biological product regulatory submissions, FDA has also issued draft guidance on AI model credibility.
That draft is specifically concerned with AI used to support regulatory decision-making regarding the safety, effectiveness or quality of drugs and biological products. It is not a universal, binding validation standard for internal quality dashboards.
Nevertheless, its emphasis on context of use, risk and credibility offers a useful conceptual framework.
Why human oversight remains necessary
Machine learning can generate inaccurate conclusions when data distributions change, relevant information is missing or patterns are interpreted incorrectly.
A model might identify an apparent relationship between supplier delays and product defects.
That association does not automatically establish causation.
Quality professionals must determine whether the relationship is credible and whether additional investigation is warranted.
Human oversight should therefore be designed into the workflow.
For example, a deviation-trending dashboard might flag an increase in recurring equipment-related events.
The assigned reviewer could examine source records, verify the categorization, assess product impact and decide whether the pattern requires formal investigation.
The AI system contributes information. It does not replace accountable quality judgment.
Monitor model performance and data drift
An analytical tool that performs satisfactorily during initial testing may become less reliable as manufacturing conditions evolve.
Possible causes include new equipment, revised processes, different material suppliers, changes in data collection or altered investigation terminology.
A monitoring plan should establish how the organization will identify meaningful performance changes.
Depending on intended use, measures may include:
- False-positive and false-negative patterns.
- Missing or unexpectedly changing data.
- Model performance against reviewed examples.
- Unexpected changes in output distributions.
- User overrides and the reasons for them.
- Changes to data sources or model configuration.
The monitoring frequency and response should be based on risk.
An internal exploratory tool used to support weekly trend review will not necessarily require the same controls as software whose output directly influences critical manufacturing decisions.
Build a continuous-improvement feedback loop
Digital quality metrics become valuable when they connect detection to action and verification.
A practical workflow is:
Observe: Collect and assess trustworthy quality information.
Interpret: Identify meaningful trends, exceptions or emerging signals.
Investigate: Determine whether the signal reflects an actual problem and evaluate its causes.
Act: Implement appropriate improvements or controls.
Verify: Assess whether the intervention produced the intended result.
Learn: Update metrics, thresholds and monitoring arrangements as necessary.
This approach is consistent with the continual-improvement principles discussed in ICH Q10 Pharmaceutical Quality System.
Practical takeaway
Before investing in another analytics platform, select three quality decisions that the organization currently finds difficult.
For each, identify the relevant data sources, the metric required, the action threshold, the responsible reviewer and the evidence needed to determine whether the resulting decision was effective.
If those elements cannot be defined, introducing AI may increase complexity without improving quality management.
The objective is not to automate every decision. It is to make important decisions more timely, transparent and evidence-based.
Explore the related EduCraft session
EduCraft's Digital Quality Metrics and AI/ML Analytics session addresses digital quality signals, AI/ML analytics, GxP software controls, data integrity and human oversight in regulated environments. Visit the session page for current access information.
