Processes: AI/ML Lifecycle Management

End-to-end data management process for AI model development covering data acquisition, understanding, processing, and quality assessment as defined in SOP-0303064. Ensures data integrity, representativeness, and fitness for AI modelling purposes.

Iterative process for designing, developing, training, and evaluating AI models as defined in SOP-0303064. Covers model architecture selection, hyperparameter tuning, performance evaluation against test datasets, and model locking prior to deployment.

Periodic review process for AI sub-systems as defined in SOP-0303064. Reviews risk assessments, data integrity, model performance, GxP compliance, security, audit trails, and change management at a frequency determined by the AI Sub-system Risk Level.

Systematic process of identifying, assessing, and controlling risks from AI sub-systems that may impact Patient Safety, Product Quality, or Data Integrity, as defined in SOP-0303064. Covers the full risk workflow from System Risk Assessment through AI-specific risk scoring and mitigation.

Risk-based testing and verification process for AI sub-systems as defined in SOP-0303064. Determines testing rigor based on AI Sub-system Risk Level, executes SAT and UAT with AI-specific test types, and manages defects against acceptance criteria.

Operational monitoring process for deployed agentic sub-systems defined in SOP-0304679, providing continuous visibility into agent behavior and acceptance. Includes automated monitoring of critical Agent Controls, structured human acceptance judgments, and defined response procedures for anomalies a

Structured multi-step process to identify, evaluate, and control risks associated with implementing AI agents in GxP applications. Defined in SOP-0304679, it encompasses system risk assessment, agent capability and complexity evaluation, GxP risk level determination, and formal AI risk assessment an

Risk-based testing process for agentic sub-systems defined in SOP-0304679, structured into unit testing, System Acceptance Testing (SAT), and User Acceptance Testing (UAT). Testing rigor is scaled according to Agent GxP Risk Level, using code-based, human, and model-based graders to verify agent con

Ongoing monitoring and maintenance process for deployed AI models as defined in SOP-0303064. Continuously tracks model quality in operation, detects data and concept distribution shifts, triggers alerts and change management when thresholds are breached.

Evaluates the stability and reliability of GenAI system outputs when the same input is processed multiple times, as described in GD-0304459. Uses chi-squared based variance testing to verify that output variability remains below a defined standard deviation threshold, ensuring reproducibility in reg

Testing of GenAI sub-system outputs based on ground truth availability and risk level as defined in SOP-0303429. Includes determination of testing approach (classification metrics, statistical scorers, or SME evaluation), definition of metrics and thresholds, and execution of testing with appropriat

Ongoing monitoring of GenAI sub-system performance during the operation and maintenance phase as defined in SOP-0303429. Ensures continued operation as intended using the same evaluation methodology from testing, with adjusted thresholds and sampling for the operational environment.

Validates GenAI sub-system performance using statistical hypothesis testing for binary, proportional, and ordinal outcomes as described in GD-0304459. Covers determination of test rigor based on system risk, selection of sampling plans with acceptance criteria, conducting tests, and calculating Wils

Development, testing, and refinement of system prompts for GenAI foundational models as defined in SOP-0303429. Covers prompt design following established practices, collection of testing sets with business stakeholder input, iterative testing and refinement, and version-controlled documentation of

Implementation and validation of Retrieval-Augmented Generation pipeline components as defined in SOP-0303429. Covers data preparation, embedding model implementation, knowledge base construction, retriever configuration, and generator setup with documented rationale for each component.

Evaluation of RAG pipeline components as defined in SOP-0303429. Covers individual testing of embedding model, retriever (context precision and recall), and generator (faithfulness, hallucination, noise sensitivity) with risk-based approach to combined testing.