AI Model Monitoring and Platform Evaluation

This unit focuses on building and operating enterprise-grade model monitoring systems in the cloud, comparing AI/ML platform offerings (including Microsoft) and applying techniques to optimize deployed models. Learners will use workbook templates and tracking sheets to plan generative AI use cases, instrument endpoints, monitor performance and drift, and document remediation strategies for ongoing operations.

Learning Objectives

• Analyze and compare enterprise AI/ML platform components and services (including the Microsoft AI/ML stack and alternative cloud providers) using an AI/ML platform evaluation sheet
• Design and build a model monitoring system that instruments model endpoints, captures key metrics (latency, accuracy proxies, data drift), and implements tracking via a model monitoring tracking sheet
• Apply model optimization techniques and endpoint configuration changes to improve performance and demonstrate measurable improvements in selected metrics
• Evaluate monitoring outputs and use the generative AI use case planner and monitoring artifacts to create an operational remediation and governance plan