Optimizing GenAI Inference Performance

This capstone unit focuses on operationalizing and optimizing large language model inference: hardware utilization, batch tuning, inference acceleration tools, monitoring and observability, cost-performance trade-offs, and future trends. Students will synthesize prior units on model design and prompt engineering to prepare robust, cost-effective deployed GenAI solutions.

Learning Objectives

• Analyze hardware utilization and profiling data to identify inference bottlenecks and recommend batch sizing, micro-batching, and parallelization strategies to improve throughput and latency
• Apply inference acceleration tools and techniques (e.g., quantization, optimized runtimes, ONNX/TensorRT, and hardware-specific kernels) to deploy models with reduced latency while maintaining acceptable accuracy
• Demonstrate monitoring and observability practices by instrumenting deployed models, building dashboards and alerts for latency, throughput, error rates, and data drift, and interpreting signals to drive operational adjustments
• Evaluate performance-versus-cost trade-offs and propose deployment strategies (autoscaling, resource selection, throughput vs. latency configurations) that balance user requirements and budget constraints while accounting for emerging GenAI trends