In this tutorial, we will practice selected techniques for evaluating machine learning systems, and then monitoring them in production. It is one of a 3-part series:
- Offline evaluation of ML systems
- Online evaluation of ML systems
- Evaluation of ML systems by closing the feedback loop (this part!)
In this particular section, we will practice evaluation in the online testing stage - when the system is serving real users - by "closing the loop" between production use of the service, and continuous evaluation/monitoring and re-training.
Follow along at Evaluation of ML systems by closing the feedback loop.
This tutorial uses: one m1.medium VM at KVM@TACC, and one floating IP.
This material is based upon work supported by the National Science Foundation under Grant No. 2230079.