-
What is Machine learning? Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it to learn for themselves.
-
What is DevOps? DevOps is a set of practices that combines software development and information-technology operations which aims to shorten the systems development life cycle and provide continuous delivery with high software quality.
-
What are container technology and 𝐎𝐒-𝐥𝐞𝐯𝐞𝐥 𝐯𝐢𝐫𝐭𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧? Container technology is a method of packaging an application so it can be run with isolated dependencies, and they have fundamentally altered the development of software today due to their compartmentalization of a computer system. OS-level virtualization refers to an operating system paradigm in which the kernel allows the existence of multiple isolated user-space instances.
-
What is MLOps? How it's important? MLOps is a practice for collaboration and communication between data scientists and operations professionals to help manage the production of ML lifecycle. MLOps is the only way to scale machine learning into production. There are a number of barriers that prevent organizations from successfully implementing ML across the enterprise, including difficulties with Deployment and automation, Reproducibility of models and predictions, Diagnostics, Governance, regulatory compliance, Scalability, Collaboration, and Business uses. A standard practice, such as MLOps takes into account each of the aforementioned areas, which can help enterprises optimize workflows and avoid issues during implementation.
- The aim is to automate a machine learning architecture and to predict only if accuracy is more than 90%. If accuracy is less than 90%, then it will tweak the machine learning model architecture automatically. Jenkins will be checking GitHub repositories every second for any changes. So as soon as a developer commits and pushes, Jenkins will start its jobs.
- Tech n Tools: ML libraries(TensorFlow, Keras, NumPy, Pandas, etc) to create a prediction model and Docker image for further automation, Jenkins(a great DevOps tool for automation), Docker for OS-level virtualization(to create images and containers on RHEL8), Red Hat Enterprise Linux 8, VMware(to run RHEL8 on our host operating system).
(1) Install VMware on your host operating system. In VMware install the RHEL8 operating system. In RHEL8 install Docker(first you have to install all the requirements of Linux and configure yum and then install docker on RHEL8)
(2) After completion of the docker installation on Linux, create a docker image for the machine learning model to run and predict using a container. So make a directory and create a Dockerfile inside that directory and then use the docker build command to create the required image. (it will download and install everything) Command: docker build -t mlopsimg:v1 .
(3) after the creation of the image install Jenkins on RHEL8 and start Jenkins services. Jenkins needs java to start services, so install java first. After installation, start Jenkins and copy-paste the IP of Linux on the browser with port 8080(Jenkins work on the port 8080) to access Jenkins GUI. example: 192.162.152.123:8080
- sudo yum install jenkins java-1.8.0-openjdk-devel
- sudo yum install jenkins
- sudo systemctl start jenkins
(4) Now create account on github and make a repository. Give anyname, i have given Project-MLops. Also make a folder in your windows or any os where you are working on machine learning code and paste all the required files in the same folder so that we can connect it with our github repository and push our updated work when ever we want.
wer have created three files here...
- now adding, commiting and pushing our files on the github repository.
- this is the repository i made and pushed all the files for my ML project.
(5) In Jenkins create JOB-1. It will Pull the Github repo automatically whenever the developer commits and pushes the repository to Github. This job will copy all the repository files in a directory /root/mlops_code/ in our RHEL8 operating system.
- here i used POLL SCM , "Poll SCM" polls the SCM periodically for checking if any changes/ new commits were made and shall build the project if any new commits were pushed since the last build.
- all the repository files are now pasted in our RhEL folder.(neglect the other files for a while)
(6) JOB-2: it will only trigger after the completion of job1. Job2 will run a container from ml_py_img image we created and it also copy-pastes the mlops_code repository files from RHEL8 to container's /root/mlops_code directory. And as soon as the command runs, it will start executing the Training_model.py file.
This .py file will create a result.txt file which will be contaning the accuracy of the model.
(7) JOB-3: this job will check the accuracy from the file result.txt. If the accuracy is less than the required accuracy then this job will automatically start adding the combination of the layers to increase the accuracy of the model.
- here accuracy is already 98% so tweaking will not be done.
- as our accuracy is good, so it will send a mail by executing sendmail.py file to the devloper
(8) JOB-4: it will monitor the container fails . If containers fails then this job will re run the containers and will start traning the model again. SO we will add a post buil as Job:2.
- Build pipeline to watch the execution of the jobs
𝐂𝐨𝐧𝐜𝐥𝐮𝐬𝐢𝐨𝐧: Automated the Machine Learning architecture using Jenkins, now as soon as Developer commits and pushes the files on the GitHub, a docker container will be created with all the required machine learning libraries and model will train automatically with above required accuracy.
https://drive.google.com/file/d/1eEGapVk0IzwfyZgkYItfm6EnLTfit-lv/view?usp=sharing