Jenkins (CI/CD) using Kubernetes, Git-GitHub, Ansible & Docker in Depth.

🔰 Welcome Back, After Integration of Ansible with Docker on EC2 Instance, It’s Time to Take it to the Next Level by Having Some More Trending DevOps Tools in the Bucket 😏.

🔰 Let’s First have a Look at which Tools & For what Reason they Will be used.

🔹 GitHub — As a Source Code Management System & Maybe Git as a Version Control System too.

🔹 Jenkins — As a CI/CD tool to Fetch the Code Every time when it get’s triggered by the GitHub Poll SCM.

🔹 Ansible — As a Node which will be Connected to Jenkins using SSH protocol & will be used to Run the Playbook.

🔹 Docker (Docker hub) — As a Tool to Build the Docker Images & Push it to the Public Registry.

🔹 Kubernetes — As a Container Management tool which will Create Deployments & Service for The New Images every time when it get’s Pushed to Docker Hub.

🔰 I know this Sounds too Boring 🥱 but Just Stay with me & Let’s witness the magic of DevOps 😀.

— — — — — — — — — — — — — — — — — — — — — — — —

✰ Let’s First Understand the Whole Architecture of this Project and How each Component will Work once the Pipeline Gets Triggered 👍.

✅ Understanding the Architecture :

A) GITHUB

1️⃣ It All Starts with Writing of Code, Developer will Write a Code & Store it in A Public open Source Registry known as GitHub (Version Control System). Why Version Control System ?, Because The Code keeps on getting Updated every time as we Get Updates on the App Store/ play Store.

✰ This is How a Repository Looks, But it’s way to Easier then what is shown in the Above Image. The Important Thing to Know Over here is What this Repo will Contain ? 🤔 So it will Contain a Dockerfile (Click Here to Know More about Dockerfile in Detail & How to use it).

🔰 The Work of Github is Now Done, Now Let’s Move towards Our Next Component i.e Jenkins

— — — — — — — — — — — — — — — — — — — — — — — —

B) JENKINS 🤵

🔰 What is Jenkins ?

✰ Jenkins is a popular open source tool for CI/CD that is free to use. While you may need some server administration skills to configure and monitor Jenkins, there are many advantages to consider. The Jenkins project includes a large plugin ecosystem, the community around it is thriving and it is actively developed.

✰ In Other Words, Jenkins is an open source continuous integration/continuous delivery and deployment (CI/CD) automation software DevOps tool written in the Java programming language. It is used to implement CI/CD workflows, called pipelines.

🔹 Jenkins will be Integrated with GitHub through a Concept called Poll SCM which we will see wherever needed. What this will do is It will Trigger the Jenkins Code which is to be Executed whenever there is Some change in the Code Committed at GitHub. In Other words whenever the Code is Updated, the Jenkins Code have to Run.

🔹 This is How the Jenkins user Interface look.

🔰 Again, Looks Very Complicated 😮 But It’s 10x Easier then How it looks. We’re not yet done with the Jenkins, its time for new Tool to be Integrated called as Ansible.

— — — — — — — — — — — — — — — — — — — — — — — —

C) ANSIBLE 🅰

🔰 What is Ansible ?

✰ Ansible is the simplest way to automate apps and IT infrastructure. Application Deployment + Configuration Management + Continuous Delivery.

✰ Again Not Discussing Much About this Tool over here Since It’s been Already Covered in My Last Article where I Explained How Use This Tool to Automate the Launching of EC2 instance and Configure Docker Container on Top of it 🌐.

🔹 Let’s Understand What’s the Role of Ansible Over Here.

✰ We Will Have a Separate EC2 Instance for the Ansible Because it will also Contain Docker Platform which will Build the Docker Image using Docker Build Command, Then Log in to Docker Hub Id using docker login -u xyzz -p xyzzz and Push it to the Docker Hub 📁.

✰ There are More Technical Things to be Noted Here which we will see while Building the Logic for Our pipeline. Now Last but Not the Least The Next Tool i.e Kubernetes ☸️ will come into Play.

— — — — — — — — — — — — — — — — — — — — — — — —

D) Kubernetes ☸️

🔰 What is Kubernetes ?

✰ Kubernetes, often abbreviated as “K8s”, orchestrates containerized applications to run on a cluster of hosts. The K8s system automates the deployment and management of cloud native applications using on-premises infrastructure or public cloud platforms.

✰ Again I’ve Explained About Kubernetes & it’s Real World Use Cases in Very Detail so You can Click Here to Land on That Blog.

🔹 Let’s Understand What’s the Role of K8s Over Here.

✰ We Will Have two Separate EC2 Instances for the k8s Because Kubernetes Requires More Computer Resources Compared to Docker & Ansible, etc. So One Instance will act as a Master Node & One as a Slave Node, The Master Node Keeps a track of Slave Node & Note : having at least one Slave Node is necessary to make Kubernetes Work the Way we Want.

✰ The Kubernetes Resources i.e (Deployment & Service) will be Automated using an Ansible Playbook. The Ansible hosts Will contain the IP address of the Kubernetes Master Node which will be used for Configuration through SSH protocol. The Things which will be Automated we will See as They Come. So Here’s A Quick Recap of The Whole Architecture.

GITHUB (SOURCE CODE MANAGEMENT)

JENKINS (USED TO TRIGGER THE CODE) 🤵

ANSIBLE (CREATE A DOCKER IMAGE THROUGHT DOCKEFILE WHICH IS RECEIVED THROUGHT GITHUB) 🅰

DOCKERHUB (AS A PUBLIC REGISTRY WHERE DOCKER IMAGE WILL BE PUSHED) 🖥️

KUBERNETES ( TO CREATE DEPLOYMENT & SERVICE SO THAT THE WEBPAGE CAN BE ACCESSED BY THE OTHER USERS) ☸️

— — — — — — — — — — — — — — — — — — — — — — — —

Note : Whatever Playbooks we will see, It’s not at all a good Practice to Write all the Tasks and variables, password ,assets, secret key directly in the playbook, but just not to mess up the things by switching multiple directories & we have many playbooks in this Project. So I've Kept all in one single File.

— — — — — — — — — — — — — — — — — — — — — — — —

✅ Install Jenkins Using Ansible Playbook :

✰ Let’s Breakdown the Playbook into Small Components. Here’s the Security Group Configuration which is Slight Important.

✰ This Configuration is Exactly Same as it was in the Ansible-Docker Project , The Only Important Thing to note here is that Jenkins by default works on port number 8080 and We don’t want our Jenkins Server to be accessed by any other across the globe like to do while accessing any web page. So the cidr_ip for 8080 port is 61.2.151.37/32 which simply specifies that the port 8080 can be only accessed by my IP, Note: This IP will be Different for Your account so to get this IP, while configuring the security group from the AWS management console 👇 Click on My IP in the Source block & You’ll get your configuration.

✰ Next Comes the EC2 Instance & ssh :

🔹 Again Exactly same as Pervious Project, only change is in the Security Group which will be created before launching of EC2 instance & Thus will be used in EC2.

🔹 Below that the public IP address is getting retrieved from the ec2 variable which gathered all the Info while launching the EC2 Instance & this IP is stored in Host Group name Jenkins which is link the hosts file at /etc/ansible/hosts but this is dynamic in Nature. & lastly waiting for SSH protocol to Start since its a Linux Machine so SSH is necessary otherwise Configuration won’t be Possible.

🔰 Now the Actually Jenkins Installation Begins:

◼ Firstly the Host as mentioned Jenkins is used & then the very first task is to upgrade all the packages like (yum update -y) command.

◼Next step is to Download the preconfigured yum repository which will be used to install Jenkins & Other Packages which it Supports.

◼ Now Import The Jenkins Key which is used to Validate and Download the most stable version of Jenkins.

◼Yum Upgrade is Not Actually Required but Still to be present and a smooth environment where we try not to get any error, we are using this command & Jenkins is build on Top of Java so we need the Most recent version of Java JDK (Java Development Kit). The command used here is supported by support by amazon Linux Flavor & it’s not present as a module to be installed/supported by ansible hence we are using shell module.

◼ Finally Installing Jenkins & Git because we want to Integrate Github with Jenkins so along with Git plugin the package should be also installed in the Jenkins instance. Then Start the Jenkins Service which will allow port 8080 through firewall.

◼ lastly this is not the part of Installation but just to save time, Jenkins when opened through the web portal asks for password To ensure Jenkins is securely set up by the administrator, a password has been written to the log. The Output of command will be stored in password variable and thus will be displayed at the end of the playbook using the debug which basically prints the Output of variable.

🔰 Now Let’s Run the playbook:

◼ Once The Execution of Playbook is Completed, as you can see we got the Password retrieved from the variable so now copy the Public IP address and open it in new tab at port number 8080.

◼ This is How the web Portal of Jenkins Look like Initially :

Enter the Password and Click continue

◼ Click on Install suggested Plugins and it will download the Required and mostly commonly used plugins.

◼ Next after Plugins Installation, Create a User for the Jenkins in the Next Window. Give Credentials of your Choice & Click Save and Continue

◼And Here we Are with the beautiful Web Page of Jenkins. How the Required Components & Features Work we will see as Required.

— — — — — — — — — — — — — — — — — — — — — — — —

🔰 Since we are Done with the Jenkins Setup, Now it’s Time to Integrate it with Ansible Instance (also known as Node). But Remember, Ansible will be Further Running the Playbook on the Kubernetes Node & For That Ansible will be Requiring the Public IP Address of the K8s Node, So It will be Better that we First Create The Kubernetes Cluster & Then Provide the IP Address of that Instance in the Playbook Itself so that we Don’t have to Manually So and Update with IP. Something Like This.

— — — — — — — — — — — — — — — — — — — — — — — —

✅ Kubernetes Cluster using Ansible Playbook :

✰ Let’s Breakdown the Playbook into Small Components. The First two i.e Security Group & Instance Creation is exactly the Same.

✰ The Security Group is allowing port 80 since we will be hosting a Web Server & The Name here is Kubernetes-SG and the hosts group name after Instance launched will be Master.

— — — — — — — — — — — — — — — — — — — — — — — —

✰ Now, There are several Ways to Setup Kubernetes example: using minikube, kops or on cloud like EKS, GKE, etc, One of the easiest way is using Kubeadm.

💠 Kubeadm is a tool built to provide kubeadm init and kubeadm join as best-practice "fast paths" for creating Kubernetes clusters.

💠 kubeadm performs the actions necessary to get a minimum viable cluster up and running. By design, it cares only about bootstrapping, not about provisioning machines. Likewise, installing various nice-to-have addons, like the Kubernetes Dashboard, monitoring solutions, and cloud-specific addons, is not in scope.

💠 Instead, we expect higher-level and more tailored tooling to be built on top of kubeadm, and ideally, using kubeadm as the basis of all deployments will make it easier to create conformant clusters.

So Let’s First Setup the Kubeadm Repository which will help downloading the necessary package's which we will see.

✰ Next Step is to Install Docker, Since K8s is Build on top of Docker, Then, Install Kubeadm which will Install kubectl & kubelet as the internal Dependencies to install Kubeam.

🔰 iproute-tc is Linux Traffic Control utility. Tc is used to configure Traffic Control in the Linux kernel. Traffic Control consists of the following: SHAPING When traffic is shaped, its rate of transmission is under control. Shaping may be more than lowering the available bandwidth — it is also used to smooth out bursts in traffic for better network behavior.

Very Complex Right, In Simple term, Why Are we Installing Kubernetes? To Run/Deploy the Pod. Now if you’re aware About Docker Containers, whenever we run a new container using docker run command, when the container is launched, it gets dynamically provisioned with a New IP right. So who is providing this IP ? 👉 iproute-tc.

🔰 The kubelet is the primary “node agent” that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider.

The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object that describes a pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn’t manage containers which were not created by Kubernetes.

✰ So this is like a Watch Man for Kubernetes Pods.

✰ Lastly Start the Service docker and kubelet.

🔰 Next Command is kubeamd config image pull command. Now There are lot’s of processes Running, It Maybe iproute-tc, kubelet, kubectl to launch pods, etc. All these processes are to be Performed while launching a Pod, Ultimately a Container inside a Pod. So obviously Containers are Created using Images, and also we got different images for Kube Scheduler, Control Manager, etc. So rather you download all these images one by one, there is a chance you might miss out some. So here kubeadm is taking all the Load of pulling the Images using This Command and so it says kubeadm config image pull.

🔹 Control Groups (cgroups) are a feature of the Linux kernel that allow you to limit the access processes and containers have to system resources such as CPU, RAM, IOPS and network.

🔹 Docker uses cgroups to limit the system resources. When you install Docker binary on a linux box like ubuntu it will install cgroup related packages and create subsystem directories.

🔹 Since Our Goal is to make Docker Compatible with Kubeam, We have to change the Default Cgroup to systemd so that kubeadm can access it according to requirement. & then restart the docker service to update the Changes.

✰ Finally the kubeadm init command is used to initialize the kubernetes Cluster. Here the Argument — — pod-network-cidr=10.244.0.0/16 specifies that we need to choose a network provider & now the iproute-tc will do the work , the 10.244.0.0/16 specifies that initial two octet i.e 10 = 8 bits & 244 = 8 bits (8 + 8 = 16 bits) will remain constant and rest 0.0 will change till 255.255 example (10.244.0.1, 10.244.32.250, etc).

✰ The — — ignore-preflight-errors=NumCPU specifies that since we are using t2.micro EC2 Instance, the minimum requirement for k8s Cluster is :

🔹 So if we can’t meet these Requirements, use — — ignore-preflight-errors=NumCPU & so — — ignore-preflight-errors=Mem Command to Initialize k8s Cluster without prompting these errors.

✰ To make kubectl work for your non-root user, run these commands, which are also part of the kubeadm init output.

✰ Flannel is an open-source virtual network project managed by CoreOS network designed for Kubernetes. Each host in a flannel cluster runs an agent called flanneld . It assigns each host a subnet, which acts as the IP address pool for containers running on the host.

🔹 This command is to create a Token, This Token will we used by the Kubernetes Slave Node to connect to this master Node. After the Playbook Execution is Completed we will see how.

🔹 This is Not mandatory, But to automate everything i’m Copying the deployment.yml file & service.yml file to the /home/ec2-user/ directory of Master Node. The Files Look like this :

↬ In Simple terms, A Kubernetes Deployment is used to tell Kubernetes how to create or modify instances of the pods that hold a containerized application. Deployments can scale the number of replica pods, enable rollout of updated code in a controlled manner, or roll back to an earlier deployment version if necessary.

↬ So what we understand from this yml file is replicas : 2 means it will create 2 instances of pod, selector, matchLables means this deployment will select those pods who have label as app: webcontainer, this is a key : value pair, and if it finds a pod with these labels, it will start monitoring it and the template specifies that, whenever this deployment has to create new pods, it will use whatever metadata info is provided as a part of Configuration to Launch a New Pod, Here it will give label as app: webcontainer, the name for the container will again be webcontainer and the image used for launching will be swar2001/cloudproject which is my docker hub id with the image name itself and port 80 will be exposed since it’s a web httpd container.

🔰 Note : swar2001/cloudproject image is not Currently Present in the Docker Hub if I show you

These two repos were created long back. But the Image specified in the Deployment will be Created by Jenkins Pipeline as a Code.

Let’s look at the Service.yml File

◎ Here the Nodeport means the Public IP address of the instance where k8s is running, and again selector will select the key : value pair as app: webcontainer since we want to associate this with the pods launched by the deployment. there are three ports: nodePort means what Port number with Ip address of EC2 instance will the webpage Display, Eg. http://123.432.13.24:30038, targetPort is the Port of Container which is exposed in the Deployment File and the port: 80 means the Pod (Deployment) will be connected to this service through port 80.

◎ So Let’s Finally Run the Playbook and See what we Get.

As you can see we got the Token to Connect the Slave Instance, but where is it ?

Well, We have to Run a Playbook for Slave instance But Excluding the Following Tasks:

This is Because our Kubernetes Cluster is already Running, We don’t want two Separate Clusters, we Want one Master and One Slave node to Work together, so since kubeadm, kubelet, kubectl is already setup at Master Node, no need to do these steps at the slave node.

The Only Thing we have to add in the Slave node is :

This is just some kernel Setting we have to do in Slave node to enable some networking Rules.

Now Run the Playbook for Slave Node :

Now the Instance is Successfully Created as You can see at the AWS Console

Now Click on Connect :

⇝Now Copy the Link provided in the Example & Paste it in your Local VM in my case its Red hat Linux.

⇝ Note: the private key (.pem) should be in same Directory where you’ll paste this link or you can specify the complete Location like -/var/www/rhel8new.pem

⇝Now Paste the Token here with sudo before it since it requires root power and hit enter

⇝ The Node has Joined the Cluster, to Confirm This, Go to the Master Node, again grep the link from the Example and paste it in local system.

⇝ Now Our Kubernetes & Jenkins Setup is Ready.

💠Now We have 3 Instances Running 1 = Jenkins , 2 = K8s master, 3 = k8s Slave, Now the 4th, the Last & The Most Important Instance which will do the Work of pulling the Code from GitHub, Build Docker Image, Push to Docker Hub and Finally Deploy the Image is the Ansible instance.

Here's the Complete playbook :

🔹 The hosts name Ansible is taken From the host Group created after launching of EC2 Instance. Firstly Updating all the Packages all we did in the Last playbook just to make the instance stable.

🔹 Installing jDK is mandatory over here because we are gonna connect this Ansible Node to Jenkins using SSH Protocol But, Since Jenkins is Build on top of java as we already saw, So it important to have the java package installed on the Host Node too so that the communication can set up without any Error.

🔹 Installing Ansible can be also done by Package Module but in Amazon Linux it prompts with this command like “ You can try using amazon-linux-extras install ansible2 ” so I used shell module to execute it.

🔹 Docker is Obviously Needed to Build Docker Images and Push Them & Git is Required cause Jenkins will be downloading the Code from Git Hub Repo using Git clone Command behind the Scene, so Having Git is necessary.
Lastly Enable the Docker Service.

🔶 These Steps are Important i.e setting up the ansible config File in the /opt/ location. & Then the reason we first Create Kubernetes Cluster was to get the Public IP which can Update over here.

🔶 Next Step is Important i.e to Copy the private key (.pem) File which will enable ansible to communicate with the K8s Instance. Change the Permission to r i.e (read only) using chmod 400 command.

🔶 Now we are Creating a directory because while configuring Jenkins with this Node, It asks for a Location where it can store all the Information which it will gather while performing different tasks, Like on the server where Jenkins is installed, the default working directory is /var/lib/Jenkins similarly inside the ansible node the working directory will be /home/ec2-user/jenkinsws.

🔶 And then this step is not Required but Just to check the Ping test with the K8s Cluster that Ansible Node is Able to Connect or not. we are using ping command in the same Directory where the ansible config file is present i.e /opt/.

🔶 Last step is to Copy the playbook which will have steps to create Kubernetes Deployment and Service Resource.

Let’s See what this yml file contains

➽ The Important thing over here is the first two shell commands, when this playbook will run for the very first time, there will be no Deployment and service currently running on the k8s master node, so it is obviously gonna give error, but we will need these commands when we’ll run this playbook for the second time cause we have to delete the Old resources and update them with New. So That’s why we use ignore_errors module. and pausing the playbook for 5 seconds ever time because sometime it takes more time to create the resources so its fine to have a short delay.

➽ Now Let’s Run the Playbook.

➽If you See green Color Means Everything Works Fine 😅. You can see Ping Pong which is the Standard Output Which Ansible Gives.

➽Now Let’s Have a Look at Our EC2 Instances.

— — — — — — — — — — — — — — — — — — — — — — — —

✅ Configure The Ansible Node on Jenkins :

➠ For This, Let’s Get Back to Our Jenkins Dashboard.

➠ Now Click on Manage Jenkins which is on the Left Side.

➠ Click on Manage Nodes and Clouds.

➠ This is the Node on Top of which Jenkins is Installed so it says Built-in-Node.

➠ Click on New Node On the Left.

➠ Give the Node Name as Your Wish, Here I’m Giving Ansible and making it Permanent Agent since we’ll require it for constant (CI/CD). Now click on Create.

➠ All these Configurations are as per users choice, The important thing over here is the Remote root Directory should be given as per mentioned in the ansible Playbook if I Show You.

➠ Number of Executors Means how many Task you want to allow Jenkins to Perform Parallelly. By default it’s 1.

➠ Labels play an Important role cause they will be used to identify the Nodes in Jenkins Pipeline Code.

➠ Launch method is SSH so Provide the Public IP address of Ansible Node. Credentials as You know we have .pem private key and not any password.

➠ So click on add and then Jenkins

➠ Now Select SSH Username with Private Key Option

➠ Give ID & Description as per choice and Make sure the Username is correct. by default for all AWS Instances it’s ec2-user.

➠ Now Click on Enter Directly & Then Add cause here we have to Directly paste the Private key Content.

➠ You can you any Text Editor to Open the Private key File and then simply copy and paste Starting from — — -BEGIN RSA PRIVATE KEY — — — ………………— — -END RSA PRIVATE KEY — — -

➠And then click on add

➠Now select the Credentials which you created just now

➠ Select Non Verifying Verification Strategy So it won’t give any error while setting up Connectivity with the Agent. Event though it’s not a Good Practice to Do This but it works fine in a Testing Environment.

➠ Click on Save.

➠ You Still might get this error, Why ?

➠If you click on see Logs You’ll see this

This Says that File or Directory Not Found, But we’ve Created One in Our Ansible Playbook, Still Error ? 🤔

Rememeber : Whenever we run Ansible Playbook, In the privilege escalation Part which is this

[privilege_escalation]
become = true
become_method = sudo
become_user = root
become_ask_pass = false

Its Says that Ansible will be allowed To configure the ec2 Instance with Root Power, so the /home/ec2-user/jenkinsws directory was Created using root Power, But Jenkins do Not Have root access power so it’s Giving an Error.

➠ So what you can do is go to ansible instance by connecting it through SSH

➠ Click on connect on the bottom right.

➠ Then You’ll be directly connected to the EC2 instance’s user console

➠Now What you can do is remove the jenkinsws directory and then again create it but this time we are ec2-user and not root user.

So if you again click on Relaunch Agent.

➠ You won’t see any error and If you go back to nodes

⇛We Now have Ansible Node with it’s all configurations.

⇛ Now we Are Ready to Build the Pipeline.

⇛ Get Back to the Dashboard by clicking on the Top left Side.

⇛ click on New Item.

⇛ Enter an item name of your choice and then choose Pipeline and click on OK.

⇛ I’m Intentionally Choosing cloudproject as the name, why will see Further.

⇛You’ll land on this page

⇛ Now Scroll down completely.

⇛ The Script is the block where You’ll Write the Pipeline Code. Don’t Worry, It’s Not hard as any programming Language, It’s very Straight Forward, we’ll see how to define the Syntax But for Quick Test with the ansible node that whether Jenkins is able to connect or not. You can see try sample Pipeline option over there, Click on that.

⇛ Now Select Hello World Script

⇛ This is what You’ll get, The Only thing to change over here is the agent, here any means it will execute the hello world Script on any of the executors that are available while running the Script. and we don’t want that so change:

From : agent any

To: agent {
label “ansiblenode”
}

This will notify that the Specific Pipeline code is to be run on ansible node. That’s why while Configuring the Ansible Node I mentioned about Labels Specifically.

⇛ So this is how the Code is looking :

⇛ Now Simply click on Save or Apply both are same.

⇛ Click on Build Now & as You can see the Build History at the bottom left side.

⇛ Here You can see No builds But as soon as you click Build Now

⇛ The Pipeline will be Triggered

⇛ You can see a Job is Triggered.

⇛ Now we are Doing These Steps manually, Here We Should try to Automate The Things as Much as they can be.

⇛ So Click on Configure.

⇛ And then Build Triggers.

⇛ Now The Triggers Simply Means Something that will Notify jenkins Pipeline to Run the Code. Each of Them will have Their Own Special Use Case but For Our Project We Will go with Poll SCM

⇛ Below that Column we can see that It Says will only run due to SCM changes if triggered by a post-commit hook. This Post-commit will Come from Github, But we haven’t Seen Github since the beginning of Project. Well, here we go Now

⇛ Open Github And Create a New Repo,

⇛Give Name of Your Choice and don’t change anything, just create it.

⇛ Now We will use these Commands to Connect this Repository with Our Git Bash.

⇛Even Though we can create a New File Manually Over here to But We still don’t want to Open Github again and again because Remember : DevOps

➡ So for Our Pooh, Open Git Bash.

❖I’m creating a Jenkins directory since we want to use git init Command.

❖ Now Copy paste Commands And Don’t git add README.md File.

❖ Now Before we Git Push, First Let’s create a Jenkinfile which will contain the Code.

❖ Here I am using Windows so Of course Notepad and Like we have Dockerfile, so Docker will only use Dockerfile to Build Docker Image and this File is without any Extension, Similarly Jenkins will only Read Jenkinsfile without ay Extension. so Hit enter and Copy paste the Code.

❖ Click on Save as and Navigate to the Folder where Git repo is Initialized.

❖ While Saving, Give Double Quotes to the File name “Jenkinsfile” so it Won’t Get an Extension.

❖ Now Save and Git Push.

❖ Some Standard Git add and Commit Commands and Then Finally Git Push, Now if you go to Git Hub.

❖we have a Jenkinsfile, Now Copy the Repo Link & Go back to Jenkins

❖ And here scroll down and Now select Pipeline Script from SCM (Source Code Management).

❖ Select Git as SCM and then paste the Repo Link which we copied and since It’s a Public Repo so it won’t Require any Credentials.

❖ Our Default Branch is main and The Script Path Means where the Jenkins File is Located in the Repository, In Our Case There’s not any Other directory so we’ll keep it as it is and Now Go back to poll SCM and type * * * * *

❖ * * * * * means every minute of every hour of every day of every month and every day of the week.

Now Click on Save.

❖ Go Back to Dashboard and Click on Build now to check whether GitHub Repo Is Successfully Integrated Or Not.

❖ 3rd Job Run And now We’ll Keep This Page and Git Bash Side By Side and Let’s Check If Jenkins Job is Getting triggered by Poll SCM or not. for This we need to do at least one change on the Jenkins File Locally so that it will reflect on Git Hub and hence forth Jenkins.

❖ Just a Random Simple Message. And standard Git Commands,

❖Now we are Ready for Git Push and in the Build History you can see the time 2:35 PM and when we will push the Changes It’s 8:12 pm in my local PC :

❖ We’ve Pushed and Let’s see if Jenkins Pipeline is Triggered or not

❖And here we Go, at 8.13 pm local PC Time means Exactly after one Minute which we’ve specified in our Poll SCM * * * * *. The Job is Triggered.

❖ and the 4th Job was successfully executed at 2:43 According to the Jenkins machine Time.

— — — — — — — — — — — — — — — — — — — — — — — —

✰ So Let’s get Back to the Notepad and See how the Pipeline Syntax Works so that we will be ready to build the code for Our K8S Deployment.

✅ Groovy Pipeline Syntax :

➲ what is Groovy?

➲ Well, Apache Groovy is an object-oriented programming language used for JVM platform. This dynamic language has a lot of features drawing inspiration from Python, Smalltalk & Ruby. It can be used to orchestrate your pipeline in Jenkins and it can glue different languages together meaning that teams in your project can be contributing in different languages.

➲ Groovy can seamlessly interface with the Java language and the syntax of Java and Groovy is very similar. If you forget the syntax when writing Groovy, you can continue to write directly in Java syntax. Groovy can also be used as one of the scripting languages for the Java platform. Groovy scripts can be called in Java, which effectively reduces time spent on Java development.

➲ It also offers many productivity features like DSL support, closures, and dynamic typing. Unlike some other languages, it functions as a companion, not a replacement, for Java. Groovy source code gets compiled to JVM Bytecode so it can run on any platform.

— — — — — — — — — — — — — — — — — — — — — — — —

☸ This is the Code we have, Let’s Break it Down !!

☸ So the Very first Keyword i.e. pipeline implies that the whole code within the curly braces of pipeline will be executed as a part of pipeline as code, so this is mandatory.

☸ Now It’s Clear That This Pipeline is being Created for CI/CD Code, but who will be taking the Responsibility to Run the Code, You need a Machine to Run the Code Right !!,So before we directly start writing the Code through Logic. It’s Important to Specify that Which machine should be Used to Run the Code, Whether it’s the Jenkins Master Node, Or The Ansible Node which we Configured, or any Other Node. So after specifying the pipeline block, the first thing that should be written inside it is that which machine i.e. which agent should be used. the Syntax is :

☸ Now we Start Writing The actually Code or Pipeline we call it as.

☸ Code is Always Written in Steps Write. Like if we want to get the sum of 2 variables then first we declare using (step 1: a = 2, b = 2.2) (step 3 : c = a + b = 2 + 2.2 = 4.2) (step 3: print(c)). A Simple Python code to Add two Numbers.

☸ So If we Write This Down in Code it will be something like this.

◉ In Groovy Language, each line of Code which implies something is written in steps.

◉ Now Here, we as a Developer Understand that we are Declaring two variables in first step, then adding them in second step, and finally print in third Step, But this is just an Example which we are using to Practice, in Real world use case It’s Important to Specify what each line/block of Code will Do. So Here we use one More Keyword i.e. stage, the Stage name can be any line which will elaborate what the code will do like :

◉ But How will The agent Come to know that how many Lines of Code it has to Perform, because we haven’t Given any Indentation i.e. block of code within ansible node, So for This, below the agent keyword where bracket ends, Use the Keyword that will make work for ansible node easy which is stages:

◉ This was the Last Thing to be Noted while Writing the pipeline script.

◉ Pipeline -> agent -> stages -> stage.

🔹 So let’s get back to Our GitHub Repo where we have the Jenkins File and Write down The Initial Step:

🔹 So Our Very First Step will be to Write a Dockerfile which will contain the webpage info and rest metadata that is to be executed while container image is getting build.

� �So this is Our Initial Repo :

🔹 Let’s Write a Docker File Now.

🔹 In the Latest Centos Images We are First Running Some Config commands that will upgrade the Yum Repo, then one the repo is upgraded, it will Use yum command to install httpd package and zip package, then a link is provided where the webpage along with html, CSS, JavaScript content which will be downloaded to the /var/www/html which is by default location. thne it will unzip and extract the content from the file which was downloaded through the link, cp command will copy and paste all the files in current directory i.e /var/www/html and the at last remove all the unnecessary file and only keep index.html, CSS, JavaScript & Finally Expose Post 80 which is by default port number for web page (httpd)

🔹To Understand How Dockerfile works and what this each keyword over here Implies, click here where I’ve Explained Dockerfile from Scratch.

🔹 So This Docker File will be Used to Create a Docker Image on Ansible Node where we’ve Installed Docker Package.

🔹 Let’s Do git Push and See if Job get’s triggered or not:

🔹 A 6th Job is Triggered the next minute of git push , so Let’s See what it did with the Dockerfile by going to the console Output:

🔹 As you can see it’s storing all the Data in the directory which we created as non-root User i.e /home/ec2-user/jenkinsws/. Now This is By default Behavior of Jenkins that whatever Directory we Specify, Whiting that Directory whenever any Job Get’s Triggered, It creates another Dir. called workspace and within that one more directory whose name is same as Our pipeline project name which you can see at top left. So This Location will Play an Important role in Future.

🔹 But You can see it did Nothing to the Dockerfile since we haven’t yet updated the Code.

🔹 So Let’s Get back to the Notepad where we have our Jenkins File.

♦ So here we give some meaningful line to the git step even though it’s gonna pull the whole repo but our work is only Related to Dockerfile so ..

♦ Now here you can see echo as a step, echo command is supported by all Linux OS which simply prints the Link, But Since we are using Groovy as a Syntax, so The Line of code which we use, For Example git clone …… or git pull …. might be different over Here.

♦ For This Jenkins has Given one More Facility. Go to the Pipeline and click on pipeline Syntax on the left side

♦ Now There are almost all the options available to run all type of code throught pipeline, Our Work is with github so we select git. And Now enter the Repository Link :

♦ Now click on generate pipeline Script :

♦ Now you’ll get a link of Code Which is supported by the Groovy Syntax. Just Copy and Paste it in the Code.

♦ So again Git Push and Let the Job run.

♦ Job Number 7 Runs

♦ It’s Says Fetching Changes From git Repo Means the GitHub Repos is downloaded successfully in /home/ec2-user/jenkinsws/workspace/cloudproject directory, So Our Docker File is Present in This Location only So We Have to Build Our Docker image either by copying it to any outside location or cd to the same location and build there. The Second One Looks more efficient.

♦ Don’t Worry, This time we Don’t any Any Pipeline Syntax because from know every thing we will do from the shell which we have & the syntax is sh ‘ …code …’ as easy as that, so let’s go back to Jenkinsfile Notepad.

♦ So here we add new stage called Build Docker Image.

♦ The step is simple, run two commands and that are to first change directory to /home/ec2-user/jenkinsws/workspace/cloudproject and build docker image at Image at the same Location.

♦ Now Here’s the Trick in this Syntax :

♦ sudo docker build -t $JOB_NAME:v1.$BUILD_ID

♦ So the name of our Docker image will be same as the name of our pipeline i.e cloudproject, this is because Jenkins have Some Predefined Built in variables like

• BUILD_NUMBER — The current build number. For example “153”
• BUILD_ID — The current build id. For example “2018–08–22_23–59–59”
• BUILD_DISPLAY_NAME — The name of the current build. For example “#153”.
• JOB_NAME — Name of the project of this build. For example “foo”
• BUILD_TAG — String of “jenkins-${JOB_NAME}-${BUILD_NUMBER}”.
• EXECUTOR_NUMBER — The unique number that identifies the current executor.
• NODE_NAME — Name of the “slave” or “master”. For example “linux”.
• NODE_LABELS — Whitespace-separated list of labels that the node is assigned.
• WORKSPACE — Absolute path of the build as a workspace.
• JENKINS_HOME — Absolute path on the master node for Jenkins to store data.
• JENKINS_URL — URL of Jenkins. For example http://server:port/jenkins/
• BUILD_URL — Full URL of this build. For example http://server:port/jenkins/job/foo/15/
• JOB_URL — Full URL of this job. For example http://server:port/jenkins/job/foo/.

➸ We are using these but more important than that is $BUILD_ID

➸ sudo docker build -t $JOB_NAME:v1.$BUILD_ID

➸ This is Because, when we will run this syntax for the First Time, It will create a Docker image called as cloudproject:v1 without Build ID

Now if we run the Same Script again But This Time the Docker File is Different. so while creating a Docker image it won’t give any error but while naming the image, it will prompt something like a name with this image is already Present and the Code won’t get updated in the Environment. So to Avoid This we use $JOB_NAME:$BUILD_ID and this build id is nothing but the Number of builds if I show You :

➸ So everytime new Image will build, it will build as

$JOB_NAME:v1.$BUILD_ID i.e.

cloudproject:v1.8

cloudproject:v1.9

cloudproject:v1.10

cloudproject:v1.11

➸ Versioning will be Done Correctly.

➸ Hence now save and Run the Code and we are running one more command just to get the image.

➸ Push & Wait for the Magic.

➸ This is How it Displays while Pipeline is in Running Stage :

➸And Here we Go :

➸ Image with latest tag 1.8

➸ in the Console Out Put You can se everything it did while creating this Image:

➸ Now We have The Image, we have to Push it to the Docker Hub. For that the first Thing you should do is Change the Image tag and associate it with docker hub id, mine is swar2001 so the image will look like swar2001/cloudproject:v1.8,

➸ back to the Notepad with one more stage :

➸ Just a Docker image tag command is used which will change the Image tag and we are tagging two images with latest and v1.$BUILD_ID as a good practice of Versioning.

➸ Also in the Above stage “Build Docker Image” I’ve added one more line called rmi because now when the pipeline will get triggered, 9th build will start, but here we are not changing the content of Dockerfile, we are changing the Jenkinsfile only. So instead of building the exact same image just different tag, after the image is build we will delete it after build since one image is of size 287 mb and we don’t want to fill the Storage up Since we are limited with t2.micro.

➸ So I’m just changing the $BUILD_ID to 8 so that it will tag the previous image and not the latest 9th Build Image.

➸ Git Push Now !!

➸ Finished Really Quick cause it has already downloaded the Centos base image last time when we build the image, Let’s go to the console output.

➸It Builds the Image, then un tag the new one i.e. remove and at last change the name with Docker Hub ID.

➸ Now Last Step is to Push the Image to Docker Hub. Notepad, Here I Come !!

➸ Let’s write Down Stage to First Login To Docker Hub.

➸Oops !! You Saw my Password & So will those who read this Script, So It’s Important to Keep The Password Secret but How ?

➸ For This Go Back to Jenkins Dashboard. & Open Manage Jenkins.

➸ Scroll Down & Click manage Credentials.

➸ Click on Jenkins below credentials

➸ Select Global Credentials (unrestricted).

➸ Click on Add Credentials to the Left, Also Here you can see the private key file our ansiblenode ec2-user is stored.

➸ Kind is secret text cause we are just storing/hiding the password.

➸ Type Password in Secret and Give it a Unique ID. click Ok.

➸ Now this is not an environmental variable like $JOB_NAME, $BUILD_ID which Jenkins have pre Created. This variable is called as Credential which we have created, so in order to make it usable as a variable, Go to The cloudproject pipeline and then Pipiline syntax.

➸ Here linking credentials to variable is known as Binding so select that.

➸ Being Password so it’s a Secret text. and Now Give a Variable name which you will use in the shell sh ‘docker login -p $ — — -’ syntax.

➸You can see the credentials we stored pops up here so now it will get associated with variable.

➸ Click on Generate Pipeline Script and Copy the Syntax.

➸ The //same block says that the line in which you’re gonna use the variable should be withing the Curly brackets that were generated in pipeline Script.

➸ To check if we are Correct till date. Git Push

➸ 10th One triggered and success

➸Just Feels Amazing When you Get no Errors !!

➸ Now the Last and Easy Step is to Push Image to Docker Hub.

➸ Notepad !!

🟢 Push the Images and immediately clear the space from the system since the image will be now available on the Docker Hub. Once we push we will open Docker Hub besides while Jenkins Job will be running

🟢 The 13th Job is Completed and if I refresh the Docker Hub

🟢A cloudproject repo come up.

the 11th & 12th Job Failed because i did some syntax error like this :

🟢So please be calm while writing the Code !!

🟢 But if we go to 13th one:

🟢 Image pushed with two versions :

🟢 Now all we have to do is to add the last Stage which is Running the Playbook to create Kubernetes Deployment. But First Let’s clear the Code properly and get it back to normal since we have v.1.8 images to docker hub.

� �The very Last step is to Run the K8s Deployment and Service Playbook which is present in /opt/ directory so first cd and run the playbook.

🟢 Right Now I’m on K8s Master Node & It has just one default service running but as soon as we push changes, Let’s see what happens.

🟢 As soon as 14th Job Completed, You can see Kubernetes Deployments & Service are created on the side

🟢Now take the public IP address along with the post as given by service i.e 30038

💠 The Web Page is Running Successfully.

💠 Now let’s Change the Dockerfile Code and Push it one more Time.

� �The New Dockerfile :

💠 This time we will add a Jewelry Web Page.

💠 15th Job

💠 These are the changes that happened after running 15th Job and now if we refresh the kindle web page

— — — — — — — — — — — — — — — — — — — — — — — —

🔰 So From the Above Article we Successfully Completed Our Integration of Jenkins with Ansible Docker Git-Github & Kubernetes. If you Find This Interesting then Do Follow & Connect on Linkedin 😄.

THANK YOU !!

— — — — — — — — — — — — — — — — — — — — — — — —