GitLab-CI Pipeline (Concepts)

10 min readJun 4, 2020

Continuous Integration

Continuous Integration is a practice followed by developers, that involves integrating code changes and commit frequently in a shared repository. Version Control Systems like GitHub, BitBucket can be used for the purpose.

This involves automated builds and tests whenever a code change is committed to the VCS. Earlier developers used to work in isolation, and when it was time to integrate the code, it became very difficult to resolve the bugs and conflicts, which would consequently waste time as well as efforts. Such problems can be avoided by the help of CI. Working on different branches may keep the work isolated for a short span just to make sure no that there is no conflict on merging and the master branch can meet the desired results.

Continuous Integration also involves automated tests on the application so that any bug in the code is caught early in the development process and fixed. These automated tests are run for every build. This doesn’t make the code bug free, but it just takes care that the codes are caught and fixed beforehand, rather than at a later stage when it becomes too complicated to fix them.

Therefore, the practice of CI is being adopted by developers to have faster builds, automating the process of builds and tests to increase efficiency and do it at a faster rate.

Continuous Deployment

Continuous Deployment is a part of the release pipeline, a plan to release software only after it has passed all the tests successfully and is automatically released into the production environment.
It just deploys the feature to the production environment but not running the code until the team decides to release the feature.

Continuous Delivery

Continuous Delivery is similar to continuous deployment, what is different is just the last manual approval of whether or not to release the product into the production. In the delivery stage, developers review the code changes, merge them, and then package them into an artifact. The package waits for approval, in which the package is opened and reviewed again with a system of automated checks. When it passes all the sets of tests, at that time, the code is deployed automatically.

About GitLab

GitLab is an open-source project, having over 2000 contributors, made to allow different teams to work together and make the software development better and faster. It is a single application, or a DevOps platform as a single application, to carry out the complete DevOps life cycle. It makes the lifecycle much shorter. The lifecycle includes general stages such as managing applications, planning, creating, verifying, packaging, configuring, monitoring, and making our application more secure. A brief description about them have been provided below.

Managing the application:

GitLab provides a feature to manage teams which lets them be aware of how the business is being performed. It helps the teams to optimize the software delivery life cycle by making use of metrics.

New groups and sub-groups can be made by the owners or the administrators.
The custom templates for groups provided by GitLab can be used which makes all the public projects available for that group.

Planning:

GitLab provides planning tools to make sure that the teams are working on the right track. It makes use of certain tools to feature planning.

Issue tracking- Any issue in the project can be created that would provide a medium to collaborate on a new idea, implementation on new code, tracking status.
Kanban Boards to list out the task status, what all has been done, and what is left.
Time tracking to find out the approximate time spent on issues.
Quality Management to track the quality of the project.

There are many more features under planning which can be explored and utilized.

Creating the code:

We can create, edit, view, and manage code, can create branches by GitLab repositories. Teams can follow the workflow without disruption.

It provides source code management, enabling collaboration, merging branches, tracking changes, and rolling back in case of failures.
Code reviews to discuss changes, identify defects, and automate them.
Web IDE to edit the code and make changes online and contribute.
Snippets can be used to share small pieces of code with other users.

Verifying the code:

GitLab also helps us to verify the code maintaining code quality, and coding standards necessary for production. These include-

Continuous Integration to automate the builds, tests, security checks, and verify each commit.
Code quality to check whether on some commit the quality of the code has been degraded or improved.
Unit testing along with the code coverage to ensure that modules perform functionality as expected.
Load Testing to ensure that changes are validated with real-world scenarios.

Packaging the code:

GitLab enables to package applications and dependencies together and build artifacts so that they work in the supply chain. The feature provides the following things for package management:

Package registry, where the project’s packages and dependencies are stored.
Container registry which is used as a private registry to store the docker images. The images can be created, pushed, and retrieved.

Configuring the application:

Application environments can be configured and managed through GitLab. Secret variables can be used to limit access to only authorized access and processes. Kubernetes integration reduces the effort to define the infrastructure required to run the application.

Auto DevOps
Kubernetes Management

Monitoring the application:

GitLab monitors metrics automatically to track how the changes made in the code impacted the production environment. Monitoring provides feedback which can be responded quickly.

GitLab displays information for the projects.
Error tracking to discover errors that the application might be throwing, increasing awareness.

Security:

Security issues are left unaddressed at the time of development of any application. GitLab provides built-in templates for security checks.

SAST to check the security issues within the system.
DAST for security checks outside the system when the application is running.
Security tests.

Releasing the application:

GitLab automates the release pipeline and the delivery of applications. The applications can be delivered with zero-touch. This makes the process quite short and faster.

Deliver changes to production with zero-touch and allows release through the path defined for production.
GitLab pages use a static site generator that creates websites and is deployed by GitHub.

Defending the application:

GitLab also takes care of the security issues of an application during runtime, such as threat detection, data security, and application infrastructure security.

In the Auto DevOps feature provided by GitLab, a Web Application Firewall(WAF) feature is enabled which examines the traffic being sent to the application and detects malicious content before it reaches the application and causes harm.
Container Network Security blocks the unauthorized network between the pods and the internet while implementing Kubernetes.

Why GitLab?

GitLab has a shorter life cycle. A shorter feedback life cycle also has some more benefits like:
Reducing risks such as:
It is easy to estimate smaller iterations.
Every minute changes in the code get attention, which results in better code quality.
The applications are more secured as many security defects can be rectified during the development time only.
The Silos and Stages are connected, as it is a single platform, where every team is aware of everything, GitLab links all the information together.
Through GitLab, every stage involved in the DevOps lifecycle can be automated.

In a nutshell, GitLab gives the following key advantages:

Even minute changes are taken care of, fully tested, secured, and are not ignored, resulting in better code quality.
A feedback loop is created through GitLab’s monitoring tools.
A common single platform, so each team member is on the same page sharing the common things.
Feedbacks are at the same place where the code is written and rectified in case of any fault.
There are complete visibility and control over the project, where it is leading to.

Also, a very good study of GitLab in comparison with the other tools has been done here.

Understanding GitLab functioning

Continuous integration and continuous deployment service are provided by GitLab. GitLab functions on one configuration file which is the GitLab-ci.yml file. For every commit that is pushed to the GitLab repository, the added GitLab-ci.yml file is triggered which is run by the help of GitLab runners. The YAML file tells the runner what all jobs need to be carried out.

The backbone for any project to involve CI/CD is pipelines.

GitLab CI/CD Pipelines

In GitLab CI, pipelines are made up of main stages and jobs.
* Jobs define the task that has been carried out and
* the stages define the order in which the jobs are to be executed.
Jobs are executed by GitLab Runners.
Parallel jobs can be executed, this usually results in longer build time, if there are more than four parallel jobs, or the GitLab runners can be configured accordingly in spite of using the GitLab shared runners. We will be covering this concept later in the blog.
If any of the stages fails in the pipeline, execution of the other further stages is stopped and the pipeline is said to have failed.

A general pipeline consists of mainly four stages, namely:

The build stage mainly carried out the compilation of the source code or the compile job.
The test stage, in which unit testing on the source code is done, or the test job is carried out.
The staging stage, where the source code is deployed to the staging environment.
The production stage, where the source code is deployed to the production environment.

Other things about the pipeline, including the types of pipelines, can be referenced from here, as it would deviate the main topic.

This is somewhat how the pipeline we are going to develop in the project looks like.

Pipeline we will be making by the end of this project

Usually, a pipeline is automatically triggered when any changes are made in the files lying in the repository, only if the GitLab’s configuration file, that is, the GitLab-ci.yml is present in the root of the repository.

GitLab keeps the whole metrics of the pipeline, the duration it ran, the number of success, which can be viewed in the Analytics Section on the GitLab project’s page.

This is the pipeline analytics that has been shown below in the picture.

GitLab Runners

The jobs in the pipelines are executed by GitLab runners. GitLab provides some runners by default, which are known as shared runners. Although, we can also configure runners according to our needs, which would eventually result in longer pipeline execution time. Below are mentioned a few features of runners:

Installation of runners is quite easy on many platforms like Windows, macOS, Linux.
It enables caching of docker containers.
Jobs can run in Docker containers, or locally, along with the parallel execution of multiple jobs.
Makes use of the token for each project, and the number of jobs per token can also be limited.

GitLab configuration file

GitLab uses the GitLab-ci.yml file to configure the things that are required to be done in the project. As soon as a change is committed, this file starts to run the job using runners for that commit.

Creating a GitLab Project

We are going to create a CI/CD pipeline that would build an artifact, perform unit tests, package into war files, perform some more tests, and finally deploys it to the Kubernetes cluster on GCP. We will be learning about every stage in the pipeline. The project is built in Java, using Maven, in which test cases have been written by the JUnit framework. So let’s get started!

Firstly, an account is required on GitLab, which is free to create. It allows GitHub integration as well, so we can log in to GitLab through GitHub also. You can create the account from here.

After creating an account, click on the “New project” button.

For this demonstration, we would be cloning a project from GitLab itself. To do that, follow with the steps:

After clicking on the new project option, select the “Import project” tab, and select the “git Repo by URL” option.

After doing this, insert “https://gitlab.com/petermyren/messageboard" in the URL tab.

And click on the “Create” button. This will clone the repository into your GitLab account.

The project will be cloned in a few tweaks. Everything is set now, and we are ready to set up our CI/CD pipeline.

Creating a sample .gitlab-ci.yml file

Pipelines in GitLab are configured using a YAML file, the “.gitlab-ci.yml” file for each project in the root of the directory. It defines the structure of the pipeline and the order in which the jobs are to be executed. Jobs are executed using GitLab runners.
We can configure our own GitLab runners, although, for this project, we will be using the GitLab shared runners.
A pipeline consists of jobs and stages. Try out first a very basic pipeline, that will give an overview of how things are going to work and how the jobs are actually carried.
Create a new file named “.gitlab-ci.yml”, in the root of the repository and put the following script into it and commit the changes done in the repository.

stages: 
    - build
    - test
    - package
    - deploy
    
build:
    stage: build
    script: 
        - echo " build"
    
test:
    stage: test
    script: echo " unit tests"
    
package:
    stage: test
    script: cat file.py file2.py | gzip > packaged.gz
    artifacts:
        paths:
            - packaged.gz
    
deploy:
    stage: deploy
    script: echo "Deployed"