Remoting over Apache Kafka with Kubernetes features

Goal: Enhance the plugin and to provide out-of-the-box management in Kubernetes environment

Status: Completed

Team

Student:
Long Nguyen
Mentor(s):
Andrey Falko ,
Pham Vu Tuan ,
Ankit Jain ,
Supun Wanniarachchi ,
Oleg Nenashev

Details

Motivation

Previous version of Remoting over Apache Kafka plugin requires users to manually configure the entire system which includes Zookeeper, Apache Kafka and Remoting agents. It also didn’t support dynamic agent provisioning so scalability was harder to achieve. This project aims to solve two problems:

Out-of-the-box solution to provision Apache Kafka cluster.
Dynamic agent provisioning in a Kubernetes cluster.

Phase 1 Summary

Launching Apache Kafka in Kubernetes feature:
- User interface (JENKINS-57667)
- Back-end function (JENKINS-57896)
- Kubernetes YAML specification files (JENKINS-57669)
Helm chart skeleton (JENKINS-57670)

Phase 2 Summary

Cloud implementation to dynamically provision Remoting Kafka Agent in Kubernetes (JENKINS-57668)
Helm chart completion (JENKINS-57671)
Configuration-as-Code support (JENKINS-58218)
Release version 2.0 (JENKINS-58218)

Phase 3 Summary

Add retention strategy for Cloud (JENKINS-58690)
Write unit tests and integration tests to bring test coverage to 70% (JENKINS-57898, JENKINS-58786)
Update dependencies and Apache Kafka versions (JENKINS-58081)
Add JaCoCo to the project

Features

Here are some screenshots of the implemented features

1. Launch Apache Kafka on Kubernetes

Apache Kafka provisioning in Kubernetes UI

2. Cloud configuration

Remoting Kafka Cloud UI

3. Remoting Kafka Agent get provisioned dynamically

Remoting Kafka Agent get provisioned

4. Helm Chart to bootstrap the whole system on Kubernetes

Kubectl get all to check Helm chart installation