Scaling Network Connections from the Jenkins Controller

Oleg Nenashev and I will be speaking at DevOps World | Jenkins World in San Francisco this year about Scaling Network Connections from the Jenkins Controller. Over the years there have been many efforts to analyze, optimize, and fortify the “Remoting channel” that allows a controller to orchestrate agent activity and receive build results. Techniques such as tuning the agent launcher can improve service, but qualitative change can only come from fundamentally reworking what gets transmitted and how.

In March, JENKINS-27035 introduced a framework for inspecting the traffic on a Remoting channel at a high level. Previously, developers could only use generic low-level tools such as Wireshark, which cannot identify the precise piece of Jenkins code responsible for traffic.

Over the past few months, the Cloud Native SIG has been making progress in addressing root causes. The Artifact Manager on S3 plugin has been released and integrated with Jenkins Evergreen, allowing upload and download of large artifacts to happen entirely between the agent and Amazon servers. Prototype plugins allow all build log content generated by an agent (such as in sh steps) to be streamed directly to external storage services such as AWS CloudWatch Logs. Work has also begun on uploading JUnit-format test results, which can sometimes get big, directly from an agent to database storage. All these efforts can reduce the load on the Jenkins controller and local network without requiring developers to touch their Pipeline scripts.

Other approaches are on the horizon. While “one-shot” agents run in fresh VMs or containers greatly improve reproducibility, they suffer from the need to transmit megabytes of Java code for every build, so Jenkins features will need to be built to precache most or all of it. Work is underway to use Apache Kafka to make channels more robust against network failures. Most dramatically, the proposed Cloud Native Jenkins MVP would eliminate the bottleneck of a single Jenkins controller service handling hundreds of builds.