Daniel's Tech Blog

Cloud Computing, Cloud Native & Kubernetes

Optimize build performance on Jenkins Azure worker nodes

Today we focus on some performance optimizations for the Jenkins worker nodes on Azure instead of security as in the last blog post.

-> https://www.danielstechblog.io/secure-jenkins-worker-nodes-on-azure/

As you might know already each Azure VM gets an additional SSD-based temporary drive beside the OS disk drive. Per default the swap file is stored on this temporary drive, because it is ephemeral.

So, instead using an Azure VM size supporting premium storage (SSD) to get the required disk performance, you can use VM sizes only supporting standard storage (HDD) and leverage the temporary SSD drive for your builds.

The table below shows some VM sizes and their disk throughputs for the OS disk and temporary drive.

VM size OS disk IOPS OS disk throughput R/W in MBps Temporary drive IOPS Temporary drive throughput R/W in MBps
Standard_A8_v2 500 60 8000 160 / 80
Standard_D8_v3 500 60 12000 187 / 93
Standard_D4_v2 500 60 24000 375 / 187
Standard_D4_v3 500 60 6000 93 / 46

You can achieve this in two ways specifying the Jenkins working directory or moving the entire HOME directory to the temporary drive.

Let us start with the first option specifying the Jenkins working directory.

As you see in the screenshot, I have set /mnt/jenkins as the working directory.

Short side note: /mnt is the mount point of the temporary drive for a Linux-based Azure VM.

Beside the working directory configuration, you add the following two commands at the beginning of the initialization script in the Azure worker node configuration section.

sudo mkdir /mnt/jenkins
sudo chown agentadmin:agentadmin /mnt/jenkins

Otherwise your builds are failing with the following error message.

Started by user admin
Running as SYSTEM
Building remotely on jenkins-workerb27e60 (azure) in workspace /mnt/jenkins/workspace/Test
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to jenkins-workerb27e60
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
    at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
    at hudson.remoting.Channel.call(Channel.java:957)
    at hudson.FilePath.act(FilePath.java:1070)
    at hudson.FilePath.act(FilePath.java:1059)
    at hudson.FilePath.mkdirs(FilePath.java:1244)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1202)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
    at hudson.model.Run.execute(Run.java:1818)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
java.nio.file.AccessDeniedException: /mnt/jenkins
  at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
  at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
  at java.nio.file.Files.createDirectory(Files.java:674)
  at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
  at java.nio.file.Files.createDirectories(Files.java:767)
  at hudson.FilePath.mkdirs(FilePath.java:3237)
  at hudson.FilePath.access$1300(FilePath.java:212)
  at hudson.FilePath$Mkdirs.invoke(FilePath.java:1252)
  at hudson.FilePath$Mkdirs.invoke(FilePath.java:1248)
  at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
  at hudson.remoting.UserRequest.perform(UserRequest.java:212)
  at hudson.remoting.UserRequest.perform(UserRequest.java:54)
  at hudson.remoting.Request$2.run(Request.java:369)
  at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

For instance, the output of a simple job after we made the changes and a new Azure worker node was instantiated.

The second option does not require to specify the working directory.

You just need to add the following commands at the beginning of the initialization script section.

sudo mkdir /mnt/jenkins
sudo chown agentadmin:agentadmin /mnt/jenkins
cp /home/agentadmin/.bash* /mnt/jenkins
HOME=/mnt/jenkins
sudo sed -i 's+/home/agentadmin+/mnt/jenkins+g' /etc/passwd
sudo mount --bind /mnt/jenkins /home/agentadmin

We ensure with those commands that tools get installed into the new HOME directory path during the worker node initialization. But only for those tools that targeting the HOME directory. Furthermore, every job that runs on this specific worker node makes use of the new HOME directory as intended.

The output of our simple job example shows us the correct configuration.


Posted

in

WordPress Cookie Notice by Real Cookie Banner