This is the second blog post out of three in a series covering k3s a new Kubernetes distribution by Rancher.
In this post we focus on the container image build and deployment of the Azure Pipelines agent on the k3s Kubernetes cluster we have set upped in the previous post.
-> https://www.danielstechblog.io/running-a-kubernetes-cluster-with-k3s-on-raspbian/
So, how we get started with the Azure Pipelines agent container image build? Looking at the official Azure DevOps Services docs is our first step. Microsoft provides in the docs the necessary Dockerfile as well the required shell script.
-> https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#linux
Instead of using the original one I have modified the Dockerfile as stated in the notes to include zip and unzip. Also added a clean-up step of the folder /var/lib/apt/lists/ to keep the container image as small as possible. Furthermore, the base image must be exchanged against an ARM-based Linux image.
FROM arm32v7/ubuntu:16.04
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
git \
iputils-ping \
libcurl3 \
libicu55 \
zip \
unzip \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh
CMD ["./start.sh"]
-> https://github.com/neumanndaniel/kubernetes/blob/master/k3s/Dockerfile
The start.sh shell script requires one adjustment. If you just copy the content from the docs, the shell script downloads the x64 Linux Azure Pipelines agent version. We need to replace linux-x64 with linux-arm instead in line 55.
... AZP_AGENT_RESPONSE=$(curl -LsS \ -u user:$(cat "$AZP_TOKEN_FILE") \ -H 'Accept:application/json;api-version=3.0-preview' \ "$AZP_URL/_apis/distributedtask/packages/agent?platform=linux-arm") ...
-> https://github.com/neumanndaniel/kubernetes/blob/master/k3s/start.sh
The Dockerfile and start.sh file should be in the same folder before we kickoff the container image build. For the image build I am using the Azure Container Registry to build and store the final container image. We only need the Azure CLI and an ACR instance for it. The command az acr build --registry azstcr --image "azp:0.0.1" --image "azp:latest" . kicks off the build and tells ACR to tag the image as version 0.0.1 and latest.
2019/06/10 14:48:20 Downloading source code...
2019/06/10 14:48:22 Finished downloading source code
...
2019/06/10 14:48:23 Logging in to registry: REDACTED.azurecr.io
2019/06/10 14:48:24 Successfully logged into REDACTED.azurecr.io
...
2019/06/10 14:48:25 Successfully scanned dependencies
...
Sending build context to Docker daemon 6.144kB
Step 1/8 : FROM arm32v7/ubuntu:16.04
...
Step 2/8 : ENV DEBIAN_FRONTEND=noninteractive
...
Step 3/8 : RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
...
Step 4/8 : RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl jq git iputils-ping libcurl3 libicu55 zip unzip && rm -rf /var/lib/apt/lists/*
...
Step 5/8 : WORKDIR /azp
...
Step 6/8 : COPY ./start.sh .
...
Step 7/8 : RUN chmod +x start.sh
...
Step 8/8 : CMD ["./start.sh"]
...
Successfully tagged REDACTED.azurecr.io/azp:0.0.1
Successfully tagged REDACTED.azurecr.io/azp:latest
...
2019/06/10 14:50:38 Pushing image: REDACTED.azurecr.io/azp:0.0.1, attempt 1
...
2019/06/10 14:50:46 Successfully pushed image: REDACTED.azurecr.io/azp:0.0.1
...
2019/06/10 14:50:55 The following dependencies were found:
2019/06/10 14:50:55
- image:
registry: REDACTED.azurecr.io
repository: azp
tag: 0.0.1
digest: sha256:c23b87f4f0a3ff00074263e38a3be13d7995f57dde8cde7e914a8411be7fbe8a
runtime-dependency:
registry: registry.hub.docker.com
repository: arm32v7/ubuntu
tag: "16.04"
digest: sha256:dfb3a42ab9f344522fcea900ae0ee359ca3b54c1e67a167064f210b0f0efa778
git: {}
...
Run ID: cgy was successful after 2m36s
After the container image build, we create an agent pool and PAT (Personal Access Token) in Azure DevOps.
-> https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops#creating-agent-pools
-> https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops#authenticate-with-a-personal-access-token-pat
Both steps are required to create the Kubernetes secret for our Azure Pipelines agent deployment.
We need to create two Kubernetes secrets in total to be able to do the deployment of the Azure Pipelines agent on our k3s Kubernetes cluster.
First, we create the secret for the agent itself that contains the Azure DevOps URL to our organization and the PAT.
kubectl create secret generic azp --from-literal=AZP_TOKEN=REDACTED --from-literal=AZP_URL=https://dev.azure.com/REDACTED/
Finally, to pull the container image from the Azure Container Registry we create the Docker credentials secret.
kubectl create secret docker-registry acr --docker-server=REDACTED.azurecr.io --docker-username=REDACTED --docker-password=REDACTED --docker-email=REDACTED@azurecr.io
We are now ready for the Azure Pipelines agent deployment. For that I have created a template file which will be applied onto the k3s Kubernetes cluster via kubectl.
[] > kubectl get pods -l app=azp-agent -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES azp-agent-57bb7cb4f6-xngph 1/1 Running 0 3m38s 10.42.0.11 k3s-master-0 <none> <none>
-> https://github.com/neumanndaniel/kubernetes/blob/master/k3s/azpagent.yaml
The only things you need to modify in the template file are the values for the imagePullSecrets, the image and the environment variable AZP_POOL for the Azure DevOps agent pool name.
...
spec:
imagePullSecrets:
- name: acr
containers:
- name: azp-agent
image: REDACTED.azurecr.io/azp:0.0.1
imagePullPolicy: IfNotPresent
env:
...
- name: AZP_POOL
value: k3s
...
Now run kubectl apply -f azpagent.yaml to deploy the agent. The startup phase of the agent can be followed by running kubectl logs -f azp-agent-57bb7cb4f6-xngph.
When everything works as expected the Azure Pipelines agent should show up in Azure DevOps.
[] > k3s logs azp-agent-57bb7cb4f6-xngph 1. Determining matching Azure Pipelines agent... 2. Downloading and installing Azure Pipelines agent... 3. Configuring Azure Pipelines agent... >> End User License Agreements: Building sources from a TFVC repository requires accepting the Team Explorer Everywhere End User License Agreement. This step is not required for building sources from Git repositories. A copy of the Team Explorer Everywhere license agreement can be found at: /azp/agent/externals/tee/license.html >> Connect: Connecting to server ... >> Register Agent: Scanning for tool capabilities. Connecting to the server. Successfully added the agent Testing agent connection. 2019-06-08 22:33:37Z: Settings Saved. 4. Running Azure Pipelines agent... Starting Agent listener interactively Started listener process Started running service Scanning for tool capabilities. Connecting to the server. 2019-06-08 22:33:47Z: Listening for Jobs