Are You Thinking What I'm Thinking? - Deploying .NET Core app to Kubernetes using GoCD

Deploying .NET Core app to Kubernetes using GoCD

Jul 172018

It's been a long time since I have written my last post. In this period, I dig into Kubernetes mostly. Kubernetes is a deployment automation system that manages containers in distributed environments. It simplifies common tasks like deployment, scaling, configuration, versioning, log management and a lot more.

In this article, you will find how can a dotnetcore app put into kubernetes using blue-green deployment and using the pipeline as code. In this case, I used GoCD and their yaml plugin: https://github.com/tomzo/gocd-yaml-config-plugin

First of all, you have to dockerise your dotnetcore app. Here is a snippet for example.

FROM microsoft/dotnet:2.0.5-sdk-2.1.4 AS build-env

WORKDIR /workdir
COPY . /workdir

RUN dotnet restore ./WebApp.sln
RUN dotnet test ./src/tests/WebApp.IntegrationTests
RUN dotnet test ./src/tests/WebApp.UnitTests
RUN dotnet publish ./src/WebApp/WebApp.csproj -c Release -o /publish

FROM microsoft/dotnet:2.0.5-runtime
WORKDIR /app
COPY --from=build-env ./publish .

EXPOSE 3333/tcp
CMD ["dotnet", "WebApp.dll", "--server.urls", "http://*:3333"]

After that, put a "kubernetes" folder in your Project's root. Folder structure can be like this:

- kubernetes
    --  deployment.yaml
    --  service.yaml
    --  switch_environment.sh
- src
    ....
- ci.gocd.yaml
- Dockerfile
- WebApp.sln

Your "deployment.yaml" should be like this :

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: webapp-${ENV}
spec:
  replicas: ${PODS}
  template:
    metadata:
      labels:
        app: webapp
        ENV: ${ENV}
    spec:
      containers:
      - name: webapp
        image: yourdockerregistry:5000/webapp:${IMAGE_TAG}
        resources:
          requests:
            cpu: "750m"
        ports:
        - containerPort: 3333
        readinessProbe:
          tcpSocket:
              port: 3333
          initialDelaySeconds: 15
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /status
            port: 3333
          initialDelaySeconds: 15
          periodSeconds: 10
      terminationGracePeriodSeconds: 30
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: webapp-${ENV}
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: webapp-${ENV}
  minReplicas: 10
  maxReplicas: 25
  metrics:
  - type: Pods
    pods:
      metricName: cpu_usage # Metrics Comming From Prometheus. List of metrics : kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
      targetAverageValue: 0.6 # If average pod CPU over %50, Pods will be scaled.

In this snippet, you will see some ENV variables for parametric values like image tag, deployment environment, blue-green deployment etc..
You can also use helm for rolling deployments, version bump-ups but I will use much more simple thing: "envsubst"

The other mechanism is horizontal scaling in the cluster. You can merge deployment and scaling in one yaml.
In this instance, I used K8s' custom metric API.

Take a look if you wanna this or just skip it: https://github.com/stefanprodan/k8s-prom-hpa

And the service.yaml should be like this :

apiVersion: v1
kind: Service
metadata:
  name: webapp-svc
spec:
  type: NodePort
  ports:
  - port: 3333
    nodePort: 30333
    targetPort: 3333
    protocol: TCP
    name: http
  selector:
    app: webapp
    ENV: ${ENV}

We will use K8s' selectors in order to get blue-green switch for deployments. The selector object will take the suitable pods and bind into service.
And I used nodePort because of binding services to Load Balancer externally.

You can bind like this :
AGENTIP1:30333 http://servicedns.com
AGENTIP2:30333 http://servicedns.com
AGENTIP3:30333 http://servicedns.com

You don't have to give each agent's IP to load balancer because K8s have also internal Load-Balancing. (That's not a good approach. Managing in Loadbalancer in K8s simply better)

Your "switch_environment.sh" file can be like this.

#!/bin/bash

if [ -z "$1" ]
  then
    echo "No argument supplied"
    exit 1
fi

if ! kubectl get svc $1
  then
    echo "No service found : ${1}"
    exit 1
fi

ENVIRONMENT=$(kubectl describe svc $1 | grep ENV | awk '{print $2}' | cut -d"," -f1 | cut -d"=" -f2)

if [ $ENVIRONMENT == "blue" ]; then
    ENV=green envsubst < service.yaml | kubectl apply -f -
    echo "Switched to green"
else
    ENV=blue envsubst < service.yaml | kubectl apply -f -
    echo "Switched to blue"
fi

After all, bind all these items in one gocd.yml file.

format_version: 2
environments:
  WebAPI:
    pipelines:
      - webapp-build-and-push
      - webapp-deploy-to-prod-blue
      - webapp-deploy-to-prod-green
      - webapp-switch-environment

pipelines:
  webapp-build-and-push:
    group: webapp
    label_template: "1.1.${COUNT}"
    materials:
      project:
        git: http://github.com/example/webapp.git
        branch: master
        destination: app
    stages:
      - buildAndPush:
          clean_workspace: true
          jobs:
            buildAndPush:
              tasks:
               - exec:
                  working_directory: app/build-scripts
                  command: /bin/bash
                  arguments:
                    -  -c 
                    - './build-and-publish.sh'

  webapp-deploy-to-prod-blue:
    group: webapp
    label_template: "${webapp-build-and-push}"
    materials:
      webapp-build-and-push:
        type: pipeline
        pipeline: webapp-build-and-push
        stage: deploy
      project:
        git: http://github.com/example/webapp.git
        branch: master
        destination: app
    stages:
      - build:
          approval:
            type: manual
          clean_workspace: true
          jobs:
            build:
              tasks:
               - exec:
                  working_directory: app/kubernetes
                  command: /bin/bash
                  arguments:
                    -  -c 
                    - 'ENV=blue IMAGE_TAG=$GO_PIPELINE_LABEL PODS=10 envsubst < deployment.yaml | kubectl apply -f -'
               - exec:
                  working_directory: app/kubernetes
                  command: /bin/bash
                  arguments:
                    -  -c 
                    - 'kubectl rollout status deployment webapp-blue'

  webapp-deploy-to-prod-green:
    group: webapp
    label_template: "${webapp-build-and-push}"
    materials:
      webapp-build-and-push:
        type: pipeline
        pipeline: webapp-build-and-push
        stage: deploy
      project:
        git: http://github.com/example/webapp.git
        branch: master
        destination: app
    stages:
      - build:
          approval:
            type: manual
          clean_workspace: true
          jobs:
            build:
              tasks:
               - exec:
                  working_directory: app/kubernetes
                  command: /bin/bash
                  arguments:
                    -  -c 
                    - 'ENV=green IMAGE_TAG=$GO_PIPELINE_LABEL PODS=10 envsubst < deployment.yaml | kubectl apply -f -'
               - exec:
                  working_directory: app/kubernetes
                  command: /bin/bash
                  arguments:
                    -  -c 
                    - 'kubectl rollout status deployment webapp-green'

  webapp-switch-environment:
    group: webapp
    label_template: "${COUNT}"
    materials:
      webapp-build-and-push:
        type: pipeline
        pipeline: webapp-build-and-push
        stage: deploy
      project:
        git: http://github.com/example/webapp.git
        branch: master
        destination: app
    stages:
      - build:
          approval:
            type: manual
          clean_workspace: true
          jobs:
            build:
              tasks:
               - exec:
                  working_directory: app/kubernetes
                  command: /bin/bash
                  arguments:
                    -  -c 
                    - './switch_environment.sh webapp-svc'

Now, You have 4 pipelines:

webapp-build-and-push
webapp-deploy-to-prod-blue
webapp-deploy-to-prod-green
webapp-switch-environment

You can define your build script to build, dockerise the application.
If you have Test, Staging environments, put them in the "gocd.yaml" too. (In order to simplify, I removed those lines)
That's it! After that, you have :

Dockerised dotnetcore app
Kubernetes Deployment pipelines
Blue-Green Switch Pipeline which controls kubernetes service (You have to configure kubectl for gocd agents)
Horizontal Pod Autoscaler (CPU based autoscale mechanism in the cluster)