Rails Docker App: Deployment (Kubernetes)
Kubernetes
Kubernetes is orchestration software that allows you to abstract a cluster of hosts into one virtual machine. Then try to model you app into Kubernetes atomic units called pods. Pods run your Docker Container. Kubernetes is nice b/c it’s declarative. You simple define the pods you want and how to connect them and Kubernetes will setup it up. Kubernetes is fairly low-level but it takes care of most of the work needed to run your app suite in a cluster, we will just need to setup a CI/CD tool to do our deployment scenarios.
Deployment
Deployments allow us to declaratively define a set of Kubernetes resources that we want allocated. Then Kubernetes will ensure that those services continue to run continuously. For example, if there is an error and the server restarts, Kubernetes ensures it’s discoverable in the cluster again.
Pod
If you need a guarantee that two components (containers) share the same file system (host), define them in the same pod. Other wise there is no guarantee where the k8 scheduler will allocate the container in the cluster.
Setup Cluster
Setup GCE cluster w/ at least 2 CPUs and 7.5GB memory. (Usually 2 std machines)
$ gcloud container clusters create jenkins-cd \
--num-nodes 3 \
--scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"
These scope settings are very important as they will allow our Jenkins install (described below) to authenticate w/ google using metadata.
- Ensure cluster is ready:
gcloud container clusters list
- Login:
gcloud container clusters get-credentials eggie5-blog --zone us-west1-a
- Ensure nodes are ready:
kubectl get nodes
Figure 1: Cluster of arbitrary size
Now remember, we don’t need to think in terms of VMs anymore, Kubernetes gives us a cluster modeled as 1 VM.
Figure 2: The Kubernetes cluster abstraction and the architecture we want to build.
DB Pod
Here is out DB deployment file which declaratively tells Kubernetes to run the PG docker container and to mount a persistent disk called harambi-disk
(which I provisioned via GCE).
db-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: eggie5-db
spec:
replicas: 1
selector:
matchLabels:
name: db
template:
metadata:
labels:
name: db
spec:
containers:
- image: postgres
name: db
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: prod-db-secret
key: password
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: prod-db-secret
key: username
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
- name: pg
containerPort: 5432
hostPort: 5432
volumeMounts:
# name must match the volume name below
- name: mypd
# mount path within the container
mountPath: /var/lib/postgresql/data
volumes:
- name: mypd
gcePersistentDisk:
pdName: harambi-disk
fsType: ext4
Next we will setup a service which will expose our DB on port 5432 to the cluster internally b/c the labels db
match in the above deployment file:
db-service.yml
apiVersion: v1
kind: Service
metadata:
labels:
name: db
name: db
spec:
ports:
- port: 5432
targetPort: 5432
selector:
name: db
Rails Pod
Now we will tell Kubernetes how we want our app server pod setup. Notice that declarative syntax:
web-deployment.yml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: eggie5-web
spec:
replicas: 1
selector:
matchLabels:
name: web
template:
metadata:
labels:
app: web
name: web
spec:
containers:
- name: web
image: gcr.io/eggie5-blog/blog:latest
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: prod-db-secret
key: password
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: prod-db-secret
key: username
#move this OUT
- name: RAILS_ENV
value: development
- name: RACK_ENV
value: development
- name: PORT
value: "3000"
imagePullPolicy: Always
ports:
- containerPort: 3000
name: http-server
livenessProbe:
httpGet:
path: /
port: 3000
initialDelaySeconds: 30
timeoutSeconds: 10
In the config above, we specify the docker image for the rails app and then pass in some ENV vars the rails app expects.
In our rails db config, we can now set the host to db
as defined in the database service config.
database.yml
production:
adapter: postgresql
encoding: unicode
database: mysite_production
pool: 5
host: db
username: <%= ENV["POSTGRES_USER"] %>
password: <%= ENV["POSTGRES_PASSWORD"] %>
Next, we need to expose our web pod via port 80, which we will then expose to the internet behind a load balancer.
web-service.yml
apiVersion: v1
kind: Service
metadata:
name: blog-frontend
labels:
name: web
spec:
type: NodePort
ports:
# The port that this service should serve on.
- port: 80
targetPort: 3000
protocol: TCP
# Label keys and values that must match in order to receive traffic for this service.
selector:
name: web
Ingress
K8s has a primitive called Ingress
which you can configure to act like a low-level load balancer. Optimally you would configure this yourself using nginx or something, however, since we are running GKE we can use the convenience.
This is the top level ingress point for the cluster DNS points *.eggie5.com here:
ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: eggie5-blog
annotations:
kubernetes.io/ingress.global-static-ip-name: "com-eggie5-ingress"
spec:
tls:
- secretName: tls
backend:
serviceName: blog-frontend
servicePort: 80
The Ingress
annotation is a GKE specific notation that references my static IP allocated in GCE. Also we are setting up *TLS termination using this Ingress
too. You then can setup your DNS to point to that static IP. When someone it’s the Ingress
at that IP the TLS will terminate and the request will be routed to the web pod on port 80 which was exposed by the previous Service
.
- See appendix for info on setting up TLS certs
Cluster CNC
Each Kubernetes cluster has an API to which you can send commands. The CLI is most commonly used: kubectl
. To send my declarative configs that I defined above to the cluster, I just run:
kubectl apply -f configs/
And it will send all the configs in the directory to the cluster and start modeling it. It will pull docker images from the repo and allocate pods across nodes/vms in the cluster.
If you want to start exploring deployment strategies. A common pattern is a staging and production environment. One **elegant way to do this in Kubernetes is with namespaces. This will require us to modify a lot of config files, so to look at some tools to automate the work for us: a build server that will deploy for us.
** See appendix for discussion about namespaces for stag/prod
Continuous Deployment
We will use a *** Jenkins CD/CI server that will run our build job when we push to our git repo. Here are the build/deployment steps
- Pull the branch
- Build Docker image
- Run Test Suite
- Push Docker Image to Repo
- Create respective Kubernetes namespace (stage/prod/dev)
- Update the web deployment file to reference the new Docker Image
- Apply the respective Kubernetes config files to the cluster
This is all automated using, the Jenkins
file:
node {
def project = 'eggie5-blog'
def appName = 'blog'
def feSvcName = "${appName}-frontend"
def configName = env.BRANCH_NAME
if (env.BRANCH_NAME == "production") {
namespace = "production"
configName = "production"
} else if (env.BRANCH_NAME == "master") {
namespace = "staging"
configName = "staging"
} else {
namespace = env.BRANCH_NAME
configName = "dev"
}
//def imageTag = "gcr.io/${project}/${appName}:${namespace}:${env.BRANCH_NAME}.${env.BUILD_NUMBER}"
def imageTag = "gcr.io/${project}/${appName}:${env.BRANCH_NAME}.${env.BUILD_NUMBER}"
checkout scm
stage 'Build image'
sh("docker build -t ${imageTag} .")
stage 'Run Go tests'
//sh("docker run ${imageTag} go test")
stage 'Push image to registry'
sh("gcloud docker -- push ${imageTag}")
stage "Deploy Application to ${namespace} namespace"
// Create namespace if it doesnt exist
sh("kubectl get ns ${namespace} || kubectl create ns ${namespace}")
//look into kubecfg to do this instead of sed
// kubectl set image deployment/web nginx=nginx:1.9.1
sh("sed -i.bak 's#gcr.io/eggie5-blog/blog:latest#${imageTag}#' ./Kubernetes/${configName}/*.yml")
switch (namespace) {
case ["staging", "production"]:
sh("kubectl --namespace=${namespace} apply -f Kubernetes/${namespace}/")
sh("kubectl --namespace=${namespace} apply -f Kubernetes/${namespace}/services/")
sh("echo http://`kubectl --namespace=${namespace} get service/${feSvcName} --output=json | jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")
break
default:
echo "Running 1-off deploy for branch: ${env.BRANCH_NAME} on namespace/env: ${namespace}"
// Don't use public load balancing for development branches
sh("sed -i.bak 's#LoadBalancer#ClusterIP#' ./Kubernetes/services/frontend.yml")
sh("kubectl --namespace=${namespace} apply -f Kubernetes/dev/")
sh("kubectl --namespace=${namespace} apply -f Kubernetes/services/")
echo 'To access your environment run `kubectl proxy`'
echo "Then access your service via http://localhost:8001/api/v1/proxy/namespaces/${namespace}/services/${feSvcName}:80/"
}
}
Then Kubernetes will work to build the defined architecture.
*** See appendix of instructions on how to setup Jenkins in your cluster
Appendix
One-off commands
Run rake task
kubectl exec [name_of_pod] rake db:setup
Rails Console
kubectl exec -ti [name_of_pod] rails c
Staging / Prod Environments (Namespaces)
The staging/prod deployment workflow can be handled by creating a separate cluster in GCE or you can use the suggested method of using a Kubernetes primitive called Namespaces
.
Create a new namespace for staging:
staging-namepsace.yml
kind: Namespace
apiVersion: v1
metadata:
name: staging-namespace
labels:
name: staging-namespace
kubectl create -f staging-namespace.yml
See current namespaces, K8s uses default
by default.
kubectl get namespaces
Now lets see the servcies running in staging:
kubectl get services --namespace=staging-namespace
Setting up Jenkins
Create the Jenkins Home Volume
In order to pre-populate Jenkins with the necessary plugins and configuration for the rest of the tutorial, you will create a volume from an existing tarball of that data.
gcloud compute images create jenkins-home-image --source-uri https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v2.tar.gz
gcloud compute disks create jenkins-home --image jenkins-home-image --zone us-west1-a
Create the jenkins
namespace:
kubectl create ns jenkins
Bootstrap Jenkins Install
Here you’ll create a Deployment running a Jenkins container with a persistent disk attached containing the Jenkins home directory.
First, set the password for the default Jenkins user. Edit the password in jenkins/Kubernetes/options with the password of your choice by replacing CHANGE_ME. To Generate a random password and replace it in the file, you can run:
$ PASSWORD=`openssl rand -base64 15`; echo "Your password is $PASSWORD"; sed -i.bak s#CHANGE_ME#$PASSWORD# jenkins/Kubernetes/options
Your password is 2UyiEo2ezG/dKnUcdPdt
Now create the secret using kubectl:
$ kubectl create secret generic jenkins --from-file=jenkins/Kubernetes/options --namespace=jenkins
secret "jenkins" created
Now, run the Jenkins K8s deployment files:
$ kubectl apply -f jenkins/Kubernetes/
deployment "jenkins" created
service "jenkins-ui" created
service "jenkins-discovery" created
Check that your master pod is in the running state
$ kubectl get pods --namespace jenkins
NAME READY STATUS RESTARTS AGE
jenkins-master-to8xg 1/1 Running 0 30s
Now, check that the Jenkins Service was created properly:
$ kubectl get svc --namespace jenkins
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-discovery 10.79.254.142 <none> 50000/TCP 10m
jenkins-ui 10.79.242.143 nodes 8080/TCP 10m
Kubernetes makes it simple to deploy an Ingress resource to act as a public load balancer and SSL terminator.
The Ingress resource is defined in jenkins/Kubernetes/lb/ingress.yaml. We used the Kubernetes secrets API to add our certs securely to our cluster and ready for the Ingress to use.
In order to create your own certs run:
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=jenkins/O=jenkins"
Now you can upload them to Kubernetes as secrets:
$ kubectl create secret generic tls --from-file=/tmp/tls.crt --from-file=/tmp/tls.key --namespace jenkins
Now that the secrets have been uploaded, create the ingress load balancer. Note that the secrets must be created before the ingress, otherwise the HTTPs endpoint will not be created.
$ kubectl apply -f jenkins/Kubernetes/lb
Wait for the load balancer to be ready:
$ kubectl get ingress --namespace jenkins
NAME RULE BACKEND ADDRESS AGE
jenkins - master:8080 130.X.X.X 4m
Create a pipeline
Phase 1: Add your service account credentials
First we will need to configure our GCP credentials in order for Jenkins to be able to access our code repository
- In the Jenkins UI, Click “Credentials” on the left
- Click either of the “(global)” links (they both route to the same URL)
- Click “Add Credentials” on the left
- From the “Kind” dropdown, select “Google Service Account from metadata”
- Click “OK”
You should now see 2 Global Credentials. Make a note of the name of second credentials as you will reference this in Phase 2:
Phase 2: Create a job
This lab uses Jenkins Pipeline to define builds as groovy scripts.
Navigate to your Jenkins UI and follow these steps to configure a Pipeline job (hot tip: you can find the IP address of your Jenkins install with kubectl get ingress --namespace jenkins
):
- Click the “Jenkins” link in the top left of the interface
- Click the New Item link in the left nav
- Name the project sample-app, choose the Multibranch Pipeline option, then click
OK
- Click
Add Source
and choosegit
- Paste the HTTPS clone URL of your
sample-app
repo on Cloud Source Repositories into the Project Repository field.
It will look like: https://source.developers.google.com/p/REPLACE_WITH_YOUR_PROJECT_ID/r/default - From the Credentials dropdown select the name of new created credentials from the Phase 1.
- Under “Build Triggers”, check “Build Periodically” and enter
* * * * *
in to the “Schedule” field, this will ensure that Jenkins will check our repository for changes every minute. - Click
Save
, leaving all other options with their defaults
A job entitled “Branch indexing” was kicked off to see identify the branches in your repository. If you refresh Jenkins you should see the master
branch now has a job created for it.
Other CD Solutions
Deis
Deis is esentially a private Heroku PaaS. It’s very impressive software packed up as a Helm Chart. I was able to install Deis to my cluster in less that 5 minutes and then deploy a simple rails app just like I did on heroku.
$ helm repo add deis https://charts.deis.com/workflow
$ helm install deis/workflow --version=v2.8.0 --namespace=deis
Then to deploy (just like heroku):
git push deis master
Deis is native Kubernetes. It does something pretty elegant: each app you deploy to your PaaS is modeled as a Kubernetes namespace: I thought that was a clean way to this Kubernetes primitiave.
Deis is a very high-level construct and from my initial usage the app developer is never exposed to Kubernetes. If your looking for a lower bill, this could be a smooth migration route from heroku.
Fabric8
Seems like a nice platform to handle all the CI/CD. Tried to get this setup
for a few hours on GCE but it doesn’t seem to install very well and I saw
a comment about how it doesn’t support the PVs on GCE. I also tried a managed Fabric8
install by stackpoint.io on EC2 but that install wasn’t functional (missing runtimes) either.
OpenShift Origin
Looks interesting, just found this today. Seems to be an opinionated layer on top of Kubernetes. I will look at this more
Secrets API
Storing passwords to databases and services
set the password in the cloud
kubectl create secret generic prod-db-secret --from-literal=username=postgres --from-literal=password=
Then expose the secrets via ENV vars in in the container:
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: prod-db-secret
key: password
Persistent Disk
The container file system only lives as long as the container does. So if your app state needs to survive relocation, reboots, and crashes, you’ll need to configure some persistent storage. To access more-persistent storage, outside the container file system, you need a volume. This is especially important to stateful applications, such as key-value stores and databases.
Our database pod, and each pod we create, comes w/ a ephemeral disk already mounted. However, we want something permanent to store out database files. To fix this we need to setup a persistent disk for out database and then configure our DB Pod to discover it and mount it. In K8s this is called a Persistant Volume
.
It’s easy to create a persistent disk using the Google Cloud Compute website, however, the hard part is formatting the disk as it will come unformatted. We’ll walk through the steps here:
Provision Disk:
gcloud compute disks create [DISK_NAME] --size 10 --type pd-standard --zone us-west1-a
Before March 2016 it was necessary to manually format disks. The process required staring up a GCE instance and mounting the volument and then formatting it as documented here: https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting The newly formatted disk can then be used in Kubernetes as a PV.
However, as of version 1.2.0 (March 2016), Kubernetes will automatically format the disk for you: https://github.com/kubernetes/kubernetes/pull/16948 This is a very nice convenience add. Unfortunately, a lot of literature online still tells users to mnaully format disks which is a burden to setup a stateful pod.
From the release notes: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#v120
Dynamic Provisioning of PersistentVolumes: Kubernetes previously required all volumes to be manually provisioned by a cluster administrator before use. With this feature, volume plugins that support it (GCE PD, AWS EBS, and Cinder) can automatically provision a PersistentVolume to bind to an unfulfilled PersistentVolumeClaim.
Tips
- It is a recommended practice to put resources related to the same microservice or application tier into the same file, and to group all of the files associated with your application in the same directory. If the tiers of your application bind to each other using DNS, then you can then simply deploy all of the components of your stack en masse.
Permalink: rails-docker-app-deployment-kubernetes
Tags: