Deploying Rails on Kubernetes
How to deploy and orchestrate a Rails 6 application with Kubernetes in Google Kubernetes Engine (GKE)
Kubernetes hellonode tutorial is great, but it didn’t answer many questions that I had when building and deploying an actual app such as “how does my database get setup?”, “when do app packages get installed?”, “how does the developer environment setup work?” and “how do my containers talk to each other?”. In Part I of this walkthrough, we created a docker-compose developer environment and a fresh new Rails 5 app. In this continuation of the walkthrough, I’ll explain the answers I found for those questions and how to continuously deploy that application with high availability, a self-healing mechanism, redundancy, and autoscaling, load balancing, service discovery, rolling deploys, and health checks.
The production environment
This is where things get interesting (fun).
In an ideal setup, our application has a load balancer between the internet and the nodes serving web requests, it autoscales up and down with increased traffic or idleness, self-heals by detecting dead or poor performing nodes and recreating them from scratch, it would be able to to connect to all other services in our cluster in an easy and clear way, and it has a continuous rolling deploy pipeline so new code can be introduced with zero downtime. Ideally, it would have all of these things without a dedicated team building each feature in the cloud provider we happen to be a customer of. The goal of this tutorial is to have said features without building, scaffolding, configuring, and connecting them from scratch—allowing Kubernetes to do the heavy lifting by fully utilizing its rich feature set.
Let’s talk about the major differences in how our application runs in production as opposed to development.
In development…
- Our containers all run on the same host (the machine we’re using).
- We probably only have 1 of each container (because more than one isn’t necessary).
- We can talk to the webservers directly. There is no need for a load balancer.
- Our database data needs to outlive our mysql container, but it doesn’t need to outlive our developer environment.
- We’ll probably run our app with
rails s -b 0.0.0.0in development for simplicity. - None of our containers need to be exposed to the public internet.
- None of our containers need to be health checked, because they aren’t running all of the time and we don’t need 100% uptime.
- Code changes are reflected immediately because we mounted a volume between our host and our app container.
- Development database credentials are not secret, thus we don’t need a solution for storing these in a safe manner.
In production…
- Our containers (rails web server and mysql) will likely run on different hosts (nodes).
- We want to scale our web server containers up and down with traffic (>1) but have only one mysql container.
- All traffic to the web server containers needs to be load balanced.
- Our database data must be available to the mysql container, but exist separately so we don’t lose it when the container is destroyed or moved from one node to another.
- The load balancer for our web servers needs to be exposed to the internet (Ingress).
- Our containers need to respond to health checks so we have an indication of when they aren’t healthy anymore and need to be recreated or taken out of load balancing.
- When new code is deployed, it should replace the old code in a rolling fashion—so there is zero downtime.
- Some of our containers will need secrets—like database credentials—available to them.
Kubernetes offers everything we listed here and more. The rest of this guide will walk through each step of the process of deploying our rails application from Part I to Google Kubernetes Engine (GKE).
Setting up a Google Cloud Project
First, you’ll need to setup a google cloud account. This conveniently connects to your google account of choice. You can sign up here. Next, go to console.cloud.google.com to create a new project. A project groups all of the infrastructure, clusters, billing and everything the console shows you. You can think of this similar to an AWS account, except that you can have many projects in Google Cloud and everything is separate (joy).
When you create your project, remember the project ID that’s given to you. — we’ll need this later.
Great. You’ve got a project. Remember the ID; mine is rails-kube-demo.
You’ll need to enable billing, but a Google Cloud account comes with $300 of credit to play with, so lean on it.
The next thing you’ll need is the Google Cloud SDK. That will provide the gcloud CLI. After you’ve installed the Google Cloud SDK, use the command below to install kubectl. kubectl is the tool we’ll use to manage all of the kubernetes infrastructure.
gcloud components install kubectlIf you’d like to run your cluster in a specific region, use gcloud config set compute/zone {your favorite zone here}to set gcloud to use your region of choice for all commands. I’ll be using us-central1-b for this tutorial.
The hard part is over. Now we’re ready to set up the infrastructure.
Creating a cluster
A cluster in google cloud is a group of nodes (VMs) with Kubernetes and Docker pre-installed and ready to go. After a cluster is created, it is immediately available for use with Kubernetes. Kubernetes Master, which kubectl talks to and controls the orchestration, will run on one of the nodes and our containers will be scheduled on all of the nodes. Google Container Engine does all of this for us. We need only to create a cluster.
We can create a cluster via the command line with the following:
gcloud container clusters create rails \
--enable-cloud-logging \
--enable-cloud-monitoring \
--machine-type n1-standard-2 \
--num-nodes 1This command creates a cluster of n1-standard-2 type VMs with monitoring and logging enabled. The name of the cluster will be “rails”. The command should hang until the cluster creation is complete.
When the cluster creation is complete, we’ll now need to give the cluster credentials to kubectl to begin deploying containers. Use the command below to provide the credentials to kubectl where “rails” is the name of your cluster (from the previous command).
gcloud container clusters get-credentials railsIf you get an authentication or permissions error, try running gcloud auth login and allow the permissions requested.
Try running kubectl cluster-info to check on the status of your cluster. We don’t need these endpoints, but if this command did not execute successfully, you’ll want to fix the issue before continuing.
Our production mysql container will have high demands of the node that is scheduled on. We don’t want it to have to share resources with web containers. If a web container was very busy serving web requests, it might prevent our database container from executing queries and performing well. Since the database is often a bottleneck and we will only run one mysql container, this next step describes how to setup a pool of nodes in GKE that only our database container is allowed to use.
gcloud container node-pools create db-pool \
--machine-type=n1-highmem-2 \
--num-nodes=1 \
--cluster=railsThe above command will create a pool of nodes named db-pool in GKE. That pool will contain one node (VM) and the pool will exist within our rails cluster, which we’re using for the rest of our deployment.
When the pool creation is done, you should see something like
Creating node pool db-pool...done.
Created [https://container.googleapis.com/....].
NAME MACHINE_TYPE DISK_SIZE_GB NODE_VERSION
db-pool n1-highmem-2 100 1.4.5Deploying MySQL
Now that our cluster is running, we can deploy some containers.
CAVEAT: running mysql as a container, even with an orchestration tool, may not be a production-quality solution for you or your business. Many people advise against deploying your database as a container, but for the purposes of this tutorial (and perhaps that it meets your needs), I will instruct you how to do so.
In preparation for some files we’ll be creating, make a kube directory in your project root. We’ll store our kubernetes config objects there for reference.
We’ll start by deploying the mysql container. The mysql container will use the value of an ENV variable called MYSQL_ROOT_PASSWORD when mysql initializes for the first time. Because we want this password to be secret and it needs to be set for our container, but not stored in our container, we’ll create a kubernetes secret object for the username and password that we’ll use for the mysql root account and we’ll allow some containers access to this secret.
Creating Secrets
Secrets can be exposed to a container via a data volume or environment variables. For our example, we’ll want these secrets as environment variables so we can use them in our application config.
First, encode your username and password as base64 and save this for later:
% echo -n 'root' | base64
cm9vdA==
% echo -n 'my secure password' | base64
bXkgc2VjdXJlIHBhc3N3b3Jk
% docker-compose run --rm app rake secret | base64
YzcwZTYyODdhYmIzNjE1NzI4MjkwYTA1ZjNmZDlkM2NlYWU2NzIzNGY0ZDFlYTc2OTQyODFkOTM0MTczNWEwYzA0NWEzYjAxZGVkMDEyYjBhODQxNzhmNDAxMjY2OTA2ZDJjMDM2ZTY3MWQ1MzZkNDZhZDhiZGVlOGEwYjQ5ODY=We’ll then create a file named app-secrets.yml in the ./kube folder and the contents should be the like the following:
# ./kube/app-secrets.ymlapiVersion: v1
data:
mysql_user: cm9vdA==
mysql_password: bXkgc2VjdXJlIHBhc3N3b3Jk
secret_key_base: YzcwZTYyODdhYmIzNjE1NzI4......=
kind: Secret
type: Opaque
metadata:
name: app-secrets
ProTip™: Use your own unique secret_key_base.
Use kubectl create -f kube/app-secrets.yml to create the secret object in kubernetes master. After this is successful, you can delete this file. (If you commit this file, you will be exposing your credentials to whoever has access to your source control. Base64 is not encryption). You can edit the file later with kubectl edit secret/app-secrets. Your EDITOR will open and you can edit the details and send the updates straight to kubernetes master in the cluster.
You should see secret “app-secrets” created. Kubernetes master now holds this object and can make this secret available as ENV variables to any container we specify.
Creating a persistent disk
In number 4 of our production requirements we stated that we wanted our database data to live independently so that it doesn’t disappear when a container is destroyed or moved. To accomplish this, we’ll create a persistent disk in Google Compute to hold our mysql data. The persistent disk outlives any clusters, nodes, and containers. Persistent disks can also be resized later. Use the following command to create a persistent disk with the name db-data and 300GB of storage in GCE (you can resize this later):
gcloud compute disks create --size 300GB --type pd-ssd db-dataWe’ll use this disk later as a volume for our mysql container.
Creating the MySQL deployment
Now we’re finally ready create a Deployment for our mysql container. A deployment is a combination of a couple of things in Kubernetes. It’s a replication controller, which controls the perpetual existence of a set number of pods. Pods are an atomic group of containers (usually 1) that travel (need to exist) together. Our web pod (web server containers) will be separate from our mysql pod (mysql container). The replication controller will ensure n number of our pods are running at all times on whatever node they fit on. A deployment also includes the setup for rollouts. A rollout is when a new container image is specified for existing pods and the containers running the old image need to be replaced with the new ones in a rolling fashion. Kubernetes will handle a zero-downtime rollout of the new container image to replace all existing, older-image containers. For instance, you could deploy a minor version update to mysql via a rollout, ensuring that your database is running at all times and a newer version replaces the existing container running mysql.
The Kubernetes spec file for our mysql deployment looks like this:
# ./kube/mysql-deployment.ymlapiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mysql
labels:
name: mysql
spec:
replicas: 1
template:
metadata:
labels:
name: mysql
spec:
nodeSelector:
cloud.google.com/gke-nodepool: db-pool
containers:
- image: mysql:5.6
name: mysql
resources:
requests:
cpu: 800m
limits:
cpu: 800m
env:
- name: MYSQL_DATABASE
value: app_production
- name: MYSQL_ROOT_USER
valueFrom:
secretKeyRef:
name: app-secrets
key: mysql_user
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: app-secrets
key: mysql_password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
# This name must match the volumes.name below.
- name: mysql-db-data
mountPath: "/var/lib/mysql"
volumes:
- name: mysql-db-data
gcePersistentDisk:
# This disk must already exist.
pdName: db-data
fsType: ext4
replicas: 1 means we want 1 pod running at all times.containers: is an array of containers that should be run for this pod. There is only one and it’s image is the canonical mysql:5.6 from the docker registry. It requests 80% of one CPU core (15% goes to kube-system pods. For ENV variables, we specify three values: MYSQL_DATABASE (the value for which is set explicitly), MYSQL_ROOT_USER, and MYSQL_ROOT_PASSWORD (the values for which are gathered from a kubernetes secret we created earlier named app-secrets. The container exposes port 3306, on which mysql runs, so we must indicate this to kubernetes. Finally, this pod requires a volume named mysql-db-data mounted at /var/lib/mysql, the definition for which is at the bottom where we indicate that a gcePersistentDisk can be found with the name db-data and should be treated as an ext4 filesystem.
The volumes definition is very important here. The db-data disk must be attached to the VM node that our mysql container will run on—and because kubernetes will ultimately decide which node in the pool the database will run on, we don’t know which node to attach the disk to. This is okay, because kubernetes will automatically attach that persistent disk to the node it has decided to schedule the mysql pod on before it runs the mysql pod itself. If our mysql pod gets scheduled or recreated on a different node, the disk will be attached before it arrives. Isn’t that nice?
Save this YAML to a file in your ./kube folder named mysql-deployment.yml. You can now create the mysql deployment with kubectl create -f kube/mysql-deployment.yml.
Run kubectl get pods and you should see your pod running after a minute or so:
NAME READY STATUS RESTARTS AGE
mysql-2390497038-gysw7 1/1 Running 0 59sCreating the MySQL service
We’re almost done with mysql. Now we need to expose mysql to the rest of the cluster so other containers can easily communicate with the mysql pod. This is accomplished with a kubernetes Service. The spec is fairly simple:
# ./kube/mysql-service.ymlapiVersion: v1
kind: Service
metadata:
name: mysql
labels:
name: mysql
spec:
ports:
- port: 3306
selector:
name: mysql
This defines a service in our cluster named mysql which Kubernetes will make available to all pods in our cluster by resolving DNS lookups for mysql to any pods matching the selector name=mysql and send traffic to the container on port 3306.
Use kubectl create -f kube/mysql-service.yml to create this service. It should now be returned when you run kubectl get services:
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.3.240.1 <none> 443/TCP 1hWe’re done with mysql!
ProTip™: if you’d like to connect to your mysql pod from your local machine, try kubectl port-forward [pod name] 3306:3306 and connect to localhost:3306 with a mysql client. The pod name will be available via kubectl get pods. Wow!
Now that the mysql pod is running and we know we can reach it via mysql:3306. We can update our rails application’s database.yml to look like this:
# config/database.ymldefault: &default
adapter: mysql2
encoding: utf8
pool: 5
username: root
password: root
host: mysqlproduction:
<<: *default
database: app_production
username: <%= ENV['MYSQL_USER'] %>
password: <%= ENV['MYSQL_PASSWORD'] %>
Where MYSQL_USER and MYSQL_PASSWORD are ENV vars we’ve made available in the bottom of our database deployment spec. The hostname mysql will resolve to the mysql pod. app_production is the database that was automatically created when mysql started for the first time because we set a MYSQL_DATABASE ENV variable on that container in our mysql deployment spec. If you skipped this step, run rake db:setup in one of your web pods after setting them up in the next section.
Deploying the application
To deploy pods of our application container for serving web requests, we’ll need a container image to run, a deployment, a load balancer that exposes our web pods to the public internet, and an automatic autoscaler to increase or decrease the number of web pods running to appropriately handle the incoming web traffic for good measure. The web container will be running phusion passenger / nginx, which we set up in Part I of this guide. This container will run our app and handle web requests. Our web pod will consist of just the one container—our phusion passenger application image.
Before we can deploy our application container, we must build it and upload it to a docker registry so that our GKE nodes can easily download it and run it.
Let’s build our docker container from the application root. The example application I’m using can be found here. As outlined in Part I, we’re running the phusion-passenger-ruby23:0.9.19 image and our Dockerfile has been written such that when it’s done building the container, the container will be ready to run in production. This includes bundle install and rake assets:precompile as those are both necessary before running the application.
# ./DockerfileFROM phusion/passenger-ruby23# set some rails env vars
ENV RAILS_ENV production
ENV BUNDLE_PATH /bundle# set the app directory var
ENV APP_HOME /home/app
WORKDIR $APP_HOME# Enable nginx/passenger
RUN rm -f /etc/service/nginx/down# Disable SSH
# Some discussion on this: https://news.ycombinator.com/item?id=7950326
RUN rm -rf /etc/service/sshd /etc/my_init.d/00_regen_ssh_host_keys.shRUN apt-get update -qq# Install apt dependencies
RUN apt-get install -y --no-install-recommends \
build-essential \
curl libssl-dev \
git \
unzip \
zlib1g-dev \
libxslt-dev \
mysql-client \
sqlite3# install bundler
RUN gem install bundler# Separate task from `add . .` as it will be
# Skipped if gemfile.lock hasn't changed
COPY Gemfile* ./# Install gems to /bundle
RUN bundle install# place the nginx / passenger config
RUN rm /etc/nginx/sites-enabled/default
ADD nginx/env.conf /etc/nginx/main.d/env.conf
ADD nginx/app.conf /etc/nginx/sites-enabled/app.confADD . .# compile assets!
RUN bundle exec rake assets:precompileEXPOSE 3000CMD ["/sbin/my_init"]
This Dockerfile references a couple of configs in ./nginx in our project root directory. These are the nginx configs for the web server. You can read more about what they contain in the Phusion Passenger Docker Image Docs. Example config files that are required fro this tutorial can be found here. Place them in a folder named nginx in the project root.
If you’re using a different dockerfile, your application should respond to a GET /_health HTTP request with a 200 status code if everything is okay for your app and be listening on 3000 (you can change 3000 everywhere you see it to your specific port if you need to). We will use this endpoint for the kubernetes health check.
Run docker build -t app . where app is the name you’d like to use for your container. I’ll continue to use app for the remainder of this tutorial. It may be helpful to be more specific with the name.
Pushing the container to the registry
Since we’ll upload this container to the private google cloud container registry that comes with our google cloud account, we need to tag the image with our project name and container name. The format is as follows: us.gcr.io/$PROJECT_ID/$CONTAINER_NAME:$TAG. My project ID is rails-kube-demo and my container name is app, so I will run docker tag app us.gcr.io/rails-kube-demo/app:v1 to tag an existing container named app that I’ve built. You could also use the git commit SHA as the tag, but v1 will suffice for now.
When your container is done building, you can push it to the google cloud registry with gcloud docker push us.gcr.io/rails-kube-demo/app:v1 where us.gcr.io/rails-kube-demo/app:v1 is what we just tagged our container with. You should see something like this:
The push refers to a repository [us.gcr.io/rails-kube-demo/app]
1a3012e3f756: Pushed
3372d3e3f26a: Pushed
10b93f2e7983: Pushed
d2015d8207ae: Pushed
044d2c25c0a4: Pushed
6583778e1b31: Pushed
5d90245e8929: Pushed
b567022f1893: Pushed
fadbdb7f7da6: Pushing [==========> ] 47.72 MB/79.08 MB
dd5bad579675: Pushing [=====> ] 19.77 MB/39.29 MB
02471478283d: Pushed
55bbe78d1c49: Pushed
28ba3922517b: Pushing [====> ] 64.81 MB/429.4 MB
872e268735cb: Pushed
5f70bf18a086: Pushed
0184e31d4eba: Pushing [====> ] 28.3 MB/92.83 MB
19a8383d6948: Pushed
0738910e0455: Pushed
21df36b5c775: Pushed
315fe8388056: Pushed
7f4734de8e3d: Pushing [====> ] 10.24 MB/124.1 MBWhen it’s done, your container is now available for your kubernetes nodes to download and run.
Creating the web deployment
The web deployment is the pod specification for our rails app web servers. This works much like the mysql deployment, but we’ll want more than one web server running. This pod will also need the app-secrets secret, which you can see near the bottom of the spec.
Our web deployment spec will look like this:
# ./kube/web-deployment.ymlapiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: web
labels:
name: web
spec:
replicas: 2
template:
metadata:
labels:
name: web
spec:
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
containers:
- name: web
image: us.gcr.io/rails-kube-demo/app:v1
ports:
- containerPort: 3000
livenessProbe:
httpGet:
path: /_health
port: 3000
initialDelaySeconds: 30
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /_health
port: 3000
initialDelaySeconds: 30
timeoutSeconds: 1
env:
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: app-secrets
key: secret_key_base
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: app-secrets
key: mysql_user
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: app-secrets
key: mysql_password
Our label for this container is web, we want 2 replicas of our application container, our container will expose port 3000 (where nginx is listening inside the container), we want kubernetes to request the /_health to check if this container is alive (liveness), we want it to hit the same endpoint to see if the container is ready to be added to the load balancer (readiness), and the app-secrets should be exposed to this container as ENV variables so it can connect to mysql and use the SECRET_KEY_BASE. In order for kubernetes to know that your application is alive and your web server is listening, the application needs to respond to a GET /_health HTTP request with a 200 status code if everything is okay. Add this endpoint to your app. We’ve used the nodeSelector option to indicate that this pod has an affinity for the default-pool node pool (so it doesn’t get scheduled on our db-pool nodes which are reserved for our mysql pod). Save this file as ./kube/web-deployment.yml and create the objects with kubectl create -f kube/web-deployment.yml. Check kubectl get pods and use kubectl describe pod [pod name] to check the status of that pod deployment. If there’s an issue, describe pod will show it at the bottom and you can use kubectl apply -f kube/web-deployment.yml to apply changes that you’ve made to your spec.
Troubleshooting
If your pod status is “Running” you’re good to go. If the pod “Ready” is 0/1 it means the pod is not passing the readiness check. This can sometimes take a minute, but if you’re having trouble or your pod isn’t ready or running, check that it’s able to schedule and start correctly. First, use kubectl describe pod [pod name] to see the kubernetes activity log for that pod. At the bottom of the output is the most recent event. If your container is having trouble starting, you can use kubectl exec -it [pod name] bash to open an interactive bash shell and try running rails s -p 4000 and seeing if the app starts or rails c to see if a console can be started. If an error is thrown, address that error. You may have to fix some code, build and push a new docker container of your app (v2). To deploy that image, change the image value in your web-deployment.yml and run kubectl apply -f kube/web-deployment.yml. Kubernetes will notice the image has changed and start rolling out that new container. The new nodes should begin ContainerCreating.
At this point, our web pods are created and running. They’re passing the liveness and readiness check when kubernetes requests HTTP GET /_health, but we don’t have a way of hitting those pods from the external internet yet. Fortunately, a kubernetes Service can also act as a load balancer. Let’s create a Service that exposes and load balances our web pods.
Exposing the web pods
First, we must allocate a static IP address from google compute. We will eventually point our DNS record to this IP address so that users can access our application. This IP address must be static, because our DNS will not know when it changes. We can reserve a static IP from google cloud with the following command: gcloud compute addresses create app-external — region=us-central1 where us-central1 is the region I prefer and app-external is a name I’d like to give this address.
The output will look something like this:
---
address: 130.211.166.211
creationTimestamp: '2016-11-12T14:56:06.857-08:00'
description: ''
id: '2218600693701943529'
kind: compute#address
name: app-external
region: us-central1
selfLink: https://www.googleapis.com/compute/v1/projects/rails-kube-demo/regions/us-central1/addresses/app-external
status: RESERVEDAnd the address is what we’ll use in our kubernetes spec of our web service. Use it for the loadBalancerIP directive below:
# kube/web-service.ymlapiVersion: v1
kind: Service
metadata:
name: web
labels:
name: web
spec:
type: LoadBalancer
# use your external IP here
loadBalancerIP: 130.211.166.211
ports:
- port: 80
targetPort: 3000
protocol: TCP
selector:
name: web
Add this to a new file at kube/web-service.yml and use kubectl create -f kube/web-service.yml to create the load balancer. selector: {name: web} indicates that we would like to exposes all containers matching the name=web selector behind this load balancer. The load balancer will accept TCP connections on port 80 and will forward those connections to port 3000 on the containers where nginx is running. Kubernetes handles running multiple pods on the same node even if more than one pod wants the same port (3000). It does so by mapping a random port to the container’s 3000 and the Service keeps track of where there are containers running and which ports will actually reach those containers’ port 3000. This is a ton of heavy lifting done for us.
Talking to the computer
You can now navigate to the external IP address we allocated earlier and reach your rails application. If you’re using the sample application in GitHub, the interaction would look like this:
% curl 130.211.166.211
<!DOCTYPE html>
<html>
<head>
<title>App</title>
<meta name="csrf-param" content="authenticity_token" />
<meta name="csrf-token" content="56FjA/PNPQG+0n2YUZqgCZif5K7VQgSrgnEkFdskpEvBno2p0rnwNRr16VFG4gAzQW6c3R4A4zLrgg2u6rwLPw==" /><link rel="stylesheet" media="all" href="/assets/application-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.css" data-turbolinks-track="reload" />
<script src="/assets/application-12006f97c59529ad7801f7c7b1ea6d03b973c67d135af5f60c22b0ad1531190b.js" data-turbolinks-track="reload"></script>
</head><body>
Hello!
</body>
</html>
Success!
Scaling up
We mentioned easily being able to scale your containers—it’s as easy as this:
% kubectl scale deployments/web --replicas 10
kubectl gdeployment "web" scaled% kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-2390497038-gysw7 1/1 Running 0 17h
web-4092422605-2mk6q 0/1 ContainerCreating 0 1s
web-4092422605-64gz1 1/1 Running 0 16h
web-4092422605-axxmo 0/1 ContainerCreating 0 1s
web-4092422605-iop0i 0/1 ContainerCreating 0 1s
web-4092422605-sow0m 0/1 ContainerCreating 0 1s
web-4092422605-stvag 0/1 ContainerCreating 0 1s
web-4092422605-t28nb 0/1 ContainerCreating 0 1s
web-4092422605-wofpe 0/1 ContainerCreating 0 1s
web-4092422605-y4uy8 1/1 Running 0 16h
web-4092422605-zhpv1 0/1 ContainerCreating 0 1s
Kubernetes immediately starting scheduling 8 more pods of our webserver. Since there’s an underlying ReplicationController for our web deployment, kubernetes will now ensure that 10 pods are running at all times. Kubernetes will schedule those pods on nodes in the cluster. When it runs out of room on those nodes, the status will be Pending. To see details about what may have gone wrong scaling up, you can see the details of a pod with kubectl describe pod [pod name]. Within seconds we should have 8 more containers running. When they respond to the readiness health check, they will be added to the web service load balancer and their Ready column will say 1/1.
Autoscaling
Autoscaling is extremely simple. Use the spec below to create a HorizontalPodAutoscaler, which continuously checks the metrics of our choice and determines if more or fewer pods need to be running.
# ./kube/web-autoscaler.ymlapiVersion: extensions/v1beta1
kind: HorizontalPodAutoscaler
metadata:
name: web
namespace: default
spec:
scaleRef:
kind: Deployment
name: web
subresource: scale
minReplicas: 2
maxReplicas: 10
cpuUtilization:
targetPercentage: 70
Create this with kubectl create -f kube/web-autoscaler.yml and you’re good to go. Our threshold for scaling is based on CPU in this example, so when the web servers become very busy, new pods will spin up.
Deploying new code
Deploying the code for the first time is half the battle. Deploying new code is easy with kubernetes (if you’re like me, the first couple containers you rolled out didn’t work and you’ve already rolled out new containers before this step). If you make a code change, you can simply rebuild your container with the same command we used the first time docker build -t app . where “app” is the name I’ve chosen for my container.
Once this container is built, you can tag it. This time, tag it with an incremental bump to the tag number. We originally used v1, and now we’ll use v2. These tags will play an important role when rolling back to old code (old containers) and knowing which container (git revision!) is currently running in your pods. After your container is built, tag the container like so: docker tag app us.gcr.io/rails-kube-demo/app:v2 and push it to the google registry with gcloud docker push us.gcr.io/rails-kube-demo/app:v2. Remember when we used that image name and tag in the web-deployment.yml? Open that file back up and change the image: value to the image name you just pushed. In our case, we’ll change the :v1 to :v2. Use kubectl apply -f kube/web-deployment.yml and kubernetes will automatically start rolling out your new container. I usually use watch -n1 'kubectl get pods' to watch the rollout occur. That’s it! Your new code will be running as soon as the image is downloaded to the nodes and your new pods take the place of your old ones.
Deploy tasks
What we know about rails deployments is that there are a few steps that need to occur during deployments. A tool like Capistrano comes in handy for these tasks, but in our case, most of the heavy lifting is done by kubernetes. The two distinct tasks that need to occur during a deployment are rake db:migrate and rake assets:precompile (okay, you might have a third, rake db:seed). Because our Dockerfile precompiles assets inside the container, our precompiled assets will come with the code we deploy.
The migrate task is much different—it must be run at some point during the deploy process before the new code starts running (or maybe you have backwards/forwards compatible migrations, which is great, but let’s assume the common scenario). The general process of a rails application deploy is
- Deploy new code
rake db:migratewith the new code- Compile assets for the new code
- Restart the app processes.
There are probably many ways that you could accomplish these tasks with kubernetes, but this is what I’ve found to work very well:
- Build a new container of new code.
- Compile assets inside the container—prepare the container to be deployed.
- Put one of these new containers in the cluster and run
rake db:migrateand any other deploy tasks inside that one container. - Rollout the new containers to replace all existing app containers.
Step three sounds complicated, but it really isn’t. Kubernetes has an object type called Job, which will come in handy for this task. A Job creates a pod for a specific purpose (command) and removes that pod when it’s done. The Job pod will use our newly-created container image (with our new code and our migration files) and run a command inside of it (rake db:migrate) right before we rollout that new container to the web pods. We’ll call this job deploy-tasks, as it will be responsible for the tasks we intend to run before we deploy.
First, we’ll need a script for the deploy-tasks job to run. Create a file like the following in a folder named script in your project root (script/deploy-tasks.sh):
#!/bin/bashset -ebundle exec rake db:migrate
# add anything else you need here!
Make that file executable with chmod +x script/deploy-tasks.sh
Now we’ll create a kubernetes spec for the deploy-tasks job. The spec for the job will look very similar to our web deployment spec, as that pod is very similar—both need the app-secrets. When we deploy, we’ll probably automate the creation of this kubernetes job and the container image: will be different each time. For that reason, we’re going to use envsubst and an ENV variable to make this job spec reusable. For image: we’ll just put $IMAGE and we’ll interpolate the actual image name before we send this to kubernetes. The command for this container is also different than our web spec. This pod will execute the ./script/deploy-tasks.sh script we just wrote. I’ve appended .tmpl extension to the end to remind me that this is a template and must be interpolated before it can be created in kubernetes.
# kube/deploy-tasks-job.yml.tmplapiVersion: batch/v1
kind: Job
metadata:
name: deploy-tasks
spec:
template:
metadata:
name: deploy-tasks
labels:
name: deploy-tasks
spec:
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
restartPolicy: Never
containers:
- name: deploy-tasks-runner
image: $IMAGE
command: ["./script/deploy-tasks.sh"]
env:
- name: SECRET_KEY_BASE
valueFrom:
secretKeyRef:
name: app-secrets
key: secret_key_base
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: app-secrets
key: mysql_user
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: app-secrets
key: mysql_password
ProTip™: envsubst is available for mac osx in the gettext package. You can install it with brew install gettext and then link it with brew link --force gettext.
Now we just need a script to tie it all together. I’ve been using the following (script/deploy.sh):
#!/bin/bash# exit on any error
set -eexport PROJECT=rails-kube-demo
export NAME=app
export TAG=$1
export IMAGE="us.gcr.io/$PROJECT/$NAME:$TAG"echo "deploying $IMAGE"# cleanup any stale deploy-tasks jobs
kubectl delete job deploy-tasks 2&> /dev/null || true# create the deploy-tasks job by creating a pod, running the migrate script
cat kube/deploy-tasks-job.yml.tmpl | envsubst | kubectl create -f -while [ true ]; do
phase=`kubectl get pods -a --selector="name=deploy-tasks" -o 'jsonpath={.items[0].status.phase}' || 'false'`if [[ "$phase" != 'Pending' ]]; then
break
fi
doneecho '=============== deploy_tasks output'
kubectl attach $(kubectl get pods -a --selector="name=deploy-tasks" -o 'jsonpath={.items[0].metadata.name}')
echo '==============='while [ true ]; do
succeeded=`kubectl get jobs deploy-tasks -o 'jsonpath={.status.succeeded}'`
failed=`kubectl get jobs deploy-tasks -o 'jsonpath={.status.failed}'`if [[ "$succeeded" == "1" ]]; then
break
elif [[ "$failed" -gt "0" ]]; then
kubectl describe job deploy-tasks
kubectl delete job deploy-tasks
echo '!!! Deploy canceled. deploy-tasks failed.'
exit 1
fi
done# deploy web containers
kubectl set image deployments/web "web=$IMAGE"kubectl describe deployment webkubectl delete job deploy-tasks || truekubectl rollout status deployment/web
It’s invoked with script/deploy.sh v2 where v2 is just the tag of the image I’m about to deploy. The variables at the top should be changed to fit your project. The script will create the deploy-tasks job, attach to it, and when it’s done, rollout the new image to the web pods.
So start to finish, the deploy process to build a container of the existing code and roll it out (with deploy-tasks) is as follows:
docker build -t app .
docker tag app us.gcr.io/rails-kube-demo/app:v2
gcloud docker push us.gcr.io/rails-kube-demo/app:v2
script/deploy.sh v2(see the Makefile in the example project for reference)
CI
The build and deploy step can be automated quite easily in your CI service. I followed This excellent guide to setup continuous delivery in CircleCI using the above deploy script. The result is a continuously integrated rails application that builds the container on the CI machine, tags it with the git revision, pushes that container to the google cloud private registry, and runs the deploy script from the CI container—using Kubernetes to trigger a rollout of the new container.
Retrospective
After spending the time to learn some of the details and newer features of Kubernetes, I couldn’t be happier with the rich set of features it offers. If you’re looking for a container orchestration tool—and maybe you deploy smaller services than a full rails/mysql stack—I confidently stand by this tool. The application is load balanced across all web pods, autoscaled and auxiliary services like redis and memcached can be added with extreme simplicity to our cluster, old revisions of the code are a rollout away and kubectl is delightfully fast. I’ve happily had a Rails application, selenium grid, mysql, and a grape application running in production for months with almost zero downtime.
If you’re interested in monitoring, check out the dd-agent DaemonSet spec to run dd-agent on every node in your cluster, sending health metrics for every node, pod, and kube-system service to Datadog with practically zero configuration.
If you have questions, please ask. I’ll be happy to help where I can. See the sample app for an example of the exact Rails 5 application I created with Part I and II of this walkthrough.
I found the following links particularly helpful:
