With any project, personal or for clients, I develop and build applications in Docker containers; so it would be really convenient to run these as containers in production. Usually I’ll spin up a small Ubuntu VM (GCP/Azure/AWS/Digital Ocean) and install Docker manually. I would love to run everything with Kubernetes as a container orchestrator, but the costs of such a cluster for personal usage seems rather high. Until I found this article by Remko Seelig, using Kubernetes on Google Cloud with preemptible nodes, which is about half the price of regular instances. You loose some availability but that’s probably no problem in my case.

I am willing to spend max 30$ per month on such a cluster, so I plan the following stack:

When I have too few resources, I can always add another node and stay under budget. Whenever I think the nodes under-perform, I can upgrade to 2 nodes of type n1-standard-2 (2CPU/7GB RAM, $14.60/mnth). Instead of using expensive Load Balancers to expose the nodes to the outside, I’ll implement a tool named KubeIP to assign free static IP’s to specific instances.

Setup takes the following steps:

1. Terraform deployment
2. Create firewall rules
3. Create static IP addresses and deploy KubeIP
4. Nginx-Ingress

1. Terraform deployment

How to setup a Google Kubernetes Cluster cluster with Terraform is perfectly explained in this article by Tim Bherry. I ended up with:

• a google_container_cluster without any nodes as recommended (remove_default_node_pool = true & initial_node_count = 1) and LoadBalancers disabled (addons_config { http_load_balancing { disabled = true } });
• a nodepool named ingress with node_config{preemptible = true, labels = { ingress = "true" }, tags = ["ingress"]} where the nginx-controller lives;
• a nodepool named main for non-exposed services.

2. Create firewall rules

Create a firewall rule for http/https traffic for instances tagged ingress:

gcloud compute firewall-rules create http-ingress \
    --network "default" \
    --priority 1000 \
    --direction "ingress" \
    --target-tags "ingress" \
    --source-ranges 0.0.0.0/0 \
    --allow tcp:80,tcp:443

3. Static IP addresses and deploy KubeIP

The setup is explained here; I made the following notes:

• The main nodepool can actually be used to run kubeip, by setting KUBEIP_NODEPOOL=main.
• The documentation launches 10 static IP addresses to be available for free ingress nodes, but unused static IP addresses cost around 7$/mnth each. So make equal number of static IP’s as ingress nodes.

4. Nginx-Ingress

Find my earlier post about setting up nginx-ingress. In the values.yaml of nginx-ingress you should set:

• controller.nodeSelector: ingress: "true" to run nginx-controller on the ingress nodes;
• controller.hostNetwork: true to run the deployment on the host’s network namespace;
• controller.service.type: ClusterIP since no LoadBalancer is allowed.

No external IP is bound to the nginx-ingress-controller (kubectl --namespace nginx-ingress get services -o wide -w nginx-ingress-controller) but this works any way due to a somewhat unclear reason for me.