Introduction
In this article I designed and build a 10 node PI cluster. The specs of this cluster are not bad. The cluster consists of:
- 5x Raspberry PI3B+ 4 cores 1.2Ghz (Broadcom BCM2837 Cortex-A53)
- 5x Raspberry PI4 4G 4 cores 1.5Ghz (Broadcom BCM2711 Cortex-A72)
The specs of the cluster:
Total Storage (Gb) : 320 Total RAM (Gb) : 25 Total CPU Cores : 40 Total Cpu Ghz : 13,5 Max Power consumption : 130 Watt
In this article I’m going to describe what my plans are for this cluster. And while the title is a dead give away, I’m not going into technical depths. If this article is going to be to long I’ll split the article up in multiple articles.
What to do with all this power?
Having the cluster up and running, it’s time to do something with it. One of the goals I have to learn is CI/CD pipelines. I have a small personal Gitlab server running. The goal is to use CI/CD pipelines with Kubernetes. And now that I have a 10 node cluster, this should be possible.
Running Kubernetes on Raspberry PI
While it’s possible to run Kubernetes on Raspberry PI, there are a couple of things to consider. And that is that most of the development being done is for major platforms like AMD, Intel etc. This means that for ARMhf and ARM64 a lot of things won’t work out of the box.
Another things to consider is: Kubernetes itself is not designed to run on a low powered platform like the Raspberry PI. However there are some Kubernetes flavors which focus on lightweight Kubernetes, by reducing for example the memory footprints. More on this later on. The bottom line is: Yes it is possible to run Kubernetes on Raspberry PI. There are however some caveats.
What is this Kubernetes anyway?
To get a overview of what Kubernetes is and what is does take a look at: What is Kubernetes – an overview The gist of it is that Kubernetes can be seen as a framework which allows the deployment of applications in containers, and to manage these containers and providing scaling and fail-over for the application being deployed. Also note that Kubernetes is also revered to as K8s, by stripping the 8 letters between K and S of Kubernetes. So don’t pronounce this as K eight, but pronounce it as: Kubernetes.
Starting with Kubernetes
Wanting to run Kubernetes on my Pi cluster is one thing. Actually getting to run it is quite a different story. So how to get Kubernetes running? Basically there are a couple of challenges hidden in this question. The first one is: Which flavor of Kubernetes to run, and the second one is: How to learn Kubernetes.
Which flavor of Kubernetes to run?
There are a couple of lightweight Kubernetes to choose from. Some of these allows you to run Kubernetes inside docker on your local machine for example K3d , Minicube. For the Raspberry Pi I investigated the following two:
-
- Microk8s
- K3s
Before installing Kubernetes onto a Raspberry PI notes to choose a 64bit OS. The reason for this is that most of the (docker) containers are for 64 bits OS’es. Therefore I used Ubuntu server (LTS) 64 bits for Raspberry PI. Normally I would use Rasbian, or Raspberry OS as it is called now, but since their 64bit version is still in beta during the time of writing I switched to Ubuntu server.
Let’s get started with MicroK8s
As the first candidate I installed MicroK8s. The reason for this is that the installation instruction sounds really simply. Install MicroK8s, and off you go. As it turns out, it was not that simple, and eventually I had to give up on MicroK8s, simply because I could not get is stable.
After I installed MicroK8s I noticed that after a couple of hours, the whole cluster was not responding to kubectl commands. All I get was timeout errors. Rebooting nodes was not helping. I did a lot of searching, and browsing through the issues on MicroK8s issues, and I did find that others where experience the same problems on Raspberry PI’s, but no solution.
So after two weeks of fighting I give up upon MicroK8s, and moved on to the next option
Installing K3S on Raspberry PI cluster
According to the documentation the name of K3s comes from:
We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation.
Installing K3s can be completely done by using k3s-Ansible. Which is perfect fit, since I do a lot with Ansible. After cloning the repository I Fowwloed the instruction (which basically tells you to change the host.ini file and hostvar file and to run the ansible playbook).
And right from the start I could tell that K3s was much more stable. I could deploy applications, and could remove them, and even after a couple of days the cluster was stable. Since the default version installed is v1.17.5+k3s1 I decided to upgrade the cluster. Due to my inexperience with Kubernetes, K3s and how to upgrade I manged to completely destroy my beautiful working cluster. So I started over, flashing all my SD cards with Ubuntu server, ran my own Ansible playbook to install the basics, and installed the latest version of Rancher’s K3s. And to my relieve everything works perfect. The cluster runs stable.
So in conclusion: Microk8s sounds great, and I really wanted to use it, but couldn’t get it stable to run. Keep in mind that at this point I have no experience with Kubernetes, so your mileage may vary. Thanks to Ansible-k3s repository I could get K3s up and running quickly. At this point I’m not interested in the details on how to install and configure Kubernetes. In case of K3s it’s one binary anyway. At this point I’m mainly interested in getting Kubernetes up and running and start learning on how to use Kubernetes.
Learning Kubernetes
Now that Kubernetes is up and running where to start learning? I came across the Kubernetes 101 series done by Jeff Geerling. I highly recommend watching his Kubernetes 101 YouTube video’s. I learned a lot watching these video’s and give me some basics to get me started.
In the next article I’m going to describe how I got Gitlab CI/CD pipelines working with my Kubernetes cluster.