Tag: Rasberry PI Cluster

Running kubernetes on a 10 node Raspberry PI cluster

Introduction

In this article I designed and build a 10 node PI cluster. The specs of this cluster are not bad. The cluster consists of:

  • 5x Raspberry PI3B+ 4 cores 1.2Ghz (Broadcom BCM2837 Cortex-A53)
  • 5x Raspberry PI4 4G 4 cores 1.5Ghz (Broadcom BCM2711 Cortex-A72)

The specs of the cluster:

Total Storage (Gb)    : 320
Total RAM (Gb)        : 25
Total CPU Cores       : 40 
Total Cpu Ghz         : 13,5 
Max Power consumption : 130 Watt

In this article I’m going to describe what my plans are for this cluster. And while the title is a dead give away, I’m not going into technical depths. If this article is going to be to long I’ll split the article up in multiple articles.

What to do with all this power?

Having the cluster up and running, it’s time to do something with it. One of the goals I have to learn is CI/CD pipelines. I have a small personal Gitlab server running. The goal is to use CI/CD pipelines with Kubernetes. And now that I have a 10 node cluster, this should be possible.

Running Kubernetes on Raspberry PI

While it’s possible to run Kubernetes on Raspberry PI, there are a couple of things to consider. And that is that most of the development being done is for major platforms like AMD, Intel etc. This means that for ARMhf and ARM64 a lot of things won’t work out of the box.

Another things to consider is: Kubernetes itself is not designed to run on a low powered platform like the Raspberry PI. However there are some Kubernetes flavors which focus on lightweight Kubernetes, by reducing for example the memory footprints. More on this later on. The bottom line is: Yes it is possible to run Kubernetes on Raspberry PI. There are however some caveats.

What is this Kubernetes anyway?

To get a overview of what Kubernetes is and what is does take a look at: What is Kubernetes – an overview The gist of it is that Kubernetes can be seen as a framework which allows the deployment of applications in containers, and to manage these containers and providing scaling and fail-over for the application being deployed.  Also note that Kubernetes is also revered to as K8s, by stripping the 8 letters between K and S of Kubernetes. So don’t pronounce this as K eight, but pronounce it as: Kubernetes.

Starting with Kubernetes

Wanting to run Kubernetes on my Pi cluster is one thing. Actually getting to run it is quite a different story. So how to get Kubernetes running? Basically there are a couple of challenges hidden in this question. The first one is: Which flavor of Kubernetes to run, and the second one is: How to learn Kubernetes.

Which flavor of Kubernetes to run?

There are a couple of lightweight Kubernetes to choose from. Some of these allows you to run Kubernetes inside docker on your local machine for example K3d , Minicube. For the Raspberry Pi I investigated the following two:

    • Microk8s
    • K3s

Before installing Kubernetes onto a Raspberry PI notes to choose a 64bit OS. The reason for this is that most of the (docker) containers are for 64 bits OS’es. Therefore I used Ubuntu server (LTS) 64 bits for Raspberry PI. Normally I would use Rasbian, or Raspberry OS as it is called now, but since their 64bit version is still in beta during the time of writing I switched to Ubuntu server.

Let’s get started with MicroK8s

As the first candidate I installed MicroK8s. The reason for this is that the installation instruction sounds really simply. Install MicroK8s, and off you go. As it turns out, it was not that simple, and eventually I had to give up on MicroK8s, simply because I could not get is stable.

After I installed MicroK8s I noticed that after a couple of hours, the whole cluster was not responding to kubectl commands. All I get was timeout errors. Rebooting nodes was not helping. I did a lot of searching, and browsing through the issues on MicroK8s issues, and I did find that others where experience the same problems on Raspberry PI’s, but no solution.

So after two weeks of fighting I give up upon MicroK8s, and moved on to the next option

Installing K3S on Raspberry PI cluster

According to the documentation the name of K3s comes from:

We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a 10-letter word stylized as K8s. So something half as big as Kubernetes would be a 5-letter word stylized as K3s. There is no long form of K3s and no official pronunciation.

Installing K3s can be completely done by using k3s-Ansible. Which is perfect fit, since I do a lot with Ansible. After cloning the repository I Fowwloed the instruction (which basically tells you to change the host.ini file and hostvar file and to run the ansible playbook).

And right from the start I could tell that K3s was much more stable. I could  deploy applications, and could remove them, and even after a couple of days the cluster was stable. Since the default version installed is v1.17.5+k3s1 I decided to upgrade the cluster. Due to my inexperience with Kubernetes, K3s and how to upgrade I manged to completely destroy my beautiful working cluster. So I started over, flashing all my SD cards with Ubuntu server, ran my own Ansible playbook to install the basics, and installed the latest version of Rancher’s K3s. And to my relieve everything works perfect. The cluster runs stable.

So in conclusion: Microk8s sounds great, and I really wanted to use it, but couldn’t get it stable to run. Keep in mind that at this point I have no experience with Kubernetes, so your mileage may vary. Thanks to Ansible-k3s repository I could get K3s up and running quickly. At this point I’m not interested in the details on how to install and configure Kubernetes. In case of K3s it’s one binary anyway. At this point I’m mainly interested in getting Kubernetes up and running and start learning on how to use Kubernetes.

Learning Kubernetes

Now that Kubernetes is up and running where to start learning? I came across the Kubernetes 101 series done by Jeff Geerling. I highly recommend watching his  Kubernetes 101 YouTube video’s. I learned a lot watching these video’s and give me some basics to get me started.

In the next article I’m going to describe how I got Gitlab CI/CD pipelines working with my Kubernetes cluster.

An easy extensible Raspberry PI Cluster

Introduction

For quite some time I want to play around with a PI Cluster. Of course “cluster” can mean many things. A Cluster can be combining all the CPU power of the nodes, to get more processing power. A cluster can also mean having couple of nodes to build a scalable platform like OpenStack. In this case, the purpose of the cluster isn’t that important. The thing I’m mainly interested in is build a frame which can hold a couple of PI’s. And the frame should be easy to extend, so that when there is a need for more PI nodes in the cluster, the frame is can be easily extended. At the end I came up with a working frame. However this implementation might not be suitable for everyone… More on that later on.

How this all started

With a friend of my we started to build a 20 node PI cluster. The cluster was split into two: 10 Nodes lived in my home, and the other 10 nodes lived by my friends house. We used a VPN to connect the cluster nodes together. This worked great. However when we used PI3 nodes, we needed cooling. And the current frame didn’t provide that. After a attempt to alter the frame to add cooling FAN’s I realized that it might be better to start from scratch. The reason for that was during the cluster was operational, we discovered the current frame could be improved by adding some features.

Designing the ultimate cluster

Well the “ultimate” is maybe a bit strong.. but since I wanted to redesign a cluster from scratch I decided that at least the following features must be implemented:

    • The nodes must be easy to remove or easy to be inserted into the cluster
    • The frame itself should be 19 inch so it could fit in a standard network rack
    • The nodes should be powered from a own power supply, and should NOT rely on PoE
    • The frame must be easy to extend
    • Each node should have proper cooling
    • It would be nice that each node has some LED’s to display status (preferred RGB LED’s)

Tackling the power requirements

Powering each node from a power supply is the hardest part to implement. After thinking about it I came to the conclusion that I could do this by developing a back-plane. This back-plane is then used to distribute the power from one Power Supply (PSU) to all the other nodes. This sound like a great idea. However it introduces a new problem:

How can I make a easy extensible frame when there is a back-plane. The answer to that problem was quite easily actually: by splitting the back-plane up into smaller back-planes. This is how I came up with a back-plane design which can hold two PI nodes, and can be extended by adding more back-planes together. And as it turns out: splitting the back-plane up into smaller back-planes also makes the manufacturing of the PCB easier.

By using a back-plane design it’s easy to come up with a PCB for each node , which would provide the interconnect between the PI node and the back-plane. Which makes it possible to remove or insert a PI node.

Tackling the cooling

Next thing problem to tackle was: How to keep the PI node cool, so that the CPU won’t overheat and throttle. I soon came with the idea to develop a PI Cluster HAT. This Cluster HAT solves a couple of other problems:

  • The HAT can be used to hold a FAN to cool the CPU (PWM controlled)
  • The HAT can be used to distribute the 5V to the back-plane
  • The HAT can also be used for other features:
    • Hold the connections for 3 RGB LED’s so they can be controlled by GPIO pins
    • Break-out I2C
    • Break out serial RX and TX
    • Break out 3V

Developing the cluster frame

Within three months I had the first version of a 10 node cluster working. It would take me another six months to get it to a a workable version. The final design uses 3 PCB’s:

  • A back-plane PCB with can power 2 PI Nodes and which are extensible.
  • A PI Cluster HAT with a lot of features (EEProm, PWN controlled FAN, 3 RGB LEDS, I2C, 3V, 5V power lines)
  • A Power board which connects the power of the PI node to the back-plane
  • A LED Board which holds the 3 RGB LED, which connects to the PI Cluster HAT

The frame of the cluster has mainly two parts: The frame which holds the PI tray. And the PI tray which holds the PI, the PI Hat with FAN, the Power Board and Led Board. The whole design is modular.

The downside of this is: To build this cluster frame, a lot of parts are needed, and the PCB’s must be soldered. So that’s why this might not be for everyone. However…

OpenSource is the way to go

Did I mention this whole design is OpenSource? No?? Well it is. And it’s on github for everyone to download. All the 3D models, Gerber files , schematics, a full hardware assembly guide is available. You can find it here