Another Raspberry Pi Cluster
by erick.truter in Circuits > Raspberry Pi
6801 Views, 64 Favorites, 0 Comments
Another Raspberry Pi Cluster
I recently build a Raspberry Pi cluster in a custom case running K3S Kubernetes. A lot of positive feedback after sharing some images of the final design, and people asking questions about the build. I decided to share the process of making it and my thoughts during the build. Hopefully this will answer some of the questions and help anyone else thinking of building one. All the designs are my own and built for educational purposes.
Supplies
1x Netgear GS110MX Wireless-N 8 Port Gigabit Unmanaged Switch w.2x10g
1x Mean Well RS-150-5 Power Supply (130W 5VDC 26A)
1x HiBay DC12V Power Supply Transformer 18W 1.5A
1x 2.1mm DC plug to bare wire lead
8x 90 Degree Right Angle USB Type C Charging Cable(30cm)
8x Cat6 Ethernet Patch Lead (0.5m)
4x GeeekPi Raspberry Pi Low-profile CPU Cooler
4x GeeekPi Raspberry Pi ICE Tower Cooler
8x Type A Female USB To DIP 2.54MM PCB Board Adapter Breakout
32x Red M2.5 Aluminum Alloy Threaded Sleeve Hexagon (M2.5xH5xL6)
32x Red M2.5 Aluminum Alloy Threaded Sleeve Hexagon (M2.5xH5xL8)
32x M2.5 Stainless Hex Socket Button Head Allen Bolts Screws (10mm)
32x M2.5 Stainless Grub Screws Cup Point Hex Soocket (10mm)
32x M2.5 Stainless Steel Button Nut & Bolt Screws Allen Hex (6mm)
4x M3 A2 Stainless Countersunk Socket Screw (M3x6mm)
4x M3 Pozi Pan Machine Screw Bolt Stainless Steel (45mm)
1x Phillips/Micro/Rounded Self-Tapping Repair Screws (kit)
1x 5ml Syringe with 18G 1.5'' Blunt Tip Needles for Experiments, Industrial Use (10 Pack)
1x Loctite 406, Low Viscosity Liquid Adhesive, High Strength Adhesive for Plastics
1x TENSOL 12 Acrylic Adhesive 20 Gram Bottle Perspex Bonding Glue Cement
1x Silicone Electrical Wire Cable Black/Red Spool
1x 15A 250V Rocker Switch Power Socket Inlet Module Plug 5A Fuse Switch with 3Pcs 18 AWG Wiring 3 Pin
3M™ Scotch Water Resistant Blue Masking Tape (18mm)
Nylon Screw Nut Washer Hex Nylon Screws Assortment Kit
Pespex
3D Print PLA Filament
The Idea
As a big fan of the Raspberry Pi, I am always keen for a new project, especially where I can learn something new. Raspberry Pi clusters is not a new thing and has been on the edge of my radar for a long time. This was something I knew about and understood in theory, but never really triggered my interest enough to dedicate an entire project to.
As the cloud is becoming more mainstream and businesses are moving processes or functions into the cloud as microservices, learning more about Kubernetes seemed to be a good idea. It was at the same time I stumbled upon Jeff Geerling’s videos on YouTube, thank you Google algorithms!
He has a great video series of running a Raspberry Pi cluster on a development board called the Turing Pi. It is effectively a board that you can plug Raspberry Pi compute modules into and thus a neat solution for a cluster. I was late to the party as this was designed for the CM3 module, and at the time, the CM4 was already out, and the Turing Pi board was sold out. So cool idea, but I can’t get hold of the hardware so I will park this, maybe in the future?
Then Google and YouTube decided to share another video and this time it was the Water-Cooled Raspberry Pi 4 Cluster by thediylife. This was a very cool design, and I really admired the look of it. After I could not get my hands on a Turing Pi, I decided we will go old school and go standard Raspberry Pi boards to build my cluster.
The Design
I wanted to design something that would look amazing and be functional. Naturally I had some grand designs which would have been a challenge to build. I probably toyed with ideas for about 3-4 weeks before settling on my design. I must add, I rent, so going for a large design was not going to be practical.
I also wanted everything to be neatly enclosed but still look impressive. I wanted the network layer visible and especially accessible so I could pull the cable on a node to simulate failure. Finally, I wanted it to be easily moved around and enclosed it to keep it clean. I did consider the standard way of stacking the Pi’s to build a cluster, but I preferred the look thediylife produced.
This was my final design which ticked all the boxes for me. I could slot in the network switch in the front to monitor activity and interact with ports to simulate failure. All the cables could be neatly hidden inside together with the power source allowing me to have a single power cable coming out of the back of the enclosure, keeping it neat.
The Materials
Choosing the materials to build the cluster enclosure was tricky for me. As I mentioned I rent a small flat so doing anything that requires a workspace would be difficult. So, what were my options? I considered off the shelf options such as Micro PC cases, Display cases (not the PC kind), small boxes, anything that I could modify to make my idea work. I was drawing blank on most ideas. Often the initial cost to buy something, then potentially to cut and modify it, put me off doing it.
I thought about plywood, laminate or even MDF (super wood). Again, this would require cutting, sanding, painting, and then add some clear cover so you can see the Pi’s. I kept coming back to the idea of a Perspex display box, something like what people put sports memorabilia or collectables in. But even something off the shelf was not cheap and, in most cases, did not fit dimensions I would be happy with. I thought, “how hard can it be”, and looked at getting some Perspex cut to size and building it myself.
The 3D Parts
I love using 3D printing to prototype and quickly throw something together to see if it will work. The two main parts needed was, something to hold the switch in place and in such a manner that I could take it out if needed, and secondly to bring the Perspex parts together on the base of the build like a frame.
For this I designed something simple that would mainly support the switch and form the base corners of the build. I like to cut my 3D models up into more easily printable parts. Not only does this make it easier to print but allows me to quickly make changes to only one or two pieces without having to reprint the entire piece.
Ensuring that the 3D base and the Perspex parts would bond properly I added a 10mm standoff at the bottom between the main black case and the accent colour pieces around the bottom. My thinking was this would help the bond in more directions (maybe?). I probably ended up with 3 draft designs in the end.
Perspex
I found a company that will cut Perspex to size and post you the pieces. This made my life a lot easier as I could just use my design to work out what pieces I need to order. The one thing that was a problem was the L shaped sides. I asked the company if they could cut me a custom shape but the quote for just those two pieces ended up more than the entire order of rectangular pieces would cost.
In the end I ordered everything in rectangular shapes and just manually cut the two L shape pieces required for the build. As a friend said, I could stuff up 11 times and still break even, rather than pay them to cut it for me. Now I fully understand my tiny order is not worth their time so there is no hard feelings towards them.
I was worried about cutting the slots for the network switch and cables. My first thought was to drill the corners and then use the Dremel with a cutting disk to cut the slot. My concern was the disk would run away and either damage the Perspex or go skew. In the end I decided to use the Dremel with a drill stand and mill out the slots. This seemed safer than a spinning disc that could run away, my opinion at the time. In an ideal world a laser cutter would have been sweet.
The pieces ordered:
- 2x Black and White Acrylic Sheet - Black - 360mm x 150mm x 5mm (top/bottom mount)
- 2x Black and White Acrylic Sheet - Black - 225mm x 200mm x 3mm (sides to be cut L shape)
- 1x Black and White Acrylic Sheet - Black - 354mm x 75mm x 3mm (front)
- 1x Black and White Acrylic Sheet - Black - 354mm x 222mm x 3mm (back cover)
- 1x Black and White Acrylic Sheet - Black - 354mm x 50mm x 3mm (top)
- 1x Colour Perspex Sheet - Dark Red - 366mm x 75mm x 3mm (front accent)
- 2x Colour Perspex Sheet - Dark Red - 203mm x 75mm x 3mm (side accent)
- 1x Colour Perspex Sheet - Dark Red - 360mm x 75mm x 3mm (rear accent)
- 1x Colour Perspex Sheet - Dark Red - 386mm x 226mm x 5mm (base)
- 2x Clear Acrylic Sheet - Clear - 366mm x 160mm x 3mm (front/back cover)
- 2x Clear Acrylic Sheet - Clear - 200mm x 160mm x 3mm (side cover)
- 1x Clear Acrylic Sheet - Clear - 366mm x 206mm x 3mm (top)
The Glue
This was the thing that slowed down my project. Knowing which glue to use was key as I did not want my project to fall apart. I am using two main materials namely Perspex and PLA filament so I needed something that will not only bond strong but work on both materials. And getting the correct answer was near impossible.
After a fair bit of research, I narrowed down my choices to Contact Adhesive (CA Superglue), Plastic Epoxy resin or Perspex Bonding Cement. So why not just pick the latter? Well couple of unknowns, one, it is mainly (if not exclusively) used for Perspex to Perspex. Secondly it needs two smooth surfaces to properly bond, which is not 100% true of 3D printed models. Thirdly it is not a “filling” glue, so it does not fill the gaps for a better bond. Lastly what will it do to PLA?
It is safe to assume where I will glue Perspex to Perspex, I will use the cement to bond them together. For the Perspex to PLA, I did three tests using, a two-part plastic Epoxy, CA, and cement. I printed a small 3D rectangular shape, took some scrap Perspex, and glued all three variations and left them for 24 hours
The results in my case, the Perspex Cement was firm but with mild effort I could separate the Perspex form the 3D print. The two-part Epoxy was better and took a bit more effort to separate. Lastly was the CA glue which stuck, and it is still stuck. My decision was made for the Perspex to PLA to use the CA glue.
The CA I chose was Loctite 406 Low Viscosity which according to the website is ideal for plastics. The cement I used for the Perspex to Perspex was TENSOL 12 Acrylic Adhesive. This stuff it strong so please use a well-ventilated area and you will need extra Syringes with blunt tip needles.
A video about gluing acrylic : https://youtu.be/hT6Ow_cBTps?t=14
Assembly
For days I would go over my assembly of the Perspex and the 3D printed parts. I was aware that once the glue hits the surface there was no going back. I needed to work out a plan of what to glue first so I would not be impeded in any way when it to assembly. In other words, it was going to be a tight fit and getting to the mounting screws would be a challenge if I forget something.
There was a lot of tape used to make mock-ups of the parts to see what should be glued first. At this point I discovered a major flaw. Each Perspex piece was 2mm out from my dimensions. Luckily this worked out for the Perspex, but I had to adjust one half of my 3D base to accommodate the 2mm offset.
Then the assembly order was, firstly, glue the 3D base parts onto the base Perspex. As described previously, these were in four pieces that formed the left and right section holding up the network switch. I taped down the two pieces I would not glue, then glue the corresponding pieces in place. Once set I would remove the tape and glue the remaining two pieces. Critical thing here was that I had holes in the base to access the bolts used to keep the switch in place. They had to be aligned and not get any glue in there.
After that it was gluing the front cover where the switch would fit through and side L shapes. Again, using tape with the back cover in place to ensure everything aligns and try to keep everything square as possible. I paused here to start working on the two pieces where I would mount the Pi’s on. Once that was complete the last steps were just to glue these two main pieces and the accent pieces which doubled as the base for the cover.
Tip: I did not glue the bottom Pi mounting piece until the very end to ensure I could easily access the wiring.
Mounting Pi’s
When you order a GeeekPi ICE Tower cooler it comes with a piece of Perspex, brass standoffs and screws, to neatly mount your Pi onto. I wanted to use this look in the way I mounted my Pi, but instead of using the kit contents, I wanted to have colours that matched my project.
For the Perspex plate under each Pi I used red tinted clear Perspex. This was again ordered in rectangular pieces which I had to round the edges and drill the mounting holes for the Pi. I made two 3D templates, one for the rounded edges so I can sand away the corners until the evenly fit through the template, and a second one to fit them into so I can drill the holes for the Pi.
I needed extra set of standoffs to raise each Pi plate from the base. I managed to find colour standoffs, but all were just through hole. This complicated matters as you need to stack them. I found some M2.5 hex grub screws I used to stack these standoffs.
To drill the holes, I again used a 3D printed template to ensure they are close to perfectly spaced. Then used the Dremel drill press with a 2.5mm bit to drill the holes. The mounting order was:
- One M2.5 10mm hex bolt though the base bottom into 8mm standoff
- One M2.5 10mm grub screw into the 8mm standoff
- Place the tinted Perspex over the grub screw
- Mount one 6mm standoff on top of the Perspex onto the grub screw
- Place the Pi onto the standoff and screw in one M2.5 brass standoff with screw (included in kit)
- Lastly mount the GeeekPi ICE Tower cooler and screw in a M2.5 6mm hex screw.
If you look at the GeeekPi ICE Tower cooler instructions it is very similar.
Cables and Power
Space was always going to be a premium in this project. I had to power 8 Raspberry Pi 4’s at the recommended 3A per Pi. That would be a recommended 24A at 5V which equates to at least 120W. I found this Mean Well switching power supply that could deliver up to 26A at 130W.
For the network switch I needed a simple 12V 1.5A power supply but I wanted something that took up minimal space. For this I used the extra slim HiBay 18W 12Volt transformer. I ordered a DC barrel plug cable with it.
For power going into the case, I used a simple kettle plug switch with leads and fuse. This made it neat as only one cable into the rear for power. Also triples the fuses, one in PSU, one in switch and one in kettle plug.
For the network cables I ordered standard Cat6 Ethernet Patch Lead 0.5m. I was worried if I bought shorter cables that they might not flex as well. With the sightly longer than needed cables I could loop them inside and position them more in line with the Pi’s.
I found some lovely 90-degree USB C power cables that would take up less space when plugged in so I could stack the Pi’s closer together and not have huge gaps between them. Inside I build a USB A power breakout rail that these plug into which in turn is wired to the PSU.
Why to PoE? Two main reasons, money and it won’t look as cool. Yes, I did sacrifice practicality over aesthetics but I would have had to pay more to get a PoE switch with at least 8 powered ports. Secondly this larger switch and PoE hats would have added more to the project costs. And who doesn’t want LED fans?
Cooling
Cooling is still a hot topic in the community. Let’s be honest, yes, you don’t need to have any cooling on your Pi if you run it vanilla without overclocking it and well ventilated. In normal use I could easily get the CPU temperature around the 60 Celsius mark. Idle Pi’s can run a lot cooler but seeing I had no idea how large I am going to scale my cluster, I had to assume 60 degrees plus.
If I were to run PoE, this would limit my cooling options. Yes, the hat can accommodate a fan, but this would be less efficient. Now with the PoE option removed I looked at what could be done. Liquid cooling was a good idea, but I would worry about leaks. It was a choice between a massive heat sink or a more active cooling option with a fan.
I already had a GeeekPi ICE Tower cooler (reasons) and really liked how it looked. It does a great job of keeping temperatures down so it was only natural that I would pick having these to cool the nodes. I use both the tower and low-profile models which just looks better in my opinion.
A good video by Michael Klements on Comparing Cooling Solutions On A Raspberry Pi 4 - Is Water Cooling Worth It?
Also ETA Prime reviewing the Ice Tower - The Best Raspberry Pi 4 Cooler! The Ice Tower Also works on the Pi 3 & Pi 3B+
Cluster
After all was set and done, I could power up the whole thing and watch all the Pi come alive. I must confess I did setup the Pi Cluster before I built the case as I first wanted to see if I can get it running and if it will be worth the build. When the Pi’s was powered on in the case for the first time, the cluster was ready to go.
For the cluster I used Jeff’s guide to install K3S Kubernetes on Raspbian OS 64 bit. I will leave links to his guide below but the method used was similar, the difference being using basic Raspberry Pi's and not the Turning Pi :
Raspberry Pi Cluster Ep 1 - Introduction to Clustering
Raspberry Pi Cluster Ep 2 - Setting up the Cluster
Raspberry Pi Cluster Ep 3 - Installing Kubernetes (K3s) on the Turing Pi
Raspberry Pi Cluster Ep 4 - Minecraft, Pi-hole, Grafana + MORE!
Raspberry Pi Cluster Ep 5 - Benchmarking the Turing Pi
Raspberry Pi Cluster Ep 6 - Turing Pi Review
I hope this gave an overview of how went about building the cluster enclosure. I did not want to do a step by step as it is really just printing some parts and gluing Perspex to it and other Perspex, cutting some slots, drilling some holes and putting it all together. Now I will focus on playing with the cluster and deploying some applications to it.