When the Typhoon open-source Kubernetes distro first introduced experimental ARM64 support in v1.19.4 (Nov 2020), it was limited to Fedora CoreOS custom images, a few AWS regions, and the flannel CNI provider. But with open-source work across projects, Kubernetes on ARM64 has come a long way.
Today, Typhoon can spin up Kubernetes clusters that use ARM64 nodes for the control plane and worker nodes (“full-cluster”). You can pick from container-optimized OSes (Fedora CoreOS or Flatcar Linux), use any AWS region or Azure region (Flatcar only) and choose between Cilium (default), Calico, or flannel CNI providers.
module "gravitas" {
source = "git::https://github.com/poseidon/typhoon//aws/fedora-coreos/kubernetes?ref=v1.25.1"
# AWS
cluster_name = "gravitas"
dns_zone = "aws.example.com"
dns_zone_id = "Z3PAABBCFAKEC0"
# configuration
ssh_authorized_key = "ssh-ed25519 AAAAB3Nz..."
# optional
arch = "arm64"
networking = "cilium"
controller_type = "t4g.small"
worker_type = "t4g.small"
}
Or you can run a “mixed-arch” cluster by adding ARM64 nodes to an otherwise x86/amd64 cluster (what Poseidon Labs uses) as shown in Typhoon’s docs.
History
In this post, we’ll explore Typhoon’s road to supporting ARM64 broadly, looking at three main pillars:
- Platform availability
- Operating system support
- Kubernetes components
Platform
Typhoon supports creating Kubernetes clusters on AWS, Azure, DigitalOcean, Google Cloud, and on-prem. Of these, only AWS and on-premise had ARM64 options in 2019, when ARM work began.
AWS was broadly used and AWS’s Graviton2 (and now Graviton3) instance types provided an accessible target that matched our budget and that of our customers. AWS was chosen as the platform to target for Kubernetes ARM64 clusters and groundwork began in early 2020.
Azure made Ampere Altra ARM64 instances generally available Sept 2022 and Flatcar Linux has begun publishing OS images there. EDIT: Typhoon v1.25.3 (Oct 2022) extended ARM64 support to Flatcar Linux on Azure.
Recently, Google Compute Engine introduced Tau T2A machine types. We’re excited to see another cloud vendor getting into ARM and look forward to Fedora CoreOS or Flatcar Linux adding OS images.
In future, we’d like to extend Typhoon bare-metal / on-prem support to ARM64. However, challenges remain in finding hardware that fits our strict budget, meets Poseidon Lab’s needs at our colocation facility, and is representative of hardware in use by our customers - from small labs to enterprises. In short, we need datacenter-grade ARM64 machines to be ubiqitous to create a robust market of pre-owned hardware akin to current x86 options.
OS Images
Fedora CoreOS AMIs
In Nov 2020, neither Fedora CoreOS nor Flatcar Linux published ARM64 AWS AMIs, so Poseidon Labs built Fedora CoreOS AMIs from source and published unofficial images to us-east-1, us-east-2, and us-west-1. To bootstrap, the first Fedora CoreOS AMI was built (very slowly) on a Raspberry Pi! Those unofficial Fedora CoreOS AMIs were then used to build subsequent AMIs. This allowed Typhoon v1.19.4 (Nov 2020) to introduce experimental ARM64 support.
In Aug 2021, Fedora CoreOS started publishing official ARM64 AMIs, so Typhoon v1.23.0 (Dec 2021) switched nodes to using official Fedora CoreOS AMIs (controllers, workers).
Fedora CoreOS is now shipping 64 bit ARM artifacts! https://t.co/KFbi6oHQLu
— Fedora CoreOS (@FedoraCoreOS) October 1, 2021
Flatcar Linux AMIs
In Dec 2021, the Flatcar Linux folks stabilized and published ARM64 images with Flatcar 3033.2.0+.
cc @dghubble as I know your eyes were on Flatcar+ARM for a while now
— Vincent Batts (@vbatts) December 15, 2021
With Typhoon v1.23.3 (Jan 2022), an arch option became available on Flatcar Linux clusters on AWS.
Components
At a minimum, provisioning an ARM64 machine as a Kubernetes node requires core components be compiled or built to run on ARM64 (aarch64).
kube-system
Kubernetes components like kube-apiserver, kube-controller-manager, kube-scheduler, and kube-proxy were compiled and built as multi-arch container images already. Nice job by upstream Kubernetes there.
Next, we needed the Kubelet and CNI providers to run on ARM64.
Kubelet
Since Kubernetes v1.18.0 (Mar 2020), upstream stopped packaging Kubelet as a container image. Instead, the poseidon/kubelet project bundles upstream Kubelet releases into multi-arch container images, suitable for use on container-optimized operating systems. The first multi-arch Kubelet image is shown below (now quite out of date).
In early 2020, Typhoon itself didn’t yet provide multi-arch Kubernetes clusters, so the Poseidon build system (which runs on Typhoon) used a combination of buildah and qemu-user-static to bootstrap the very first kubelet multi-arch images. Today, the Poseidon Labs build system uses Typhoon ARM64 nodes to more efficiently build multi-arch container images for all our projects.
CNI
In 2020, only the flannel CNI published multi-arch container images, so if ARM64 was used, Typhoon’s Calico and Cilium CNI options could not be used.
Cilium v1.10 (Apr 2021) started building multi-arch container images, so Typhoon v1.21.2 (Jun 2021) relaxed CNI requirements for those wishing to use Typhoon with ARM64. There were some problems with the Fedora CoreOS SELinux policies interacting with Cilium (during cgroups v1 to v2 migration), but those were ironed out by Typhoon v1.22.0 (Aug 2021).
Finally, Calico v3.22.0 (Jan 2022) started building multi-arch container images, so Typhoon v1.23.5 (Mar 2022) clusters and beyond using Calico can leverage ARM64 nodes as well.
Going Further
There’s still much future work to extend Kubernetes ARM64 support to all of Typhoon’s platforms, but the work is paying off:
At Poseidon Labs, investing in ARM64 support has improved our own Typhoon clusters, decreased build times, and cut costs. And as an open-source Kubernetes distribution, Typhoon spurs broader ARM64 adoption and multi-arch app development.
At enterprises where I’ve done Kubernetes ARM64 work, AWS Graviton instances and mixed architecture Kubernetes clusters (like those provided by Typhoon) have reduced compute usage and batch processing times, increased spot availability, and saved millions of dollars.
Excited about Kubernetes on ARM64?
- Follow @poseidonlabs for our blog posts or @typhoon8s for Typhoon distro news
- Consider supporting Poseidon’s open-source work by joining our amazing sponsors
- Get help building multi-arch applications or using ARM64 nodes in your Kubernetes clusters. Email tech@psdn.io about a consultation
