Provisioning Bare Metal the K8s Way

Every time I tell someone I run Kubernetes on physical servers, I get the same look. The one that says "why would you do that to yourself."

Fair. But there are good reasons. Cloud bills get ugly fast once you're running anything sustained, especially GPU workloads. Some data legally cannot leave your building. And when you care about latency or you're squeezing every bit of performance out of NVMe and 100G NICs, a hypervisor sitting in the middle is not doing you any favors. Telco, edge, HPC, and anyone running serious AI training all end up back on metal eventually.

The problem was never whether to use bare metal. It was that provisioning it felt like it belonged to a different decade than everything else I was doing.

The part nobody wants to do

Here's what actually has to happen before a physical box can join a cluster:

• Find out what's in it: CPU, RAM, disks, NICs

• Power it on, off, or reboot it remotely

• Set boot order, PXE or UEFI, pick a boot device

• Push an OS image onto it

• Inject config via cloud-init

• Pick the root disk and partition it

• Configure RAID before the OS lands

• Set BIOS and firmware settings, maybe update them

• Later: wipe it, securely erase disks, hand it back to the pool

All of this runs through the BMC. Every server has one. iDRAC on Dell, iLO on HPE, some flavor of Redfish or plain old IPMI depending on the vendor and how old the box is. They all do roughly the same job and none of them agree on how.

I have written the shell script that SSHes into a BMC and issues IPMI commands. So has everyone else who's touched a rack. It works until you have thirty servers from two vendors and someone asks for a fourth cluster by Friday.

What Metal³ actually is

Metal³ (Metal Kubed, written Metal3 when you're not fighting your keyboard) gives you components for managing bare metal hosts with Kubernetes. You enroll your physical machines, provision an OS onto them, and deploy Kubernetes clusters on top.

The bit that took me a second to internalize: Metal3 is itself a Kubernetes application. Controllers, CRDs, reconcile loops. You describe the physical world you want and something in a pod goes and makes the racks match. Servers become objects you can kubectl get.

Detour: Cluster API

You can't really talk about Metal3 without Cluster API, so a quick recap.

CAPI is the Kubernetes project for managing Kubernetes clusters using Kubernetes. One management cluster, a set of CRDs like Cluster and Machine, and a provider that knows how to turn those objects into real infrastructure. There's a provider for AWS, one for Azure, one for GCP, one for vSphere. Write a Machine, the provider calls the cloud API, an instance appears.

That last step is the whole trick, and it's also where bare metal falls apart. There's no API endpoint for your rack. Nobody at Dell is running a control plane that will spin you up an r740 on request. The provider pattern assumes something on the other end is listening.

Metal3 is the provider that fills that gap. CAPM3 (Cluster API Provider Metal3) handles the CAPI side, and underneath it there's a stack that eventually reaches out and touches actual BMCs.

Ironic does the dirty work

Ironic is the piece at the bottom. It's an OpenStack project, written in Python, and it has been managing bare metal in production for years. It discovers nodes, catalogs them in a database, and handles the whole server lifecycle: enrolling, provisioning, maintenance, decommissioning.

The reason it matters here is that it's API-first. Ironic exposes a REST API and speaks whatever the server speaks on the other side. iDRAC, iLO, Redfish, IPMI. You make one kind of call and Ironic figures out the vendor-specific mess.

This is the single biggest reason Metal3 works. Nobody had to reimplement BMC support from scratch. Ironic already did it, badly-behaved firmware and all.

Bare Metal Operator: the bridge

BMO is the controller that translates between Kubernetes and Ironic. It watches a CRD called BareMetalHost, and one of those looks like this:

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: eselda13u31s05
spec:
  online: true
  bootMACAddress: 80:c1:6e:7a:5a:ab
  bootMode: legacy
  bmc:
    address: ipmi://192.168.1.14:623
    credentialsName: bml-ilo-login-secret-05
    disableCertificateVerification: true

That's a physical server. The BMC address, the credentials in a Secret, the MAC it PXE boots from, and whether it should be powered on.

Apply it and BMO registers the host with Ironic. Ironic powers the box on, boots a deploy ramdisk over the network, and the ramdisk reports back what it found inside. CPU count, memory, disks, NICs, all of it lands in the status of your BareMetalHost. Now you have hardware inventory as a Kubernetes resource, which means label selectors work on it, which means you can say "give me a node with at least 256GB of RAM" and mean it.

When it's time to provision, Ironic tells the ramdisk to pull an OS image over HTTP, write it to the local disk, drop a config drive next to it with your cloud-init data, and reboot into it.

The stack, top to bottom

                Cluster API
             /              \
          IPAM              IRSO
             \              /
          Cluster API Provider Metal3
                     │
            Bare Metal Operator + Ironic
                     │
              ───────┼───────
                 CPU   Disk   Network

IPAM handles IP address management, because bare metal networks usually don't come with DHCP that you're allowed to trust. IRSO manages Ironic itself as a standalone service, so the thing provisioning your servers is also declaratively managed.

Each layer has its own idea of what a node is, and the chain is worth memorizing because you'll be debugging across it:

Four names for the same server. When something goes wrong you walk down that list until you find the layer that's actually stuck. Usually it's the bottom one, and usually it's the BMC being weird.

What deployment looks like end to end

1. Stand up a bootstrap cluster. Minikube, kind, whatever. It's temporary.

2. Deploy the Metal3 stack onto it.

3. Enroll your bare metal servers by applying BareMetalHost resources.

4. Apply Cluster, control plane, and worker resources.

5. Watch the machines get provisioned.

6. Cluster is ready.

The first time I did this I sat there refreshing kubectl get bmh while servers rebooted in another room. Watching a physical machine change state because you applied a YAML file does not stop being satisfying.

Features worth knowing about

Pivoting. After the target cluster is up, you move the Metal3 controllers off the bootstrap cluster and into the managed cluster. It becomes self-managing and the bootstrap environment goes away. No permanent VM sitting in a corner that everyone is afraid to touch.

Remediation through reboot. A node stops responding, the controller power-cycles it through the BMC. Basic, and it fixes a genuinely large fraction of real incidents.

Node reuse. Keep the same physical host across upgrades and changes instead of reprovisioning from scratch every time.

Upgrades. Chained and in-place, both built on CAPI's machinery.

Automated cleaning. Disks get wiped when a host is deprovisioned, before it goes back in the pool. If you've ever inherited a server with someone else's data on it, you understand why this is non-negotiable.

Scale. Verified up to 500 nodes.

Top-of-rack switch configuration is in progress, which would close one of the last gaps where you still have to leave Kubernetes to get something done.

People actually running this

Not a research project. A partial list of who's using it in production:

• Ericsson for bare metal deployments and cluster lifecycle management

• Red Hat, which ships Metal³ inside OpenShift for automated bare metal cluster deployment

• Fujitsu, as a server vendor, using it to offer their hardware as Kubernetes bare metal nodes

• Deutsche Telekom Technik, whose "Das SCHIFF" platform uses parts of Metal³ for provisioning in a CAPI-based infrastructure

• IKEA IT AB for provisioning and lifecycle management in their CAPI-based bare metal cloud

• SUSE, as part of SUSE Edge

• Mistral AI for bare metal cluster management in Mistral Compute

• Mirantis in Mirantis Container Cloud, with plans for k0rdent and GPU workloads

The pattern I notice: telcos, server vendors, and AI companies. All three have the same problem from different directions.

Where to go next

The community is active and easy to join:

• Slack: #cluster-api-baremetal on the Kubernetes Slack

• Docs: book.metal3.io

• Mailing list: groups.google.com/g/metal3-dev

• Community meeting every Wednesday at 14:00 UTC on Zoom

• GitHub: github.com/metal3-io

• Website: metal3.io

If you want to try it without a rack, the dev-env repo will spin up virtual "bare metal" hosts using libvirt and sushy-tools, which emulates Redfish. You get the full workflow on a laptop. It's the fastest way to understand what's happening before you point any of this at hardware you care about.

Bare metal will always be more work than clicking "launch instance." But the gap has gotten a lot smaller than it used to be.