This milestone continues a journey we started last year when we introduced support for an additional hardware vendor. With Dell adding to the list of supported hardware, metal-stack takes another step toward broader hardware compatibility — giving users greater flexibility when setting up bare-metal private cloud environments and reducing dependency on any single vendor.

The testing process validated the operation of metal-stack.io on Dell servers, confirming its readiness for enterprise-grade deployments.

Vendor-Agnostic Serial Console Access​

Alongside Dell support, our pixiecore component was improved: it can now detect the server manufacturer during the boot process. As a result, the serial console is now available for maintenance tasks across all machines in your partition, regardless of vendor — something that was not possible before this release.

More Information​

You want to know more about everything that landed in this release? Check out the full release notes.

And luckily there was the perfect opportunity, as we now start to migrate our control-plane from the public-cloud to our own data-center. In the last 5-6 months I analyzed the concepts on autonomous control-planes presented in MEP-18, built up a minimal prototype of it in a mini-rack and even did some failure scenarios to test my theories. So in this article I want to provide you some details on the hard- and software we used to build the mini-rack and the results I found in my thesis.

But first of all I need to say a big "Thank you!" to the whole team at FI-TS for supporting me during this journey, especially to @honigeintopf, who helped me a lot building up the mini-rack, and to my supervisor at Hochschule München for accepting the thesis and giving scientific advice on how to write it.

As always: bare metal​

At first we had to find some hardware, which is physically small enough to fit into our 10-inch-rack but at the same time functional enough to host a complete metal-stack/Gardener-Setup including the Control-Planes and a Partition with Shoot-Clusters. Inspired by Jeff Geerlings mini-rack guide we bought the following devices:

• GeeekPi / DeskPi RackMate T1
• 1x Teltonika RUTX10 - Wireless Router 4-Port-Switch
• 3x CWWK Mini PC N150 (Upgraded N100) Firewall Appliance
• 5x Beelink EQR6 Mini PC
• GL.iNet KVM Remote Control via the Internet
• AIMOS HDMI KVM Switch, 8 In 1 Out

The router is used to provide internet access for the whole rack through Wifi and is also our endpoint for tailscale to access the rack from remote. With the CWWKs having six ethernet ports we can use them to simulate the CLOS-architecture of the switches. So we had one spine-switch connected to the router and each of the leaf-switches. The leaf-switches are linked with four of the Beelinks, which are the machines managed by metal-stack. The remaining Beelink is used for an initial k3s-cluster, that is hosting the control-planes of metal-stack and Gardener. The KVM devices are used to remotely access the console of the machines.

Deploying the stack​

Next we brought the system to life by deploying our software on it. The initial k3s-cluster could be easily started with the Ansible roles provided by the k3s-project. For the control-planes we were able to re-use the mini-lab playbooks.

The biggest challenge was to set up the whole network infrastructure. As the metal-core running on the leaf-switches is dependent on the SONiC operating system, we had to somehow get SONiC on those CWWK machines. We solved this problem by installing a virtual machine and mapping all of the physical ports into that VM.

As we do not have a dedicated management network and servers, we also had to dually attach the leaves to the spine and the spine to the router: one cable for the management-address provided by a dhcp-server and one cable for the BGP-network. On top of that underlay BGP we have the standard overlay routing with Vxlan and VRFs.

The machines and switches could then reach the control-planes and communicate with it using self-signed certificates. For the PXE-Booting process of the machines we had to install another dhcp-server and the pixiecore on one of the leaves. As the machines do not have a BMC we weren't able to do the lifecycle management with metal-stack, but luckily they have a nice little Power-Button on the front and can be PXE-booted again.

The four machines were used to install two shoot-clusters with Gardener, each with one firewall and one worker-node. The DNS-entries of the clusters are provided by PowerDNS. As described in the "Matryoshka principle" in MEP-18 we then installed another metal-stack- and Gardener-Control-Plane in these two clusters. In a real setup these two would be used to manage the productive partition and clusters, which we do not have in the mini-rack.

Everything failed ...​

With that the first question of my thesis was answered: Building an autonomous control-plane the way it is described in MEP-18 is a very good solution for our bootstrapping-problem regarding automation and low complexity for the operators. But the second question is still open: What consequences do specific failure scenarios have and how can we return back to normal operation? That's why I did three different failure scenarios on the mini-rack. First I deleted only the k3s-cluster, then only the worker-nodes of the shoots and lastly all five Beelinks at once.

The short-term consequences of the outage were just as expected. When the k3s-cluster fails, there is no direct impact on the productive control-planes. But the automatic restore-process with the backup-restore-mechanisms of metal-stack and Gardener made some trouble. Because sometimes there was made a new backup instead of restoring an old one. In these case it is important that you have more than just one backup version in your S3-Storage to restore it manually.

And of course this S3-Storage must still be placed on external infrastructure in the public cloud since we do not want to have the backup in the same data-center as the running control-plane. And also we do not have S3-Storage outside of the Finance Cloud Native. So the backup itself would be dependent on the control-planes, if we placed it there.

The "smallest functional metal-stack installation"​

The DIY-rack that @qrnvttrl brought to our stand in Brussels was an absolute eye-catcher. It was built for his bachelor thesis at FI-TS and is such a good starting point for conversations. It allowed us to show data center technology in a very compact, functional format for the first time. Quirin even offered to write a blog article about it to provide some background information on how it came into existence and what parts were used to build it. I guess a lot of people wanted to know it in more detail, so keep an eye on the blog to find out soon.

We have a GUI?​

With the metal-ui that @ostempel implemented specifically for the FOSDEM, it became so much easier to show the possibilities of our new V2 API (as described in MEP-4). Thanks Oliver for bringing this shiny little thing to our stand and showing it to people! In my opinion we should invest more time into it, give people a chance to contribute to the repository in case they see a need for managing metal-stack not only through CLI but also through a desktop application.

People noticeably shifting to sovereign infrastructure​

From speaking to so many people, it became even more apparent than last year, that the current political world situation forces many deciders and engineers to explore sovereign infrastructure technology. The times are definitely over when we are asked why one should not just move all the workload to the cloud. Almost everybody had an understanding by now of what chances are in there for Europe to define our version of data and cloud sovereignty. We would be more than happy if we can give back something to you with the metal-stack project on your journey to modern data center infrastructure. It was really incredible that so many of you actively approached us and just wanted to talk and know more about what we do.

Another thanks go out to the organizers of the event and all the people who made the conference possible. It's a safe space for everyone who attends, and you can just be who you are. We appreciate this a lot. During a perfect Belgian beer this weekend, the following sentence was said (not sure if I am allowed to say who said it, but it was great): Computer science is a treasure of our planet that represents what we're able to achieve. It's collective intellectual property and everybody can contribute to that. Openly and without fear. 🍻

Before the tide comes in.

Frequently Asked Questions​

To round up this blog article, I would like to include some of the most common questions we heard at the conference. In many respects, we never really covered them anywhere on the web, so I guess it's a good moment to answer them here. Of course, you can also ask again if we meet one day. 😛

Does this run on Raspberry Pi?​

Unfortunately, we do not release many ARM64 artifacts by now. Depending on which parts you want to run on a Raspberry Pi it would mean to add the build to all our repositories, which in our opinion burns quite a lot of resources without having an idea what should be done with it. If there's a more serious need to release artifacts for certain platforms, we are more than happy to add it to our CI. Please reach out to us, if you need it. ARM64 artifacts are specifically built already for csi-driver-lvm or metalctl.

Can I install this in my home lab?​

Theoretically, yes. Theoretically, you can do a lot of things. 🤓

metal-stack is definitely classified as data center technology and installing this at home is kind of overkill. The bigger your environment gets, the more value you can get from metal-stack. It enables a small team of just a few people to manage thousands of servers. Maybe consider the company you work for to use metal-stack, as it fits their needs more likely than at home.

However, for educational purposes, like learning about networking, booting Linux, switches and so on, metal-stack might be a perfect playground for that. Specifically, containerlab turned out to be a really cool project to test out ideas regarding network topologies and using BGP in the data center. This is used in our virtual lab, called the mini-lab, too.

When did you start?​

We started in 2018 with metal-stack and went into production with the software in 2020. Today, we manage more than 2000 servers with metal-stack, and we do not see any particular bottlenecks yet. Through the years, we are quite confident to say you can use it for production.

But if you're open source, how do you make money with this? Do you offer support for this?​

As we are a consulting company, Yes, we do offer support for metal-stack! We can help you to plan, support and operate Kubernetes as a Service on Bare Metal in your own data center and also have experts on digital transformation regarding other topics. If you need more information on who we are, check out https://x-cellent.com/.

Also keep in mind we have a hosted version of metal-stack that is named metalstack.cloud.

What sets your solution apart from similar projects like OpenStack Ironic or Ubuntu MaaS?​

When we built metal-stack we wanted to make it a driver for Kubernetes as a Service in an on-premise data center. For this reason, we are more opinionated on certain topics than other projects. For example:

• Networking: The network is part of the solution of metal-stack. We require BGP in the data center and a switch that can run metal-core to dynamically apply port reconfiguration during machine allocation. With this, we can lower operational overhead, run Kubernetes CNIs with native routing (i.e. without overlay networks), provide services of type LoadBalancer with MetalLB and achieve efficient routing even in case of failover.
• Firewalls: We did not want to see any more big external firewalls that hold a complex state. This is why firewalls are an essential part of our infrastructure that can be managed by the firewall-controller through Kubernetes resources. This brings firewall rules as close as they get to the applications that require them and are automatically cleaned up when, for instance, the service resource does not exist anymore.
• Slim and Fast: Kubernetes needs to be able to scale quickly, and we wanted the provisioning process to be really quick. The provisioning time of a machine (depending on the vendor) can take only a minute. We wrote everything in Go in an API-driven manner such that users can easily access services without requiring manual interaction from operators.
• Runs on K8s but also without: We want metal-stack to not rely on Kubernetes per se. We have an imperative REST (soon only gRPC) API that does not require the Kubernetes API to operate. To us, this provides us the best of two worlds: Staying open to future platforms (because we're not locked into Kubernetes) while utilizing effective infrastructure under the hood to run the control plane.

The conference brings together thousands of developers and enthusiasts to talk, learn and collaborate.

It will take place on the weekend from January 31st to February 1st. Entrance is free for everyone.

This year the stand will be located in the K-Building (K1-B-08). We're looking forward to meet you again soon. 😊

Venue and Mood Snapshot​

The house was filled to the brim with 27 engineers from different companies and backgrounds—almost twice as many as the early days of the hackathon. That energy was palpable from breakfast to the late-night laptop sessions. We still followed the “everyone cooks, everyone cleans” rule, but this time the kitchen was more of a welcome distraction than another meeting room. Getting a break from terminals while chopping vegetables together was honestly one of my favorite parts of the week.

Track Overview​

With so many people on-site we divided into multiple tracks. The recap page lists everything, but a few threads stood out to me.

Autonomous Control Plane and gardenadm​

I'm knee-deep in MEP-18 at the moment, so I gravitated toward the gardenadm workstream even though it wasn't the main hackathon theme. The Gardener team pushed the tool to roughly 95% completion, ironing out installation flows and HA concerns so that bootstrapping an autonomous Gardener control plane finally feels repeatable. A handful of annoyingly small bugs are still blocking day-to-day usage, yet the remaining work is well understood and in flight.

Networking Modernization​

Around the middle of the week we mapped the migration path from legacy Ingress definitions to the Gateway API. Having everyone around in person helped us compare production experiences and validate the analysis with folks maintaining multi-cluster setups. The outcome is a clear plan for introducing the Gateway API gradually without shocking existing landscapes, plus a backlog of prototype PRs that we can continue remotely.

Reliability & Scale Testing​

Another team stress-tested Gardener to understand how many seeds and shoots a realistic deployment can host before hitting bottlenecks. These scale-out experiments surfaced valuable data for capacity planning and gave us confidence about the limits before we need to add more control planes or tune etcd. It's one thing to assume Gardener will grow with our needs; it's much better to measure it together.

Community Takeaways​

Personally, I learned a ton by pairing with people I usually only see in issue trackers. The cross-company collaboration was excellent—everyone brought their perspective, whether they were touching networking, control planes, or documentation. Watching first-timers blend in so quickly, and seeing how much the event has grown compared to previous years, makes me even more excited for the next one.

Call to Action​

If this sounds like a week you would enjoy, join the Gardener Slack, follow the hackathon announcements, and bring your ideas next time. The momentum is real—each event gets bigger, kinder, and more impactful, and we would love to build the next chapter with you.

This update extends metal-stack's open infrastructure capabilities from OCP-based network switches to server hardware, enabling:

• Seamless provisioning and lifecycle management of OCP Open Rack line systems
• Improved scalability for data center environments
• Standardized, energy-efficient hardware integration

Release Notes: https://metal-stack.io/docs/release-notes/v0.21.10

GitHub Release: https://github.com/metal-stack/releases/releases/tag/v0.21.10

• Gardener Operator
• Cluster API Integration
• SBOMs for Release Artifacts
• MEP-4 in Alpha Stage
• Improvements on SONiC Integration
• Gardener Ontap Extension
• More Information

Check out the direct link to the release here.

Gardener Operator​

With the introduction of the Gardener Operator, the Gardener Project has started to provide a standardized way to deploy and manage Gardener installations for the community. It comes with a lot of beneficial traits allowing high-availability of the Virtual Garden (no downtime during updates anymore), provisioning extensions through OCI artifacts and enforced kubeconfig secret rotation such that there is no static admin kubeconfig anymore, etc.

Historically, metal-stack shipped with an own approach for deploying the Gardener through the metal-roles based on Ansible. It utilized the upstream helm charts for the Gardener Control Plane and a self-managed Virtual Garden Helm chart (which was based on the garden-setup repository). Luckily, some of these charts are now obsolete, minimizing the maintenance burden, and it's sufficient to rely on a single Helm chart during the deployment: The one that sets up the Gardener Operator. So, we still ship the Ansible roles but with a new set of Ansible roles that are using Gardener Operator resources to install the Gardener.

In general, the migration path that we use is described here. The idea is to restore the data of the Virtual Garden from the backup, re-registering the existing Gardenlets and migrating the shooted seeds to a new Gardenlet. As the migration of existing production setups can be pretty complex, please reach out to us at our Slack Community. We're here to help if necessary.

After this release, we will try to catch up with the most recent Gardener releases again in order to be able to provide K8s 1.33 support soon.

Cluster API Integration​

Another big aspect of this release are extended integration tests for the cluster-api-provider-metal-stack. The tests are running in the big integration suite hosted at the FI-TS, ensuring compatibility with our metal-stack components for every release from now on. This makes Cluster API an integral part for providing Kubernetes on metal-stack for users that do not rely on the Gardener integration.

The release adds new OS images on images.metal-stack.io that include expected components for Cluster API bootstrapping like kubeadm.

The tests for the provider are based on the official e2e framework. For now, they only include cluster creation tests. In the following releases we will work on extending the test cases and we are also planning to run CNCF conformance tests against CAPI clusters, too.

SBOMs for Release Artifacts​

In order to provide a common basis for identifying software vulnerabilities, all our release artifacts consistently contain an SPDX-formatted software bill of material now. Usually, we embed the SBOM directly into the container artifacts using Buildx. Please note that these SBOMs are also available for the OS images of metal-stack.

Using these artifacts for CVE scanning can be done pretty easily. Please find examples for this in our documentation on metal-stack.io/docs.

This task was mainly driven by @mac641. Kudos to him!

MEP-4 in Alpha Stage​

MEP-4 is definitely the longest running enhancement proposal that we have ever worked on. It finally makes it to the alpha stage and is soon to be expected to become GA. It contains an entire update of the metal-stack API, deprecating the Swagger-based REST API and replacing it with protobuf and ConnectRPC.

The project that implements our V2 API definition is called the metal-apiserver. The implementation is capable of creating API tokens with fine-grained access permissions, such that technical components can access the metal-stack API with minimum access privileges and without the possibility to reach into unintended project and tenant scopes.

In addition to that, the network business layer was completely modernized, such that it will be possible to set up namespaced networks for allocating IP addresses, creating super networks (scoped and unscoped) instead of just having a single super network for a partition, and more.

Also, the metal-apiserver drops the dependency on NSQ and will instead use valkey to introduce asynchronous task queueing for the implementation of more complex endpoints.

The metal-apiserver is already deployed in mini-lab by default. You can access it using the metalctlv2 CLI. There are also first Ansible modules available using the fairly new and awesome connect-python project for implementation.

We will keep you informed on the development progress. If you want to contribute thoughts to the new API you may want to visit our public planning meetings and discuss ideas together with us.

Improvements on SONiC Integration​

A lot of work from our contributor @iljarotar was put into improving and stabilizing SONiC switches over the course of this year. The deployment now follows a completely new approach using a generator to provide a config_db.json.

In case you still use the sonic role we advise operators to migrate to the sonic-config, which utilizes the generator for the switch configuration.

Another important step was made in the mini-lab, which now allows spinning up the lab with different versions of SONiC. In our integration tests we can also ensure support for Enterprise SONiC now. We're aiming for more sophisticated testing of different flavors of SONiC to suit more real-world scenarios.

Gardener Ontap Extension​

As another storage solutation in Gardener setups, we included beta integration for NetApp ONTAP storage. This integration is provided by a dedicated Gardener extension, which is called gardener-extension-ontap.

The person who made this possible is @honigeintopf and we'd like to thank him for the huge amount of efforts that went into this extension.

More Information​

Please check out the release notes to find a full overview over every change that went part of this release.

As always, feel free to visit our Slack channel and ask if there are any questions. 😄

The meetings are held biweekly on odd calendar weeks from 14:00 to 14:30 on Microsoft Teams. The purpose is to provide an overview of our current projects and priorities, as well as to discuss new topics and issues within the group.

You can find the link to join in our documentation here. If you want to get an invitation to the event, please drop us a line on our Slack channel.

Planning meetings are currently not recorded. The meetings are held either in English or German depending on the attendees.

Hope to see you there!

As the previous three times, we doubled down to choose the Schlosshof in Schelklingen, Germany, as our location as it served us well in the past.

Topics​

Coordinating a sprint of a new team for one week is no easy task and needed to be prepared beforehand. We collected all possible topic ideas, prioritized by votes. Thereafter everyone commented below their favorite topics where they'd like to participate. Before the event started everybody knew their track and we mixed the teams up to ensure a diverse range of skills and perspectives.

What made this hackathon's topics special was the inclusion of frontend developers, which broadened the scope of tracks and allowed for significant improvements in user interfaces and user experience of the Gardener Dashboard and the documentation website itself.

Documentation Revamp​

While Gardener is well documented, finding the right information can be challenging. This is what this track is all about. For one the glossary was updated to reflect the latest changes and technologies.

But more importantly metadata was attached to every single documentation page. This drastically improves the search results.

As a result the documentation should be much more welcoming for new users and for more experienced users alike.

In addition to that, the team made a proposal for exchanging the existing docs framework with vitepress. The result looks very promising and even simplifies the developer experience significantly. Maybe we'll gonna see a new Gardener docs website soon?

Dashboard Usability Improvements​

What do the users of Gardener actually see? In many cases it's mostly the Gardener Dashboard. Currently all options are displayed and can be edited - regardless if they would break the cluster or not.

To mitigate this, the project administrators are now able to define defaults and even hide UI elements within the form to prevent users from unintended or overly complex configurations.

L7 Load Balancing for kube-apiserver​

In the previous hackathon proper load balancing of the Gardener kube-apiservers was implemented for external traffic. Though the internal traffic is still not load balanced leading to bad distributions of requests and potentially overloading a single kube-apiserver. The resulting throttling cannot be solved by simply scaling things up.

Instead the internal traffic will now be load balanced, too. This is achieved by routing the internal traffic through the Istio Gateway and thereby treating the internal traffic similar to the external traffic.

Single-Node Ready Gardener Operator​

Gardener is configured and deployed with high availability in mind. Though in some bootstrapping bare-metal scenarios a reduced amount of nodes are available. When running only on one node multiple replicas of some components impossible or simply overkill.

In this track lots of small adjustments and tests were made to ensure supporting this scenario as well.

Community and Collaboration​

This time 19 contributors from x-cellent technologies, SAP and noris network joined. Most from Germany, but some from Bulgaria. In general we were younger and welcomed more first-timers. What sets this event apart is the way it elevates collaboration across company boundaries, leading to a rich exchange of ideas and perspectives.

During breaks and evenings, we had the opportunity to cook, relax and bond over good food and drinks. The relaxed atmosphere fostered open discussions, idea exchanges, and a sense of camaraderie among participants. Quite a few kilometers were walked and jogged, multiple Volleyball matches played and many table tennis balls were hit.

Conclusion

A huge thanks to all contributors: you were awesome. If you are keen to dig deeper into the full list of topics and pull requests, check out the gardener-community/hackathon writeup or watch the demos from the review meeting.

And we are to proud to be able to announce the next Hackathon in early December 2025. If you'd like to join, head over the Gardener Slack. Happy to meet you there!

• Breaking Change in Semantic Versioning for OS Images
  • Naming of OS Image Releases
• More Information

Check out the direct link to the release here.

Breaking Change in Semantic Versioning for OS Images​

A change in the semver library that is used by metal-stack and in the Gardener project forces us to rename the identifiers that we typically use for OS images like Ubuntu 24.04. The library now requires stricter semantic versions, not allow leading zeroes in version segments.

In case you use for example ubuntu-24.04.20250228 as an ID for an image in the metal-api, this needs to become ubuntu-24.4.20250228.

In order to introduce the new identifier-format, before updating to this release of metal-stack, an image has to be created according to the new version format. This image then co-exists with the old image format. After this, all machines referencing the old image must be reprovisioned with the new image ID format.

After all the references were migrated to the new image format, the old versions must be removed from the metal-api before upgrading to this release. Please adapt your deployments accordingly.

Unfortunately, there is no better way to migrate this ID. Another option was to fork the Gardener project, which we did not want to do. If you encounter bigger issues during this step, please contact us in our Slack channel.

Naming of OS Image Releases​

The Ubuntu OS images we release through metal-images will continue to use the existing naming scheme. The download paths for our OS images will still contain leading zeros for Ubuntu LTS versions.

More Information​

Please check out the release notes to find a full overview over every change that went part of this release.

As always, feel free to visit our Slack channel and ask if there are any questions. 😄

As we all know, good things take time, and that time has finally come! In this release, IPv6 addresses can be provisioned to machines through the metal-api. Read on to learn how it works.

• Basic IPv6 Support
• Gardener Support to v1.106
• Audit Backend Based on TimescaleDB
• metal-core Reporting BGP States
• Relaunch of Cluster API Provider
• More Information

Check out the direct link to the release here.

Basic IPv6 Support​

Back in 2021 we published a first blog article talking specifically about IPv6. However, work on the topic was frequently interrupted and postponed. As IPv6 has become a recurring track in the Gardener Hackathons, this year we finally gained enough confidence to merge our first version of basic IPv6 support in the metal-api.

A really big bunch of the work was done by @majst01, who also wrote the corresponding enhancement proposal MEP-13. Thanks for the effort and the never-ending will to finish this up. 😌

With the new API, operators can add a list of prefixes containing both IPv4 and IPv6 addresses, which looks like this:

---
id: tenant-super-network-mini-lab
name: Project Super Network
description: Super network of all project networks
partitionid: mini-lab
prefixes:
  - 10.0.0.0/16
  - 2001:db8:0:10::/64
defaultchildprefixlength:
  IPv4: 22
  IPv6: 96
privatesuper: true
consumption:
  ipv4:
    available_ips: 65536
    available_prefixes: 16384
    used_ips: 2
    used_prefixes: 0
  ipv6:
    available_ips: 2147483647
    available_prefixes: 2147483647
    used_ips: 1
    used_prefixes: 0

Both families have specific default prefix lengths that are used for child network allocation. Also there is dedicated usage reporting per IP address family. The consumption of IPv6 address families is only an approximation, as counting free addresses would otherwise be costly.

By default, metalctl users allocating a child network automatically inherit the prefixes from the address families defined by the parent network:

❯ metalctl network allocate --name my-node-network --partition mini-lab --project 4b9b17c4-2d7c-4190-ae95-dda44e430fa6
---
id: 2d2c0350-3f66-4597-ae97-ef6797232212
name: my-node-network
parentnetworkid: tenant-super-network-mini-lab
partitionid: mini-lab
prefixes:
- 10.0.0.0/22
- 2001:db8:0:10::/96
projectid: 4b9b17c4-2d7c-4190-ae95-dda44e430fa6
vrf: 20
consumption:
  ipv4:
    available_ips: 1024
    available_prefixes: 256
    used_ips: 2
    used_prefixes: 0
  ipv6:
    available_ips: 2147483647
    available_prefixes: 1073741824
    used_ips: 1
    used_prefixes: 0
privatesuper: false
...

With the --addressfamily flag it is also possible to extract only child prefixes from the given address family. However, this release also introduces the ability for users to create child networks with a custom prefix length, so it is also possible to allocate smaller or larger prefixes.

When an IP address is allocated from a network without explicitly specifying an address family, a user acquires an IPv4 address, unless the network consists only of IPv6 prefixes. In the latter case, a user gets an IPv6 address by default.

❯ metalctl network ip create --network 2d2c0350-3f66-4597-ae97-ef6797232212 --project 4b9b17c4-2d7c-4190-ae95-dda44e430fa6
---
allocationuuid: 2dde5c08-78b4-4765-9862-c24dc073b64f
ipaddress: 10.0.0.1
networkid: 2d2c0350-3f66-4597-ae97-ef6797232212
projectid: 4b9b17c4-2d7c-4190-ae95-dda44e430fa6
tags: []
type: ephemeral
...

Again, the --addressfamily flag can be used to explicitly specify the kind of IP address to obtain:

metalctl network ip create --network 2d2c0350-3f66-4597-ae97-ef6797232212 --project 4b9b17c4-2d7c-4190-ae95-dda44e430fa6 --addressfamily IPv6
---
allocationuuid: 0312f0b7-2a87-460f-95dd-7b67fdfcddd7
ipaddress: 2001:db8:0:10::1
networkid: 2d2c0350-3f66-4597-ae97-ef6797232212
projectid: 4b9b17c4-2d7c-4190-ae95-dda44e430fa6
tags: []
type: ephemeral

IPs and networks can be associated with machines and firewalls as usual. In case network IP auto-acquisition is used, a machine or firewall retrieves an IP from all available IP address families of the corresponding network.

At this stage, the implementation works for metal-stack without the integration of the Gardener. These parts will require adaption as well. However, it is required to run Gardener with at least version v1.109 in order to support dual-stack thoroughly.

Gardener Support to v1.106​

With this release, metal-stack supports Gardener to version v1.106, which offers shoot clusters running on Kubernetes version 1.31.

In addition to this, the mini-lab release integration now has a new gardener flavor, which uses our Gardener deployment role from the metal-roles repository. With this flavor, the mini-lab spins up the Gardener Control Plane while the release integration checks that all components are running and report readiness. Shoot creation was not yet tried out but we are keen to support this in the mini-lab for one of the future releases.

One upcoming topic will also be the migration of the Gardener installation using Helm charts to the Gardener Operator. This requires thorough testing and hopefully we can integrate the migration into one of our next releases.

Audit Backend Based on TimescaleDB​

As an alternative to the Meilisearch backend, it is now possible to use TimescaleDB as the audit backend for the metal-api audit traces. It has useful features like fast inserts and searches in hypertable chunks, data retention and compression. As this is built as an extension to Postgres, we have good experience maintaining this integration and can reuse our integration with the backup-restore-sidecar including its database update capabilities.

The interface of querying the backend is identical to what it was with Meilisearch. So, users do not feel any difference.

In the future, we plan to offer at one more audit backend for Splunk.

metal-core Reporting BGP States​

In order to keep track of the BGP connections between the leaf switches and the provisioned machines for operators there is now a way to see the connection state directly through the metal-api.

For instance, this can be observed through the switch connected-machines command in combination with -o wide:

❯ metalctl switch connected-machines -o wide
ID                                           NIC NAME                           IDENTIFIER   PARTITION   RACK        SIZE           HOSTNAME   PRODUCT SERIAL
leaf01                                                                                       mini-lab    test-rack
├─╴00000000-0000-0000-0000-000000000001      Ethernet0 (BGP:Established(54s))   Eth1/1       mini-lab    test-rack   v1-small-x86   test
└─╴00000000-0000-0000-0000-000000000002      Ethernet1                          Eth1/2       mini-lab    test-rack   v1-small-x86
leaf02                                                                                       mini-lab    test-rack
├─╴00000000-0000-0000-0000-000000000001      Ethernet0 (BGP:Established(58s))   Eth1/1       mini-lab    test-rack   v1-small-x86   test
└─╴00000000-0000-0000-0000-000000000002      Ethernet1                          Eth1/2       mini-lab    test-rack   v1-small-x86

Mainly, @mwennrich was responsible for this handy addition. Thanks! 🐕

Relaunch of Cluster API Provider​

As time went on, we decided to give our cluster-api-provider-metal-stack another try and rebuild it from scratch. So from a broken state we are back: It works again!

Over time, the Cluster API has added experimental support for the ignition file format, which we now use for machine provisioning. We also support installing the provider using clusterctl. Commands like clusterctl move also work.

The entire solution can be fully developed in the mini-lab, which simulates the entire stack from the API down to the switches and machines.

If you are interested, feel free to check out the local setup of the cluster-api provider by following our developer guide.

More Information​

This is only a small extract of what went into our v0.20.0 release.

Please check out the release notes to find a full overview over every change that went part of this release.

As always, feel free to visit our Slack channel and ask if there are any questions. 😄

FOSDEM is an open source conference held in the heart of Europe in Brussels, Belgium. It is one of the largest, if not the largest, open source software conference in the world. As a visitor we have been there in the past and enjoyed the high amount of technical talks and the diversity of the conference. In fact, it is a melting pot of people from all kinds of areas - from project maintainers to decision makers, from eminent authorities to hobbyists, from computer enthusiasts to the press.

All in all, we are very grateful for the opportunity to present our project to a wider audience. We have never met so many new faces and people to talk to about this project in such a short time. It was great to be able to talk about the software and the ideas and concepts that we have come up with with metal-stack - to exchange thoughts with others, get new inspiration and identify potential partners for future collaborations.

You can find our talk here. At the time of writing the video has not become publicly available yet, but we are sure it will be available soon.

In the meantime, we would like to thank everyone who came and showed interest. And of course a big thank you to the organizers of FOSDEM, the Virtualization and Cloud Infrastructure Dev Room, for their trust. We had a great time and hope to stay in touch with all of you! See you soon!

For the third time in a row, we visited the Schlosshof in Schelklingen, Germany, because the location and the space fit so well.

Topics​

An event of this magnitude cannot begin without proper preparation. One of the biggest keys to success is knowing the topics and who will work on them. So, as in previous events, we collected all the ideas in advance, voted on interesting topics, and said what everyone would like to work on. That way, we were able to form teams before the event started. As always, the teams came from different companies to benefit from a wider range of experience.

As the hackathon progressed, the list of topics grew considerably. This time a whopping 23 topics were covered! A complete list with an overview of each topic can be found at gardener-community/hackathon.

IPv6 Support​

A Gardener hackathon is no hackathon if IPv6 support is not a topic. This time this was no different. Here we tested Gardener's DualStack compatibility against antother IaaS. After all the previous hackathons working on this topic, we finally found no bug in Gardener itself and new shoots could be reconciled to 100%!

Next time Gardener with metal-stack and DualStack is on the list. Stay tuned!

Version Classification Lifecycle in Cloud Profiles​

In Gardener versions of Kubernetes and machine images can be declared using a CloudProfile. Versions can be classified as preview, supported and deprecated. Automatic updates are only considered for supported versions. Once the administrators decide to promote a given version to the next stage of its lifecycle, a new deployment is required. The only exception is marking a version as expired.

With the changes proposed in GEP-32 it is now possible to plan the entire classification lifecycle of a version without manual interaction based on date. Of course it is still possible to change the planned lifecycle later. A corresponding reference implementation has also been created.

Gardener SLIs: Shoot Cluster Creation and Deletion Times​

Do you know how long your shoots take to prepare or how much time they spend in deletion? Have you observed how this has changed over time and have you introduced regression? Which phase should be improved first? These questions and many more can now be answered with new metrics.

The goal is to improve productivity and overall quality. Here, a new Prometheus instance has been deployed in the Prow cluster and dashboards for the new metrics provide nice insights.

Conclusion​

This hackathon was incredibly productive, we were able to tackle many challenges and connect with each other! If you want to dig deeper into all these topics, have a look on the Gardener Community Hackathon repository. We are already looking forward to the next Hack the Garden!

1. Running metal-stack in Offline Environments
2. Gardener Integration From Version 1.81 to 1.97
3. New Features on Machine Sizes
4. Encryption of backup-restore-sidecar backups
5. Migration From Cumulus to SONiC
6. Organizing User Memberships Through the masterdata-api
7. More Information

Check out the direct link to the release here.

Running metal-stack in Offline Environments​

Our users typically run metal-stack in regulated environments that have certain restrictions on internet connectivity. With our isolated clusters feature introduced in metal-stack v0.17.0, we have ensured that machines do not need to ever reach the internet during the machine allocation time. We have now completed the same requirement for machines during their provisioning phase, theoretically allowing metal-stack to run in offline or air-gapped environments.

This has been made possible by providing configuration options for specifying DNS and NTP servers at all necessary locations in a metal-stack installation. Note that the problem of providing your own container registry, DNS provider and NTP servers is not solved by metal-stack and must be done manually. We have only removed hard-coded dependencies on internet resources.

The necessary changes were described in MEP-14 and implemented by @simcod. 👏

Gardener Integration From Version 1.81 to 1.97​

Primarily, metal-stack is designed for provisioning Kubernetes nodes. As a main cluster orchestrator, we recommend using metal-stack along with the power of Gardener. The Gardener project has a biweekly release cycle, which forces our Gardener integrations to be updated, tested and released regularly.

To give you an idea, here is a list of changes that have been made along the way:

• Migration from the cloud-config-downloader to the Gardener Node Agent
• Migration to the new audit extension called gardener-extension-audit
• Firewall auto updates through our firewall-controller-manager
• Offering K8s clusters up to version 1.30
• Updating Calico and Cilium CNI extensions including new configuration options for both of them
  • Calico can now be run "kube-proxyless" utilizing eBPF and Direct Server Return (DSR)
  • Cilium can now leverage its own BGP load balancer capabilities such that MetalLB is not required anymore
• The gardener-exstension-acl can now be used without static CIDR whitelisting to ensure connectivity between kube-apiserver and kubelets, outgoing source IPs are now dynamically configured

We cannot emphasize enough our respect for the work that goes into the Gardener project, and we are very happy to announce that another Gardener Hackathon will take place in December of this year, in which the metal-stack team will once again participate. We look forward to seeing you all again guys. 😄

New Features on Machine Sizes​

There are two notable new features for machine sizes.

First, it is now supported to match sizes against specific vendors/models of CPUs, GPUs, and disk types. This has been made possible by some improvements to our size matching algorithm in the metal-api, and by extending the hardware detection information reported by metal-hammer.

To match specific hardware, there is now an identifier field that can be used with glob pattern matching to select machines by their discovered hardware specifications. This can look like this:

- id: n2-medium-x86
  constraints:
  - type: cores
  	identifier: "Intel Xeon Silver*"
    min: 8
    max: 8
  - type: gpu
    min: 4
    max: 4
    identifier: "H100*"
  - type: storage
    identifier: "/dev/nvme*"
    min: "{{ '800GB' | humanfriendly }}"
    max: "{{ '1000GB' | humanfriendly }}"

We have also removed the restriction that only certain NVIDIA graphics cards could be used. There is no longer a restriction, so popular models like the H100 are now supported.

Second, we did another iteration on the size reservation API. Starting as an operator only feature, it is now possible to easily pass this feature to higher level APIs more easily. Reservations can now be selected by IDs and are included throughout the API, e.g. reservation counts are included in the partition capacity calculation. This is a nice feature when certain customers want to reserve machines in the datacenter without having to allocate them immediately.

In previous releases of metal-stack, the metal-api attempted to ensure size reservations and rack spreading (MEP-12) on a best effort basis. In the case of concurrency, it was possible for the metal-api to make non-ideal decisions for both of these features. Starting with this release, both features are now guaranteed to make the right decision for each allocation by serializing the machine allocations in the metal-api.

Encryption of backup-restore-sidecar backups​

Another mentionable feature for this release is the ability of our backup-restore-sidecar to encrypt backups before uploading them to a backup provider. The used encryption algorithm for it is AES-256.

This work was implemented by our new colleague @ostempel. Really glad to have you in the team and looking forward to see more soon! 😻

Migration From Cumulus to SONiC​

Initially, metal-stack started with support for Cumulus Linux on network switches. Keep in mind that metal-stack is tightly integrated with the network to minimize management costs, provide the best possible performance, and ensure the availability and scalability of these components. After NVIDIA acquired Cumulus Linux, we decided to migrate our switch infrastructure to SONiC, an open source alternative that works better with metal-stack.

In order to migrate existing environments, we developed special migration commands to simplify the switch migration. It also allows the migration to occur without any downtime to the customer's production workload.

There is now a new command added to metalctl called switch migrate. The general idea behind the migration flow is as follows

• Deploy the new switch through CI and metal-roles without connecting the machines.
• The metal-core will register with the metal-api as a new switch in the rack.
• Now run metalctl switch migrate to copy the existing machine connections in the metal-api to the new switch, port mappings will be automatically translated from Cumulus to the SONiC naming scheme if necessary.
• Cables can now be swapped out without downtime, as the machines are dual-connected to two switches by design.

Much of this migration was designed and written by @iljarotar! ❤️

Organizing User Memberships Through the masterdata-api​

The birth of metalstack.cloud - our hosted version of metal-stack - left some traces in our code base. With metalstack.cloud we implemented a multi-tenant API and UI for metal-stack, which required us to implement some user management.

Therefore we added new entities to the masterdata-api called ProjectMembership and TenantMembership. With these entities it is now possible to create n:n relationships between users and their memberships within organizations and projects.

Check out metalstack.cloud if you haven't already and get a free trial. We host our infrastructure in Germany and are ISO 27001 certified. You can also ask us if you want to set up metalstack on-premises in your own datacenter and we will be happy to support you there if necessary.

More Information​

This is only a small extract of what went into our v0.19.0 release.

Please check out the release notes to find a full overview over every change that went part of this release.

As always, feel free to visit our Slack channel and ask if there are any questions. 😄

As last time, the event took place at Schlosshof nearby Schelklingen.

Preparation​

Before we come together for the Hack the Garden event, we collect all the ideas which might be worth hacking on from the attendees. After that everyone votes for 3 topics he is interested in. The voting result are a good starting point. We also try to have people from different companies taking care of each topic to ensure the mixed knowledge will bring the best ideas.

The following sections contain a brief summary of the topics that we were working on. A fully detailed summary including all topics can be found in the Gardener Community Hackathon Repo.

Topics​

Tailscale VPN Access to the Kubernetes API Server​

A Shoot cluster created by Gardener can be secured with a Gardener ACL Extension to restrict who can access the Kubernetes API Server by specifying an access control list which restricts the allowed Source IPs. This is fine for a lot of use cases. Though in some scenarios like working from home this does not work.

Here comes Tailscale VPN in handy. This is a very simple to use VPN, which is based on Wireguard. Tailscale already offers an Operator for Kubernetes which simplifies the setup for Kubernetes a lot.

With this, it was not necessary to implement something for Gardener to make this work, writing a How-To-Guide was enough.

And it has already been published: Tailscale for Kubernetes API Server

VPN2​

VPN2 is a critical component which enables the Kubernetes API Server in the control plane to talk to Pods and Services in the Shoot. This is done via an OpenVPN Server on the control plane side and an OpenVPN client on the Shoot. The logic to setup this VPN is mostly written in bash.

During the Hackathon several topics regarding VPN2 have been addressed:

• Rewrite of the whole setup logic in go PR84
• Use of IPv6 only as transfernetwork PR83

Node Agent​

The node-agent replaced the former logic written in bash which is responsible to bootstrap and further manage the worker node since the last Gardener Hackathon. As it is now a controller-runtime based operator, a lot of further improvements are possible.

One of the topics on the wishlist for the node-agent was to make its configuration more type safe. Up to now, the configuration is based on files and systemd units. This is problematic for configurations files where multiple parties want to ship modifications, e.g. the containerd.toml which requires modifications when a different container runtime should be used, a GPU is in the worker node or a Registry Mirror should be used.

To make such modifications possible without destroying the configuration of another requirement, a type safe configuration for the node-agent was introduced and some extensions where already converted to take advantage of this new feature.

Branch

Garden Local Setup​

The garden local setup is critical for development, as it allows developers of gardener to run a whole setup with seed and shoot in a kind cluster on the laptop. Since a few month, development of the gardener operator is ongoing. The gardener operator will simplify the installation of the garden cluster dramatically. The gardener operator was not configured in the garden local setup to spin up the initial gardener components. During the hackathon this was achieved.

PR

Conclusion​

This was the second Hack the Garden Hackathon celebrated at the same location Schlosshof, we all tend to keep this as our go-to location for the next ones.

• Support for Graphic Cards
• Switch Port Management
• Hello Ubuntu 24.04
• More Information

Check out the direct link to the release here.

Support for Graphic Cards​

Since ChatGPT everybody talks about AI and is looking for use-cases to utilize GPU-assisted processing in the own data center.

For metal-stack to support GPUs, we enabled the metal-hammer to detect a server's installed graphic cards during machine discovery. The information reported by the metal-hammer for a server with installed graphic card may look like this:

❯ metactl machine describe cc37651f-a9ec-4b28-808d-2b78f4d4bfde
...
hardware:
  cpu_cores: 32
  cpus:
  - cores: 16
    model: Intel(R) Xeon(R) Gold 6426Y
    threads: 32
    vendor: GenuineIntel
  - cores: 16
    model: Intel(R) Xeon(R) Gold 6426Y
    threads: 32
    vendor: GenuineIntel
  disks:
  - name: /dev/nvme0n1
    size: 1600321314816
  - name: /dev/sda
    size: 240057409536
  gpus:
  - model: AD102GL [RTX 6000 Ada Generation]
    vendor: NVIDIA Corporation
  memory: 274877906944
...

This specific hardware configuration of a server type can be matched to a dedicated machine size by the metal-api. For this to happen, the size API was extended to contain GPU type constraints. Coming with this release, a matching size can also be generated through metalctl (thanks to @m1kepeter for this neat little feature!) using the size suggest command:

❯ metalctl size suggest g1-medium-x86 --machine-id cc37651f-a9ec-4b28-808d-2b78f4d4bfde
---
constraints:
- max: 32
  min: 32
  type: cores
- max: 274877906944
  min: 274877906944
  type: memory
- max: 1840378724352
  min: 1840378724352
  type: storage
- identifier: AD102GL [RTX 6000 Ada Generation]
  max: 1
  min: 1
  type: gpu
id: g1-medium-x86
labels: {}

The new identifier field also allows glob patterns to allow more variations.

After creating this size in the API, machines registering with the presented hardware configuration are successfully identified as g1-medium-x86 type servers, ready for being allocated by the users.

In order to further optimize the user-experience, we decided to introduce a new OS image called debian-nvidia-12, which is based on our Debian-based operating system image. The new image additionally contains the proprietary graphic cards drivers and the CUDA toolkit from NVIDIA. When using a GPU-typed servers in a Kubernetes cluster, it is still necessary to run the NVIDIA operator in order to utilize the GPUs inside a Pod. However, there is no need to reboot a server before being able to utilize the GPUs inside a container. Further information about this can be found in our documentation here.

Currently we only support the NVIDIA RTX series. It is not possible to have a mixed set of graphic cards inside a server. As soon as we can get our hands on more hardware, we will expand the support for more graphic cards and can probably also relax the constraint for mixed graphic cards inside a server.

This epic was driven by @majst01! 👏

Switch Port Management​

With this metal-stack release it is possible to observe and change the link statuses on a switch through the metal-api. Despite this feature being handy for our own release integration in order to ensure BGP-failover scenarios work properly throughout every metal-stack release, this feature is also beneficial for operators to get a better overview over their switch hardware inside the data center partition.

The new API endpoints can be accessed through the metalctl switch port subcommand.

The describe command shows the actual state and the desired state for the switch port including the machine that's connected on the switch port:

❯ metalctl switch port describe fra01-r01leaf01 --port swp1s0
---
actual:
  machine_id: a44d5600-d332-11ec-8000-3cecefcda340
  nic:
    actual: UP
    identifier: ""
    mac: 6c:9c:6a:4e:40:0b
    name: swp1s0
desired:
  actual: UP
  identifier: ""
  mac: 6c:9c:6a:4e:40:0b
  name: swp1s0

The switch port can be toggled using the switch port up and switch port down commands. During the next sync the desired state is reconciled by our controller running on the leaf switch called metal-core.

Unexpected down ports are reflected in the switch list and switch connected-machines view in order to quickly identify unexpected port states through the CLI.

This feature was developed by one of our original core developers of the metal-stack project, who lately re-joined our team. @ulrichSchreiner, we're glad to have you back! 😊

Hello Ubuntu 24.04​

We now provide new worker and firewall images based on the latest version of Ubuntu, which is 24.04 LTS. Please note that the 22.04 will not be maintained any longer.

More Information​

This is only a small extract of what went into our v0.18.0 release.

Please check out the release notes to find a full overview over every change that went part of this release.

As always, feel free to visit our Slack channel and ask if there are any questions. 😄

• What's Cluster Isolation?
  • How Does it Work?
• TimescaleDB Update Support for backup-restore-sidecar
• More Information

Check out the direct link to the release here.

What's Cluster Isolation?​

For certain users it is a strict requirement that servers never face the public internet. In the world of geo-distributed, cloud-driven container workloads this scenario is rarely looked at because usually users assume that applications come from publicly reachable container registries and external services are consumable from inside our Kubernetes cluster at any times, be it DNS, NTP or Let's Encrypt certificate management.

To minimize the burden to move to Kubernetes for those users that require to avoid a connection to the internet as much as possible, we came up with the design for cluster isolation. The solution adds restrictions on external network connectivity to firewalls, effectively preventing the situation that a cluster worker node ever sees a connection to the internet.

How Does it Work?​

This is how it works:

• The new oci-mirror project gives an easy possibility to specify container images that an operator wants to mirror to another registry.
• The operator now provides a private registry, DNS and NTP servers residing inside the data center partition.
• The private container registry contains the minimal set of images required to spin up a Kubernetes worker node (e.g. the Calico CNI images).
• The firewall now drops the entire traffic from the private node network by default as long as the firewall-controller is not connected to its control plane in the seed cluster.
• As soon as the firewall-controller has connected, it opens the connections to the partition-specific container registry, DNS and NTP servers.
• The worker node will then be able bootstrap, the container runtime is configured to pull the mirrored images from the private registry.

From there we offer two different flavors of cluster isolation, one that is called restricted and one that is called forbidden.

For clusters of type forbidden there is no possibility for users to effectively deploy ingress or egress ClusterwideNetworkPolicy resources (CWNPs) that would open connections outside of operator-provided network ranges. This way, the operator can prevent connectivity to the internet or other external networks. Also, services of type load balancer do not acquire IP addresses for services to these external networks as the connection to the service is not being allowed by the firewall-controller anyway.

This also implies that the user needs to provide own private registries that reside inside private networks attached to the cluster. Otherwise it is not possible to deploy any applications to the cluster.

For restricted clusters, the responsibility is handed over to the user to open up external network connections through CWNPs, which for us is the compromise between a non-isolated cluster (we call this baseline) and forbidden mode.

With this feature we are starting to deprecate DMZ-clusters as proposed in MEP-6 and planning to remove the RestrictEgress feature gate from our metal-stack shoot specification. The new cluster isolation approach replaces both these ideas from the past.

Full documentation of cluster isolation feature can be found in our docs.

TimescaleDB Update Support for backup-restore-sidecar​

In our landscape we use the backup-restore-sidecar in combination with the popular TimescaleDB extension for Postgres. Even though the backup-restore-sidecar already supports easy upgrades for standalone Postgres instances running in Kubernetes, we struggled updating databases that utilize the TimescaleDB extension.

We were able to identify the problems in the upgrade process and can now also support raising Postgres versions for databases that use the TimescaleDB extension. 😇

More Information​

This is only a small extract of what went into our v0.17.0 release.

Please check out the release notes to find a full overview over every change that went part of this release.

As always, feel free to visit our Slack channel and ask if there are any questions. 😄

• Machine Size Reservations
• Gardener Compatibility to v1.73
• Birth of gardener-extension-audit
• backup-restore-sidecar support for Meilisearch
• More Information

Check out the direct link to the release here.

Machine Size Reservations​

When a metal-stack partition runs out of machines some unpleasant scenarios can arise: For example when the operator who maintains the metal-stack infrastructure needs to replace a machine that runs another crucial infrastructure service for the cloud platform. When other users allocated all the available compute resources already, you get stuck as an operator. For these cases it would be great to reserve some spare machines, which other projects cannot allocate. This is now possible with size reservations.

The reservation can now be added to the size entity, which may look like this:

❯ metalctl size describe n1-medium-x86
---
...
id: n1-medium-x86
labels:
  size.metal-stack.io/cpu-description: 1x Intel(R) Xeon(R) D-2141I CPU @ 2.20GHz
  size.metal-stack.io/drive-description: 960GB NVMe
name: n1-medium-x86
reservations:
- amount: 3
  description: Keep machines for firewall updates
  partitionids:
  - fra-equ01
  projectid: 55a2f43f-d297-4cdf-990d-ff042e001f58

This configuration will ensure that three machines of the size n1-medium-x86 are kept for allocation of project 55a2f43f-d297-4cdf-990d-ff042e001f58 in the partition with the ID fra-equ01. There is also the possibility to show an overview over all size reservations:

❯ metalctl size reservations list
PARTITION   SIZE            TENANT       PROJECT                                PROJECT NAME             USED/AMOUNT   PROJECT ALLOCATIONS
fra-equ01   c1-xlarge-x86   metal-stack  55a2f43f-d297-4cdf-990d-ff042e001f58   gardener-seeds-workers   4/8           4
fra-equ01   n1-medium-x86   metal-stack  55a2f43f-d297-4cdf-990d-ff042e001f58   seed-firewalls           1/2           1

Gardener Compatibility to v1.73​

During the lifecycle of the last metal-stack release, we primarily focused on catching up with our Gardener integration. This is tough as we need to cycle through every version, understand the changes for this release and make necessary adjustments to our self-maintained controllers, integration test all our components and finally update our production landscapes with hundreds of Kubernetes clusters to the new version. Within the last four months we were able to catch up 13 minor versions of Gardener, which is a huge step for all platform users enabling support for Kubernetes cluster up to 1.27.

We will continue this effort in order to make all metal-stack users benefit from most up-to-date versions of Gardener and Kubernetes as soon as possible.

Birth of gardener-extension-audit​

During the last Gardener Hackathon in Schelklingen we started with a new Gardener extension called gardener-extension-audit. As the name already suggests this extension is intended for shipping the audit logs of the shoot's kube-apiserver instances which reside in the Gardener seed cluster. Somehow it seems that within the community everybody started with an own, closed-source solution for this problem. With this extension we want to offer a publicly available solution to tackle the problem of shipping API server audit logs.

The extension deploys a buffering sink based on fluent-bit, which is receiving the audit logs through the webhook configuration from the kube-apiserver instances. From there, audit logs are carried further to a user-configured destination.

The extension is still under development but we already started to use it in our production landscapes as it is way superior to our previous solutions which were directly integrated into the gardener-extension-provider-metal#.

backup-restore-sidecar support for Meilisearch​

As our metal-api audit traces are stored in Meilisearch, we extended the support for this database in the backup-restore-sidecar project. Just like for Postgres, we additionally implemented the Update interface, such that updating Meilisearch becomes as easy as deploying the newer container image in the StatefulSet definition.

For this to be possible, the backup-restore-sidecar preserves the current version's binary in its data directory. When the sidecar starts up with a newer version of Meilisearch, the sidecar detects the upcoming version mismatch and begins to dump the data with the old preserved binary into the dedicated backup directory. After that it spawns a process with the new database version and restores the dump into it. As soon as the data was migrated to the new version successfully, the actual Meilisearch container will start up and take over the data from there.

Because this project is so important for meltdown scenarios, we now provided integration tests directly inside the project. The tests transparently proof that the entire backup-restore functionality works with every new commit from now on. The tests can be run easily through Go tests against a kind cluster. Even the deployment example manifests are now generated through the integration tests to make sure manifests are always up-to-date and can be adopted easily by the community.

More Information​

This is only a small extract of what went into our v0.16.0 release.

Please check out the release notes to find a full overview over every change that went part of this release.

As always, feel free to visit our Slack channel and ask if there are any questions. 😄

This time attendees from four different companies joined. Obviously, the Gardener core and onmetal team from SAP, but also from STACKIT, FI-TS and x-cellent technologies GmbH.

The event took place in the heart of the Swabian Alb nearby Schelklingen at Schlosshof.

Preparation​

Before we come together for the Hack the Garden event, we collect all the ideas which might be worth hacking on from the attendees. After that everyone votes for 3 topics he is interested in. The voting result are a good starting point. We also try to have people from different companies taking care of each topic to ensure the mixed knowledge will bring the best ideas.

The following sections contain a brief summary of the topics that we were working on. A fully detailed summary can be found in the Gardener Community Hackathon Repo.

Topics​

Generic Extension For Shoot Cluster Audit Logs​

Audit logs of shoot clusters need to be managed outside of Gardener (no built-in/out-of-the-box solution available). Every community member has developed their own closed-source implementations of an audit log extension.

A new design has been proposed for reworking the existing implementation (contributed by x-cellent) to be more reliable and reusable: A new StatefulSet is added to the shoot control plane that can receive the API server's audit logs via an audit webhook. The webhook backend's logs can be collected via fluent-bit and transported to a desired sink from there. The backend basically acts as an audit log buffer. The first steps for implementing the new design were finished and collecting the audit logs in the buffer works.

Code: Gardener Extension Audit

Stop Vendoring Third-Party Code In vendor Folder​

The vendor folder in the root of Go modules contains a copy of all third-party code the module depends on. This blows up the repository and source code releases, makes reviewing pull requests harder because many different files are changed, and creates merge conflicts for many files when both master and a PR change dependencies. Committing the vendor folder to version control systems is discouraged with newer versions of Golang.

This is already merged into the master branch of Gardener.

Make ACL Extension Production-Ready​

The ACL extension for restricting shoot cluster API server access via IP allow-lists only had support for the OpenStack infrastructure and single istio-ingressgateways (i.e., it did neither support HA control planes nor the ExposureClass feature of Gardener).

Code: Gardener Extension ACL

Discussion: Air-Gapped Shoot Clusters​

Customers are interested in limiting internet access for shoot clusters. Today, the clusters need access to the internet by default for bootstrapping nodes, pulling images from the container registry, resolving DNS names, etc.

Ideas and the underlying problems were discussed. The conclusion is that it would be possible to achieve restricting internet access by hosting a container registry with the required images, and by either running the seed cluster in a restricted environment or by explicitly allow-listing the access to the seed networks. Overall, the concrete goals of the interested customers are not perfectly clear yet.

Conclusion​

It is getting a tradition to celebrate this Hack the Garden event, which everyone enjoys and benefits from. I am sure we will continue this for the foreseeable future, maybe even twice a year as before. We are all sure that the amount of progress and good ideas is not possible for one party alone. It is collaboration in its purest form and best sense. This is also why the idea of Open Source works, and we all are strong believers in this form of software development.

The interview is in German, you can find it here: