Hetzner AX162 Server Reliability Nightmare: A Painful Debugging Journey

2025-02-19
Hetzner AX162 Server Reliability Nightmare: A Painful Debugging Journey

Ubicloud encountered serious reliability issues with Hetzner's new AX162 servers: a 16x higher crash rate than its predecessor, AX161. After months of debugging, they suspected power limiting by Hetzner and motherboard defects as the root causes. Multiple hardware upgrades, especially motherboard replacements, ultimately resolved the issue. This experience taught them the risks of early adoption and led to process improvements, including more thorough vetting and gradual hardware rollouts.

Read more

Deep Dive into Cloud Virtualization: Red Hat, AWS Firecracker, and Ubicloud Internals

2025-01-24
Deep Dive into Cloud Virtualization: Red Hat, AWS Firecracker, and Ubicloud Internals

This blog post delves into the core architectures of cloud virtualization, using Red Hat, AWS Firecracker, and Ubicloud as case studies to compare their differences in virtual machine monitors (VMMs), kernel virtualization, and resource isolation. It explains the roles of key components like KVM, QEMU, and libvirt, and analyzes the use of technologies such as cgroups, nftables, and seccomp-bpf in achieving resource and security isolation. The author also contrasts the AWS Nitro system, summarizing the evolution of cloud virtualization technology and the importance of open-source technology in this field.

Read more