A Weird Node Image Patch: The Mystery of Jar Order
A Node image patch update caused a prolonged outage of production JVM applications. The root cause was the use of a wildcard `/jars/*` in the JVM classpath. An ext4 filesystem's directory hash seed changed after the patch update, altering the jar loading order. This prevented a client library dependent on a specific version of the Bouncy Castle library from initializing correctly, resulting in a `NoSuchFieldError`. The author investigated, ruling out buildah layer squashing and OverlayFS layer order issues. The problem was ultimately traced to the change in the ext4 filesystem's directory hash seed. Modifying the hash seed in the ext4 image file confirmed this. This incident highlights how seemingly minor system details can have serious consequences, emphasizing the importance of deep understanding of underlying system intricacies.
Read more