Loki's Structured Metadata: A Logistical Nightmare
Grafana Loki, often touted as 'Prometheus for logs,' initially adopted a data model similar to Prometheus. However, this proved disastrous for system logs (syslog or systemd journal). Unlike Prometheus, Loki stores each label value set separately and lacks log compaction, leading to cardinality explosions. To address this, Loki introduced 'structured metadata,' but as of version 3.0.0, it remains underdeveloped. Structured metadata labels aren't treated as regular Loki labels, requiring different query syntax. Migration from existing labels is complex and potentially catastrophic, with the risk of unintentionally creating high-cardinality labels. Upgrading requires caution, migrating existing data is incredibly expensive, and careful consideration is crucial before using it in new projects.