dataframely: A Polars-Native Data Frame Validation Library
2025-04-30
QuantCo's effort to modernize a legacy codebase revealed a critical flaw in their data frame processing: lack of validation. Migrating from pandas to polars for performance gains highlighted this issue. Existing libraries like pandera and patito proved insufficient. To address this, they developed dataframely, a Polars-native library enabling declarative data frame validation. dataframely defines schemas, validates data frame content, supports cross-data frame validation, and offers soft validation, significantly improving pipeline robustness and readability. It's already in use across multiple teams and projects.
Development
data validation