Unifying Deep Learning Operations: The Generalized Windowed Operation
This paper introduces the Generalized Windowed Operation (GWO), a theoretical framework unifying deep learning's core operations like matrix multiplication and convolution. GWO decomposes these operations into three orthogonal components: Path (operational locality), Shape (geometric structure and symmetry), and Weight (feature importance). The paper proposes the Principle of Structural Alignment, suggesting optimal generalization occurs when GWO's configuration mirrors the data's intrinsic structure. This principle stems from the Information Bottleneck (IB) principle. An Operational Complexity metric based on Kolmogorov complexity is defined, arguing that the nature of this complexity—adaptive regularization versus brute-force capacity—determines generalization. GWO predicts superior generalization for operations adaptively aligning with data structure. The framework provides a grammar for creating neural operations and a principled path from data properties to generalizable architectures.
Read more