Learning meaningful representations via deep neural networks has been an important challenge for computer vision tasks. Auto-encoders have been among the most popular approaches as they are shown to enable so-called disentangled representations. Unfortunately, most auto-encoder-based methods are built upon no or very little inductive biases and lack a theoretical foundation. This (ongoing) work aims to provide insights into how group theory could be beneficial for learning modular representations. Given object-centric datasets, we show how to learn latent matrix Lie groups that model underlying transformations the objects undergo within an auto-encoder framework. The resulting framework learns (almost perfectly) disentangled group actions on toy datasets.