Networks pose novel challenges for inference and learning due to their discrete, high-dimensional nature. This inherent complexity necessitates the development of statistically principled unsupervised learning objectives that steer clear of ad hoc heuristics to distinguish meaningful structure from noise in real networks. In this talk I will discuss how to develop principled unsupervised learning methods that parsimoniously summarize structural and dynamical regularities in network data. These methods are unified under the Minimum Description Length principle from information theory, which readily permits fully nonparametric inference while explicitly highlighting particular regularities of interest in discrete datasets. I will discuss the motivation for this family of methods as well as a general procedure for applying this framework to problems in network inference. I will then cover a few examples of recent work in this area where I looked at identifying hub nodes in networks and extracting meaningful cohesive hypergraph structures from temporal datasets using the MDL principle.