2.2 Disentanglement 17 generative factors with the highest and second-highest mutual information with a particular latent dimension. The Separated Attribute Predictability score (SAP) (Kumar et al., 2017) first tries to predict the factor values from each dimension of the latent representation, and then measures the average difference of the prediction error of the two most predictive latent dimensions for each factor. The Disentanglement metric by Eastwood and Williams (2018), which we refer to as DCI Disentanglement for clarity, first computes the importance of each latent dimension for predicting a generative factor, using a Lasso or a Random Forest Classifier, and then measures the entropy of the normalised importance. Disentanglement definition and limitations The variety of different metrics to quantify disentanglement indicates that there is not a single formalised definition of disentanglement that is widely accepted, although all metrics are built around the intuition that a disentangled representation should separate the distinct generative factors of the data. This lack of a formal definition is further exemplified by Mathieu et al. (2018), who propose a new concept called decomposition, which they claim is a generalisation of disentanglement. Further limitations of disentanglement are highlighted by Locatello et al. (2018), who theoretically prove that unsupervised disentangled representation learning is fundamentally impossible without inductive biases on both the models and the data, which can include explicit or implicit supervision. Furthermore, in a large-scale experimental study covering the disentanglement methods and metrics described above, they observe that well-disentangled models seemingly cannot be identified without supervision, since random seeds and hyperparameters seem to matter more than the disentanglement model but tuning these appears to require supervision. In an effort to identify some implicit biases that allow for disentanglement, Horan et al. (2021) show that unsupervised disentanglement is possible if the generative factors are locally isometric and sufficiently non-Gaussian. Semi-supervised disentanglement methods Apart from unsupervised disentanglement models, other methods aim to obtain disentangled representations using various degrees of supervision. This addresses the above-mentioned limitation that disentanglement is impossible without supervision or inductive biases. We highlight a few approaches here.
RkJQdWJsaXNoZXIy MjY0ODMw