Multi-Input Multi-Output training of neural networks

Multi-Input Multi-Output training leads to the emergence of smaller subnetworks within the base network.

This has a number of benefits such a better utilization of network parameters, better training and strong regularization of learned features.

My investigations show input mixing can be used as an input compression method to train multiple subnetworks in a base network from compressed inputs. Indeed, by formalizing the seminal multi-input multi-output (MIMO) framework as a mixing data augmentation and changing the underlying mixing mechanisms, we obtain strong improvements of over standard models and MIMO models. Furthermore, given proper mechanisms, the subnetworks trained by MIMO MSDA can learn very general features by sharing these features.