Juggler: Multitask Learning with Task Performance Constraints

Consider multitask learning (MTL), in which N models share parts of their architecture (e.g., a backbone with a head for each task). Our goal is to train the overall model so that the training loss of each task falls below a given threshold. However, we may harm others when varying the shared parameters to help one task. A weighted total loss could balance between the different tasks to achieve the target threshold. Nevertheless, the weights that correctly balance the tasks are unknown in advance. To overcome that, we propose a scheme that divides the total training time into epochs of increasing length. The scheme adjusts the weights every epoch based on the performance of the tasks at the end of the epoch. We prove that our scheme asymptotically converges to a model that satisfies the target loss constraints (if feasible), provided the learning rate, control step size, and epoch lengths are properly tuned. We experiment on deep neural networks to demonstrate that our scheme is effective even beyond our theoretical assumptions.