Transfer learning is a popular strategy to overcome the difficulties posed by limited training data. It uses the parameters of the source task to initialize the parameters of the target task. In this study, we cast transfer learning as a regularization procedure. In addition to initialization, we incorporate the source task parameters into the cost function used to train the target task. We regularize the learned parameters by penalizing them if they deviate too much from their initial values. We demonstrate the power of the proposed transfer learning scheme on the task of COVID-19 opacity https://www.overleaf.com/projectsegmentation. Specifically, we show that it can improve the segmentation of coronavirus lesions in chest CT scans.