The Northern and Southern Hemispheres reflect on average almost equal amounts of sunlight due to compensating hemispheric asymmetries in clear-sky and cloud albedo. Recent work indicates that the cloud albedo asymmetry is largely due to clouds in extratropical oceanic regions. Here, we investigate the proximate causes of this extratropical cloud albedo asymmetry using a cloud-controlling factor (CCF) approach. We develop a simple index that measures the skill of CCFs, either individually or in combination, in predicting the asymmetry. The index captures the contribution to the asymmetry due to interhemispheric differences in the probability distribution function of daily CCF values. Cloud albedo is quantified using daily MODIS satellite retrievals, and is related to range of CCFs derived from the ERA5 product. We find that sea surface temperature is the CCF that individually explains the largest fraction of the asymmetry, followed by surface wind. The asymmetry is predominantly due to low clouds, and our results are consistent with prior local-scale modeling work showing that marine boundary layer clouds become thicker and more extensive as surface wind increases and surface temperature cools. The asymmetry is consistent with large-scale control of storm-track intensity and surface winds by meridional temperature gradients: persistently cold and windy conditions in the Southern Hemisphere keep cloud albedo high year-round. Our results have important implications for global-scale cloud feedbacks and contribute to efforts to develop a theory for planetary albedo and its symmetry.