Evolution of vertical and oblique transmission under fluctuating selection

The evolution and maintenance of social learning, in competition with individual learning, under fluctuating selection have been well-studied in the theory of cultural evolution. Here, we study competition between vertical and oblique cultural transmission of a dichotomous phenotype under constant, periodically cycling, and randomly fluctuating selection. Conditions are derived for the existence of a stable polymorphism in a periodically cycling selection regime. Under such a selection regime, the fate of a genetic modifier of the rate of vertical transmission depends on the length of the cycle and the strength of selection. In general, the evolutionarily stable rate of vertical transmission differs markedly from the rate that maximizes the geometric mean fitness of the population. The evolution of rules of transmission has dramatically different dynamics from the more frequently studied modifiers of recombination, mutation, or migration.