The predominance of sexual reproduction in eukaryotes remains paradoxical in evolutionary theory. Of the hypotheses proposed to resolve this paradox, the ‘Red Queen hypothesis’ emphasises the potential of antagonistic interactions to cause fluctuating selection, which favours the evolution and maintenance of sex. Whereas empirical and theoretical developments have focused on host-parasite interactions, the premises of the Red Queen theory apply equally well to any type of antagonistic interactions. Recently, it has been suggested that early multicellular organisms with basic anticancer defences were presumably plagued by antagonistic interactions with transmissible cancers and that this could have played a pivotal role in the evolution of sex. Here, we dissect this argument using a population genetic model. One fundamental aspect distinguishing transmissible cancers from other parasites is the continual production of cancerous cell lines from hosts’ own tissues. We show that this influx dampens fluctuating selection and therefore makes the evolution of sex more difficult than in standard Red Queen models. Although coevolutionary cycling can remain sufficient to select for sex under some parameter regions of our model, we show that the size of those regions shrinks once we account for epidemiological constraints. Altogether, our results suggest that horizontal transmission of cancerous cells is unlikely to cause fluctuating selection favouring sexual reproduction. Nonetheless, we confirm that vertical transmission of cancerous cells can promote the evolution of sex through a separate mechanism, known as similarity selection, that does not depend on coevolutionary fluctuations.

在进化论中，真核生物有性生殖的优势仍然是自相矛盾的。在提出的解决这一悖论的假说中，“红色女王假说”强调了对抗性相互作用导致波动选择的潜力，这有利于性的进化和维持。尽管经验和理论发展都集中在宿主-寄生虫的相互作用上，但红皇后理论的前提同样适用于任何类型的拮抗相互作用。最近，有人提出，具有基本抗癌防御作用的早期多细胞生物可能被与可传播癌症的拮抗作用所困扰，并且这可能在性别进化中发挥了关键作用。在这里，我们使用种群遗传模型剖析该论点。将可传播的癌症与其他寄生虫区分开来的一个基本方面是，宿主自身组织不断产生癌细胞。我们表明，这种涌入抑制了波动的选择，因此，与标准的红皇后模型相比，性别的进化更加困难。尽管在我们模型的某些参数区域内，协同进化循环仍足以选择性别，但我们证明，一旦考虑到流行病学限制，这些区域的大小就会缩小。总而言之，我们的结果表明癌细胞的水平传播不太可能引起有利于有性生殖的波动选择。尽管如此，我们确认癌细胞的垂直传播可以通过称为相似选择的单独机制促进性进化。