• The hybrid sterility gene S5 comprises three types of alleles in cultivated rice. Such tri-allelic system provided a unique opportunity to study the molecular bases of evolutionary changes underlying reproductive isolation in plants. • We analysed the sequence diversity and evolutionary history of S5 in 138 Oryza accessions. We also examined the effect of the two functional variations (C819A and C1412T) in determining hybrid sterility by transformation. • Nineteen haplotypes were identified, which were classified into the indica-like, the japonica-like and the wide-compatibility gene (WCG)-like group, according to the sequence features of the tri-allelic system. The origin and evolutionary course of the three allelic groups were investigated, thus confirming the independent origins of indica and japonica subspecies. There were perfect associations between C819A and C1412T in the rice germplasm assayed, and the combination of C819 and C1412 was required for hybrid sterility. Evidence of positive selection in the WCG-like alleles suggested that they might have been favored by selection for higher compatibility in hybrids. • The complex evolution of S5 revealed the counteractive function of the three allelic groups at the species level. S5 might perform an important primary function in an evolutionary scale, and hybrid sterility acts as a 'byproduct' of this speciation gene.
© 2011 The Authors. New Phytologist © 2011 New Phytologist Trust.