Within evolutionary biology, signalling theory is a body of theoretical work examining communication between individuals, both within species and across species. The central question is when organisms with conflicting interests, such as in sexual selection, should be expected to provide honest signals (no presumption being made of conscious intention) rather than cheating. Mathematical models describe how signalling can contribute to an evolutionarily stable strategy.
Signals are given in contexts such as mate selection by females, which subjects the advertising males' signals to selective pressure. Signals thus evolve because they modify the behaviour of the receiver to benefit the signaller. Signals may be honest, conveying information which usefully increases the fitness of the receiver, or dishonest. An individual can cheat by giving a dishonest signal, which might briefly benefit that signaller, at the risk of undermining the signalling system for the whole population.
The question of whether the selection of signals works at the level of the individual organism or gene, or at the level of the group, has been debated by biologists such as Richard Dawkins, arguing that individuals evolve to signal and to receive signals better, including resisting manipulation. Amotz Zahavi suggested that cheating could be controlled by the handicap principle, where the best horse in a handicap race is the one carrying the largest handicap weight. According to Zahavi's theory, signallers such as male peacocks have "tails" that are genuinely handicaps, being costly to produce. The system is evolutionarily stable as the large showy tails are honest signals. Biologists have attempted to verify the handicap principle, but with inconsistent results. The mathematical biologist Ronald Fisher analysed the contribution that having two copies of each gene (diploidy) would make to honest signalling, demonstrating that a runaway effect could occur in sexual selection. The evolutionary equilibrium depends sensitively on the balance of costs and benefits.
The same mechanisms can be expected in humans, where researchers have studied behaviours including risk-taking by young men, hunting of large game animals, and costly religious rituals, finding that these appear to qualify as costly honest signals.