What songbirds can teach us about language: Does syntactic complexity require selection?

[since I had to write this up elsewhere for a class, might as well double post this here anyway]

Here's another story of how alleviated selective pressures can enable increased complexity, without said increase in complexity needing to be driven by positive selection; the paper later relates this to its implications for language evolution:

Ritchie & Kirby 2005 Evol Ling Comm Selection, domestication, and the emergence of learned communication systems
(Also see Ritchie & Kirby 2007 Emergence of Communication and Language)

In a prior study, the Bengalese finch was found to have a more complex song syntax than its wild ancestor (the white-backed munia); furthermore, while the finches could learn the songs of munia, the latter could not effectively learn the complex songs of the finches, suggesting that a part of the capability was physiological. The author of that prior study, Okanoya (2002), argued that the song complexity in the Bengalese finch was driven through sex selection, as the more basic pressures (food and predation) were relieved by domestication, enabling sex selection to finally drive up the complexity; furthermore, this would have been an honest signal of the male’s fitness, as a fitter bird could produce a more complicated song.

A competing hypothesis by Deacon agrees that song complexity is kept low in the wild munia through selective pressure, but claims that the lifting of basic selective pressures after domestication enabled the songs to get more complex through other means, namely drift. Thus, processes that previously paled in comparison with the selective pressures against excess song complexity became prominent, such as the effect of songs heard at an early age and mnemonic biases; that is, songs with a more regularised syntax may be easier to recall. Deacon further extends this concept to the evolution of human language; he calls the concept “selective masking”. In short, complexity may arise in the finches’ songs without being driven by direct selective pressure.

Ritchie and Kirby set out to test the competing hypotheses through computational modeling; long story short, a bunch of learning filters are set up amid evolutionary models, and the simulation is run trough three phases: 1) Population is filtered to have a particular song type; variation is reduced (modeling the case among wild munia. 2) The population, having learned (and become “attached” to) a particular kind of song, is now bombarded with a bunch of random songs, and demonstrates resistance to be affected much by it: the simulated birds still learn the ‘correct’ song over incorrect ones. 3) Selective pressure is alleviated altogether by simply ceasing to calculate the fitness values. This simulates domestication. Population was once again bombarded with random songs.

Complexity was initially defined by Okanoya as the number of unique song notes divided by the number of unique note-to-note transitions (aka ‘Song Linearity’); Okanoya found this ratio to be lower in the Bengalese finches than the wild munia, meaning their songs were more complex (less ‘linear’). Ritchie & Kirby’s simulation also yielded similar results; though they argue that a completely random song would have the maximum complexity by such measures. Additionally, they also used Grammar Encoding Length, or the number of bits required to describe a [in this case, Probabilistic] Finite State Machine, which was used to model song learning. [Now the structural linguistics and information theory loses me completely...]. Turns out, in phase 3, the grammar encoding length did increase and the linearity did go down, supporting the increase in song complexity after domestication.

Most importantly, their simulation showed that song complexity can increase in the absense of direct selective pressure, as selection was eliminated altogether in phase 3. This suggests that [once again,] one need not necessarily evoke often-absurdly-complex selection stories (like sex selection and ‘honest signals’) to explain the song complexification in Bengalese finches. Furthermore, these results can be extrapolated further to linguistic evolution, suggesting that perhaps not all of syntax complexification requires selective pressures behind it. In fact, the eliviation of such pressures can allow more complex syntax to arise. As a sidenote, it has been observed that the rise of writing resulted in higher complexity of clause embedding, and this complexity also rose gradually, not immediately after writing systems first appeared (Karlsson in Sampson et al. 2009 Language Complexity as an Evolving Variable). This can also be seen as a case of the lifting (aka ‘masking’) of a selective constraint resulting in allevated complexity, in this case probably not particularly adaptive either. One can convey complex ideas just as well, and in some cases better, without the [ab]use of intricate clausal embedding…

Ritchie and Kirby conclude with an idea that perhaps one mustn’t look for selective advantages of elaborate syntax found in human language, but instead investigate what may have prevented syntactic elaboration from arising in the past – what selective pressure may have been alleviated, and what may have caused them?

Ritchie GRS, & Kirby S (2007). A Possible Role for Selective Masking
in the Evolution of Complex, Learned
Communication Systems Emergence of Communication and Language, 387-401 DOI: 10.1007/978-1-84628-779-4_20

(Ritchie + Kirby 2005 turns out to be a draft for the 2007 paper, and I'm far too sleepy to reference that properly as it's too complicated...)