The slippery slope of motive and opportunity in academic research: an info-postcard

Three common sources of error in peer review, and how to minimize them

Researchers have an odd love-hate relationship with peer review.  Most think it sucks, but at the same time, necessary.  Peer review is of course a good thing when it provides the value that is expected of it: weeding out junk papers, and improving the rest.  Unfortunately, however, the former often doesn't work particularly well, and when the latter works, it usually happens only after a lot of wasted time, hoop-jumping and wading through bullshit. Perhaps we put up with this simply because the toil and pain of it all has been sustained for so long that it has come to define the culture of academia — one that believes that no contribution can be taken seriously unless it has suffered and endured the pain, and thus earned the coveted badge of 'peer-reviewed publication'.

Here, I argue that the painful route to endorsement payoff from peer review, and its common failure to provide the value expected of it, are routinely exacerbated by three sources of error in the peer-review process, all of which can be minimized with some changes in practice.  Some interesting data for context is provided from a recent analysis of peer-review results from the journal, Functional Ecology.  Like many journals now, Functional Ecology invites submitting authors to include a list of suggested reviewers for their manuscripts, and editors commonly invite some of their reviewers from this list.  Fox et al. (2016) found that author-preferred reviewers rated papers much more positively than did editor-selected reviewers, and papers reviewed by author-preferred reviewers were much more likely to be invited for revision than were papers reviewed by editor-selected reviewers.

Few will be surprised by these findings, and there is good reason to be concerned of course that the expected value from peer review here has missed the mark.  This failure is undoubtedly not unique to Functional Ecology.  It is, I suspect, likely to be a systemic feature of the traditional single-blind peer-review model — where reviewers know who the authors are, but not vice versa. The critical question is: what is the signal of failure here? — the fact that author-preferred reviewers rated papers more positively? — or the fact that editor-selected reviewers rated papers more negatively?

Either one could be a product of peer review error, and at least three explanations could be involved:

(1) In some cases, there will be ‘author-imposed positive bias’ — i.e. author-preferred reviewers are more likely to recommend acceptance because authors have an incentive to suggest reviewers that they have reason to expect would review their paper positively.

gatekeeper

(2) Other cases, however, will suffer from ‘editor-imposed negative bias’ — i.e. editor-selected reviewers are more likely to recommend rejection because editors have an incentive to impose high rejection rates in order to elevate and maintain the impact factors of their journals, and thus compete with other journals for impact factor status.  Hence, in order to look like they are trying to meet the rejection rate quota imposed by their publisher or EIC, associate and subject editors are sometimes inclined to favour reviewers who they suspect are competitors, or even bitter rivals of the author, since they are more likely to recommend rejection and less likely to offer suggestions for improving the manuscript.  To achieve this, some editors even select non-preferred reviewers identified by authors.  I conducted an experiment to test for this a few years ago in a submission to a high-end ecology journal, where I named a non-preferred reviewer who was in fact a good friend (and who knew I was conducting the experiment). Sure enough, shortly after my submission, my friend contacted me to report that he had been invited to review my paper (to which he declined).

whoops

(3) Finally, in some cases there will be ‘unintended reviewer mismatch’ — i.e. editor-selected reviewers are more likely to recommend rejection because editor-selected reviewers are likely to be less equipped to understand the contribution of the manuscript, or to appreciate how or why it is interesting or important.  In some cases, this results because of editor ignorance; after all, in spite of best intentions by editors, authors will generally know better who is most qualified to review their papers, and best equipped to recommend effective revisions that can bolster the quality and impact of the paper.  In other cases (where authors choose not to name non-preferred reviewers), editors may inadvertently invite reviewers who are competitors or likely to provide a ‘retaliatory’ review, without being aware of this conflict of interest (an error that is not risked with author-preferred reviewers).  In still other cases, editors simply have little opportunity for quality control because they are forced to settle for whoever is willing to volunteer to provide a gratuitous review (and no one except the editor has knowledge of the reviewer’s identity, credibility or track record of reviewing quality).  With many traditional journals — because of low reviewer incentive — editors commonly end up sending a dozen or more requests before willing reviewers for a manuscript can be arranged, and so they are not the most ‘preferred’ — and hence not the best possible — reviewers for judging the quality of the manuscript.

Minimizing errors

All three of these peer-reviewing errors can be minimized by open, author-directed, peer review that combines identification of reviewer names in accepted papers, together with published declarations of ‘no conflict of interest’ (from both authors and reviewers), and incentives for reviewers to work together with authors to improve their papers (Aarssen and Lortie 2012):

Open review.  Some researchers prefer to be anonymous reviewers because this enables them to voice criticism and recommend rejection of a paper without fear of later retaliation by the author. These concerns may be reasonable.  But those who have them should abstain from peer-review, because these concerns are vastly outweighed by the cost — of single-blind review — to the progress of science:  by allowing reviewers to hide behind anonymity, there is no deterrent against biased and poor-quality reviews with draconian recommendations for rejection.  For many people, the reason why they volunteer their time to review is precisely because they can remain anonymous, not because they are nice people wanting to help advance science. Anonymous reviewing provides power over colleagues — power to approve manuscripts that support the reviewer’s own research and reject those that conflict with it.

In contrast, with author-directed open peer review, authors can seek and arrange review of their papers from the best reviewers and most reputable researchers in their fields — and can also avoid reviewers that the author suspects might be a ‘competitor’ or likely to provide a ‘retaliatory’ review.  [Editors, in contrast, are usually not sufficiently informed — nor as inclined — to avoid such biased reviewers].  Having the endorsement of a top quality, unbiased reviewer/researcher in hand when submitting to a journal (and acknowledged in the published paper) represents strong evidence in support of the paper’s merit.  The quality/impact of an article therefore can be judged by who the acknowledged reviewers are (combined with the article’s citation metrics), rather than relying on the usual inferior metric (the impact-factor of the publishing journal) .

No-conflict-of-interest declarations.  A conflict of interest occurs in peer-review when the quality of a review is potentially compromised because circumstances exist that could limit the ability of the reviewer to be objective and unbiased.  With NCOI declarations, signed by both authors (Fig. 1) and reviewers (Fig. 2) and published together with accepted papers, readers can be confident that the paper was peer reviewed and endorsed legitimately. Authors, in this case, will not be inclined to request reviews from close colleagues in order to avoid the perception of cronyism (and many editors and readers of published papers tend to know (or can easily discover) the identities of an author’s previous collaborators and close associates).  In addition, with reviewers’ names so identified, their reputations will be ‘on the line’.  Most, therefore, are likely to be honest, fair and rigorous in their reviews. Reviewers will not want their names used as public endorsements for inferior papers, or for papers whose publication will benefit the reviewer's own research reputation — at least not reviewers that will be regarded as having integrity with journals, authors, and readers. With this model then, reviewers have opportunity to develop reputations for high-quality, unbiased reviewing service.

Figure 1

Figure 2

Collaboration of authors and reviewers.  Most human efforts are better when people collaborate with a spirit of honesty and good will. This doesn’t always come easy, but it is guaranteed to be virtually non-existent under the traditional single-blind peer-review model.  When authors and reviewers collaborate to improve the quality of a paper, this can sometimes result in production of a reviewer response commentary that can be published alongside the author's paper, if accepted.  This can provide important inspiration for readers that exceeds that available from the reviewed paper on its own, and also gives credit to the reviewer for providing this contribution — thus, importantly, serving as incentive to participate productively in the dissemination of discovery that the author's paper represents. 

References


Aarssen LW, Lortie CJ (2012). Science Open Reviewed: An online community connecting authors with reviewers for journals. Ideas in Ecology and Evolution 5: 78-83.

Fox CW, Burns SC, Muncy AD (2016) Author-suggested reviewers: Gender differences and influences on the peer review process at an ecology journal. Functional Ecology. DOI: 10.1111/1365-2435.12665

Religion is not in conflict with evolution — it is a product of it

In a previous post, I argue that humans have evolved unique categories of needs and motivations not shared with other animals.  One of these I call ‘legacy drive’.  This is a central theme in my recent book (Aarssen 2015), where I interpret religion as a fundamental domain for legacy.  Here I have posted an excerpt from the book.  

“The human individual knows that he must die, but has thoughts larger than his fate. … Religion is an effort to be included in some domain larger and more permanent than mere existence.”
—  Feibleman (1963)

A sense of legacy from religion is associated of course with faith in doctrines that promise some kind of afterlife.  Belief in a future life fulfills what Sigmund Freud (1928) recognized to be “… the oldest, strongest and most insistent wish of mankind”.  As discussed in Chapter 4 [in Aarssen 2015], evidence from paleoanthropology strongly suggests that the imaginations of our ancestors were sufficiently creative for conjuring such superstitions and cultivating them in symbolisms and rituals dating from at least 50 thousand years ago.  In the words of Malinowski (1931), “Religion … can be shown to be intrinsically although indirectly connected with man’s fundamental, that is, biological needs.  Like magic it comes from the curse of forethought and imagination, which fall on man once he rises above brute animal nature.” 

Organized religion is of course still very alive and well today, with many dozens of main varieties to choose from — providing reassurance, for the faithful, that the ‘self’ need not be impermanent, even while knowing that the body is.  This is the so-called transcendent, or ‘vertical’ component of the fitness benefit of religion, both ancestrally and today — i.e. as a domain for legacy, through everlasting life of the ‘soul’.  As Kaufman (1958) put it, “Man is the ape that wants to be a god.”  Even the most devoutly religious people know, however, that all religions are just delusions (except one, of course).  

For our ancestors, as well as today, organized religion has also had an important ‘horizontal’ component: congregational affiliation.  Worship-ping memberships like churches, synagogues, mosques, and temples can provide at least three significant benefits for genetic fitness:

(i) by reinforcing one’s confidence in the ‘vertical’ component (i.e. ‘our God and his promises of salvation must be real if there are so many fellow believers’);

(ii) as a vehicle for bolstering self-esteem (in terms of membership within a ‘larger-than-self’ cultural world-view), and a sense of memetic legacy (from attainment of social status/power through personal testimony before fellow parishioners, and personal accomplishment in the business of the religious institution); 

(iii) by serving as an incentive to behave in ways that promote pro-social reciprocal exchange benefits of group membership — e.g. by not stealing, lying, murdering, etc. — because the threatened consequences of transgression involve not only shaming by the group against the perpetrator (and hence compromising one’s intrinsic ‘need to belong’), but also banishment of the soul to eternity in a bad place (e.g. hell).  What was good for the prosperity of the social group was good for the gene transmission success of resident members.

Our ancestors probably also enjoyed an additional, perhaps even more ancient, benefit from belief in the supernatural: answers (when no practical ones could be found) regarding the mysteries of life and nature — thus satisfying the restless human curiosity, and calming fears of the unknown.  The answers here were of course interpreted in terms of favours, judgements and interventions of a ‘higher power’, involving spiritualism and/or deity.  And the early shamans, priests, prophets, and their esteemed disciples, were also likely to have enjoyed elevated social status and greater attractiveness to potential mates. 

Abundant evidence now indicates that attraction to religiosity has a partial genetic basis, and that religious people generally have more children than non-believers (Rowthorn 2011).  A predilection for superstitions then is in our genes.  Religiosity, for those who ‘believe’ (and ‘behave’ accordingly), is not only an effective self-impermanence anxiety buffer (calming the intrinsic fear of failed legacy);  it can also calm general fears of the unknown and unexplained, and promote social order and cohesion — group prosperity — and hence individual prosperity of resident members.  Plus — because public dedication as a ‘follower’ normally evokes trust from other in-group members — religion bolsters one’s local reputation, including with potential benefits through mate attraction.  All of it delivered genetic fitness for ancestors.   

Religiosity then is a fairly obvious cultural product of natural selection.  Interpre-tations of its evolutionary roots have been explored in no less than 18 recent books (published in the span of just a decade) [Box 10.1; in Aarssen 2015].

The old debates, therefore, between evolution and creationism (still active in some realms) are misguided; creationism is not in conflict with evolution — it is a product of it.

References

Aarssen LW (2015) What Are we? Exploring the evolutionary roots of our future.  Queen’s University, Kingston.

Feibleman JK (1963) Mankind Behaving: human needs and material culture. Charles C Thomas, Springfield.

Freud S (1928) The Future of an Illusion. Hogarth Press, London.

Kaufmann W (1958) Critique of Religion and Philosophy. Harper and Row, New York.

Malinowski B (1931) The role of magic and religion. In: Reader in Comparative Religion (Lessa WA, Vogt EZ, eds). Row Peterson, Evanston, IL.

Rowthorn R (2011) Religion, fertility and genes: a dual inheritance model. Proceedings of the Royal Society B 278: 2519–2527.

A wake-up call for PhD education in biology

For some people, going to graduate school may be an important experience just for the opportunity to explore interesting and important questions, to satisfy an intrinsic and pressing curiosity about the world or about life, and/or how to make them better.  A percentage of these people, at the PhD level, manage to turn that quest into a lifetime research career similar to that of their grad school mentors — in academia.  

Today however, that percentage (in the life sciences at least), is very small and shrinking (as a few mouse clicks on Google search will quickly show).  Another percentage, also relatively small, will find employment as researchers in government or in the private sector (most of the latter positions require only MSc or Bachelor's qualifications).  Overall then, the news looks bad:  there is now a large oversupply of PhD students spending typically about five years of their lives as research apprentices within universities, training to be career researchers that the vast majority of them will never be.  And all while living below the poverty line.

Clearly undergraduates need to think twice and hard about what they can realistically expect to get from going on to graduate school, especially in doctoral studies.  But as the grad students in my own department have been asking lately: maybe universities should also think about how to change what they can expect to get.

This calls on universities to revisit the definitions of their ‘learning outcomes’ (LOs) for graduate education, particularly for PhDs.  Traditionally, these LOs (at least in my own field of Biology) are virtually all about preparing students to become frontline researchers: asking good questions, collecting good data, making important discoveries, and publishing them vigorously.  Recognizing that most of them will never be directly involved with these activities after graduation (and will consequently soon thereafter be largely out of touch and inexperienced with the latest advances in methodology), PhD students are now asking (and doubting) whether — after five or more years of getting groceries from the food bank — they will at least have good LOs associated with other kinds of broader and peripheral expertise (e.g. in networking, collaboration and interpersonal skills, teaching, budget management, grantsmanship, people management, and other workplace ‘smarts’) that will equip them (and make them competitive) for other kinds of employment, e.g. as corporate executives, teachers, university/college administrators, supervisors in government, and managers in industry — positions in which they will inevitably not be called ‘researchers’.   

Universities then need to address an important question:  Does the ordinary working environment of grad school not already include sufficient opportunity for students to get these ‘broader skills’ LOs simply by ‘watching, asking, and doing’ in the course of routine research activities and interactions with colleagues and supervisors?  If the answer is no, then there is a second and tougher question to address:  How can universities do a better job of delivering these ‘broader skills’ LOs for PhD students, without compromising other things that universities aim to do?

An important consideration here is the perspective of the faculty supervisor.  Recruiting graduate students is part of the employment obligation of faculty, but only secondarily.  Faculty have graduate students mainly because they need them to fulfil one of their more primary employment obligations (and career goals):  to publish research (a lot of it).  Usually this involves competing successfully as a PI ('Principle Investigator') for research grants (especially NSERC, in Canada) that will pay for research costs.  And in order to accomplish the latter they need a team of research collaborators to spend the grant money on and thus generate publishable research.  And in order to win these grants, NSERC requires that the team consists mostly of members that will receive training as HQP (‘Highly Qualified Personnel’) — particularly, graduate students, and particularly with evidence that publication success for PhD students has been effective for their success in landing university postdoctoral and tenure-track jobs.  

The expected training involved here will normally include, in varying degrees, the ‘broader skills’ LOs mentioned above.  But one thing is certain:  the priorities and motivations of most supervisors will necessarily be driven to a very large extent by the accomplishment that is most rewarded by the university employer (and of course is also most important to the supervisor’s reputation) — i.e. whatever it takes to generate a high quantity and/or quality of published research.  This product is probably correlated to some extent with good mentoring of ‘broader skills’ for the grad students within a lab.  But it need not be, and probably isn’t strongly correlated.  Instead, publication success, and hence the employer reward to the faculty member (tenure, promotion, salary increases) will be strongly correlated with the number of graduate students that he/she has supervised, and the proportion that go on to obtain academic positions. 

This necessarily means that the most conspicuous LOs that grad students can presently expect to get will be as research apprentices — essentially, to become career researchers like their supervisors — with some ‘broader skills’ of course thrown in (including from grad courses, or by ordinary osmosis) — but only as time permits, and to the extent that they do not compromise the reputational and employer rewards to the supervisor. 

The current PhD graduate education experience then — as with the faculty job experience — is a product of the culture of academia.  Changing the first will require changing the second, and neither can be changed without changing the culture. 

If change is needed, and if it is going to happen, two things will be required from universities: (1) consultation with graduate students to better define, and/or revise (and publish) the expected learning outcomes of a PhD graduate education; and (2) ensuring that these LOs have substance and are taken seriously, by finding a way to make faculty supervisors accountable for delivering them.  But these measures will never get off the ground as long as granting agencies, like NSERC, continue — as part of the adjudication criteria for grant applications — to count how many graduate students an applicant has supervised, and how many have gone on to post-doctoral or tenure-track positions in academia.

Same sex attraction — A Darwinian paradox, no more

Like most human behavioural traits, a preference for same-sex sexual relationships can be informed in part by learning and environmental / developmental experiences.  A role for genes is also now abundantly clear.  So-called ‘gay genes’ have not yet been precisely identified, but pedigree and twin studies have shown that homosexuality tends to run in families (reviewed in Ngun etal. 2011), and a recent genetic analysis of 409 pairs of gay brothers links sexual orientation in men with particular regions of the human genome (Sanders et al. 2015).  The pressing question then — described often as a ‘Darwinian paradox’ — is how do we account for the common occurrence of homosexuality in evolutionary terms, given that it would seem to present a severe limitation on evolutionary fitness through one’s direct lineage? Several explanations and speculations have been offered (see the very accessible review in Barash 2012) — and many illustrate, as the saying goes: ‘things are not always as they seem’.

For male homosexuality, one of the best explanations so far comes from recent studies suggesting that female relatives of gay men generally have more offspring than the female relatives of straight men.   In other words, there are genetic factors transmitted through the maternal line (partly linked to the X-chromosome) that increase the probability of becoming homosexual in males, but they promote higher fecundity in females (Camperio-Ciani et al. 2004, Iemmola and Camperio-Ciani 2009).  Hence, the genetic factors that “… influence homosexual orientation in males are not selected against because they increase fecundity in female carriers, thus offering a solution to the Darwinian paradox and an explanation of why natural selection does not progressively eliminate homosexuals” (Iemmola and Camperio-Ciani 2009).

A similar hypothesis (that applies to either male or female homosexuality) is suggested by Zietsch et al. (2008):

“The genes influencing homosexuality have two effects.  First, and most obviously, these genes increase the risk for homosexuality, which ostensibly has decreased Darwinian fitness.  Countervailing this, however, these same genes appear to increase sex-atypical gender identity, which our results suggest may increase mating success in heterosexuals. This mechanism, called antagonistic pleiotropy, might maintain genes that increase the risk for homosexuality because they increase the number of sex partners in the relatives of homosexuals.” … “The traits most reliably associated with homosexuality relate to masculinity–femininity; homosexual men tend to be more feminine than heterosexual men, and homosexual women tend to be more masculine than heterosexual women.”

In other words, this ‘sex atypicality’ may be advantageous when expressed in heterosexuals.  Do some (perhaps many) females tend to be more attracted to males with certain feminine behavioral traits such as tenderness, considerateness, and kindness?  In this study, the results indeed show that “... psychologically masculine females and feminine men are (a) more likely to be nonheterosexual but (b), when heterosexual, have more opposite-sex sexual partners” (Zietsch et al. 2008).

Another, more general hypothesis for homosexuality is that same-sex attraction never really imposed a significant penalty on fitness in our deep ancestral past, because heterosexual sex was still routinely practiced in spite of it.  There are two very different contexts in which this effect would be expected: 

Bivariate trait space continuum for sexual orientation (Aarssen 2015)

The first obtains from just straightforward bisexuality, i.e. where ancestral sex lives commonly involved a mix of both same-sex and heterosexual activity but in varying proportions (informed in part by genotypic variability) ranging from bisexual with a more dominant opposite-sex attraction, to bisexual with a more dominant same-sex attraction.  But random mating within this mix would also have produced genetic variants that informed strictly homosexual as well as strictly heterosexual orientations — with only the latter of the two strongly favoured by natural selection. 

Nevertheless, despite being strongly disfavoured by natural selection, strict homosexuality would have persisted in low frequency simply because of genetic factors informing same-sex attraction that were inherited from bisexual ancestors.  Under this hypothesis, homosexuality is not really a Darwinian paradox at all; instead it, along with asexuality, were just periodic maladaptive genotypic by-products of ancestral gene transmission.

Most ancestral bisexuals, however, were probably female. Research has shown that men are generally attracted to one sex or the other, whereas women are more likely than men to have a bisexual orientation. According to one hypothesis, a ‘fluid sexuality’ that enabled same-sex sexual behavior in women made it easier for women to raise children together. 

Painting of King Solomon and his wives

by Giovanni Venanzi di Pesaro (1627-1705)

This, according to Kuhle (2013): "… would have been particularly beneficial to ancestral women when their male mates were unable to adequately care, protect, and provide because they were injured, away on prolonged hunts, or preoccupied finding, courting, and mating with other women.  The latter scenario was particularly likely to occur within polygynous mating systems. ...

If so, it is possible that men’s relative lack of aversion to a female mate’s homosexual, rather than heterosexual, affair … and men’s common fantasy of simultaneously mating with multiple women … is an outgrowth of a male psychology designed to promote their mates’ same-sex sexual behavior.”

The latter would have commonly benefitted the offspring of not just bisexual women, but also the men who fathered these offspring, many of whom were often not around to help raise them.  Ironically, therefore, for many of our male ancestors, it was probably better for their own genetic fitness, if their partners were bisexual.

The second context for predicting successful heterosexual practice (and hence gene transmission) — despite the presence of same sex attraction — applies even in the case of strong preference for same-sex encounters, including strict homosexuality.  According to what we might call the ‘failed disfavouring selection’ hypothesis (Aarssen 2015), female homosexuality probably never had widespread opportunity to be strongly disfavoured by natural selection.  This is because, throughout much (probably most) of human history, males were to a large extent in control of the sexual activity and fertility of females.  Many or most women, therefore, were essentially forced — by patriarchal subjugation, socio-cultural expectations, and / or religious imperatives — to mate with men and bear their (frequently many) offspring, regardless of their sexual orientation. [The same would also have been true regardless of the intensity of female sexual impulse (drive)]. 

Accordingly, so called ‘gay genes’ in many of our female ancestors — including those that might have been inherited by their sons as well as their daughters — were never significantly limited in their transmission success to future generations.  And so in this context, the prevalence of homosexuality today is (again) not really a Darwinian paradox at all.

Importantly however, women more than ever are now in control of both their fertility and their sex lives, and their empowerment for this and other basic human rights continues to grow rapidly on a global scale.  For our predecessors, choosing successfully to be a practicing homosexual meant zero gene transmission through direct lineage.  But today this is not necessarily the case, with reproductive technologies for sperm banks and in-vitro fertilization (and perhaps, in the future, human cloning).  But unless the latter become widely practiced as a popular cultural norm, the widespread mating and reproductive freedom for women today means that selection against an exclusively or predominantly lesbian orientation may soon be ramping up (Aarssen & Altman 2006).  If female bisexuality, remains alive and well, however, so also would some homosexuality (because of genetic inheritance from bisexual maternal ancestors).

The above scenario is a good example of how some traits — like homosexuality, the generally weaker sexual impulse in females compared with males, and the child-free culture — rather than a consequence of being favoured by natural selection in the ancestral past, may instead be a consequence of not having been disfavoured by natural selection.

References

Aarssen LW (2015) What Are We? Exploring the Evolutionary Roots of Our Future. Queen’s University, Kingston.

Aarssen LW, Altman S (2006) Explaining below-replacement fertility and increasing childlessness in wealthy countries: Legacy drive and the “transmission competition” hypothesis. Evolutionary Psychology 4: 290-302.

Barash DP (2012) Homo Mysterious: Evolutionary Puzzles of Human Nature. Oxford University Press, Oxford.

Camperio-Ciani A, Corna F, Capiluppi C (2004) Evidence for maternally inherited factors favouring male homosexuality and promoting female fecundity. Proceedings of the Royal Society of London, Series B: Biological Sciences 271: 2217–2221.

Iemmola F, Camperio-Ciani A (2009) New evidence of genetic factors influencing sexual orientation in men: female fecundity increase in the maternal line. Archives of Sexual Behavior 38: 393–399.

Kuhle BX (2013) Born Both Ways: The alloparenting Hypothesis for sexual fluidity in women. Evolutionary Psychology 11: 304-323.

Ngun TC, Ghahramani N, Sa´nchez FJ, Bocklandt S, Vilain E (2011) The genetics of sex differences in brain and behavior. Frontiers in Neuroendocrinology 32: 227–246.

Sanders AR, Martin ER, Beecham GW, Guo S, Dawood K, Rieger G, Badner JA, Gershon ES, Krishnappa RS, Kolundzija AB, Duan J, Gejman PV, Bailey JM (2015) Genome-wide scan demonstrates significant linkage for male sexual orientation. Psychological Medicine 45: 1379-1388.

Zietscha BP, Morley KI, Shekar SN, Verweij KJH, Keller MC, Macgregor S, Wright MJ, Bailey JM, Martin NG (2008) Genetic factors predisposing to homosexuality may increase mating success in heterosexuals. Evolution and Human Behavior 29: 424–433.