A critical evaluation of EFSA’s environmental risk assessment of genetically modified maize MON810 for honeybees and earthworms

Abstract

Background

In the European Union (EU), genetically modified (GM) crops are permitted for cultivation only after a thorough risk assessment and a decision by the European Commission (EC). The central scientific body assessing food-related risks is the European Food Safety Authority (EFSA). It aims to provide high-quality scientific advice for EU decision-makers. However, both the way EFSA performs risk assessment and the independence of its panel members have been subjected to consistent criticism. In this paper, I examine part of the environmental risk assessment in the Scientific Opinion issued by the EFSA GMO Panel, specifically, the impacts of GM maize MON810 on honeybees and earthworms. The evaluated EFSA document forms the scientific basis of the pending EC Draft implementing decision to renew the authorisation for the lawful cultivation of MON810. I assess the reliability of scientific information cited in the Opinion, the use of this information by EFSA, and the safety conclusions drawn in a form of an extended peer review.

Results

My research indicates that the scientific studies cited in the EFSA Opinion in the sections concerning the possible impacts of GM maize on honeybees and earthworms stem predominantly from reliable sources in terms of authorship, financial support, and status of the study. However, the reliability of the studies varies significantly concerning the ecological relevance of the experiments. Moreover, the body of referenced evidence is insufficient to draw conclusions on risk. Relevantly, several types of shortcomings in the use of scientific information in the risk assessment were identified as prevalent, namely: EFSA omits relevant available studies, selectively cites information, misquotes studies, fails to acknowledge uncertainties, fails to call for further research where needed, and fails to critically interpret studies and their findings.

Conclusions

Overall, the findings indicate that the reliability of scientific information and particularly its use by the EFSA GMO Panel produces low-quality scientific advice, which is inconsistent with the Authority Mission Statement. My research would support the call by the European Parliament and NGOs on the EC to withdraw its Draft implementing decision intended to renew the authorisation of MON810 cultivation.

Background

The current state of genetically modified crops

Genetically modified (GM) crops have been grown on an increasing area globally since 1992 [1, 2]. However, the number of countries growing them stopped increasing in 2011 and there has been a decreasing trend since then. In the European Union (EU), the situation is even more pronounced. The number of member states growing GM crops has been dropping, and it has predominantly been only Spain that keeps the area planted with GM crops in Europe fluctuating upwards [2]. Currently, only one type of GM crop is permitted for cultivation in the EU, the insect-resistant maize MON810.

The situation in the European Union

The authorisation process for applications for the marketing or cultivation of GM crops in the EU is considered by its proponents and the European Commission (EC) as one of the strictest in the world. Each new event of a genetically modified organism (GMO) has to undergo a thorough risk analysis before a committee of member states votes on the approval for commercialization. The assessment of the first applications for genetically modified products (first authorised in 1995) were under the auspices of the Scientific Committees on Plants and on Food. This task was taken over in 2003 by the then newly created GMO Panel of the European Food Safety Authority (EFSA).

The GM maize MON810 was authorised for cultivation and use as/in food and feed in 1998 for 10 years. Monsanto applied for a renewal of the authorisation for MON810 cultivation in 2007. Despite a favourable 2009 EFSA Opinion [3] on the application, the EC has not decided yet (as of June 2019) and the Bt maize remains lawfully on the market authorised under the Directive 90/220/EEC.

Although the cultivation of this GM maize has been allowed since 1998, it was first grown in 2003 in Spain with the majority of other member states never joining in. On the contrary, the number of member states who ceased the cultivation or imposed a ban on MON810 has been increasing since that time.

The restrictions on GM crops’ cultivation in the EU

Until recently, member states who refused to grow GM crops invoked the safeguard clause¹ of Directive 90/220/EC and subsequently Directive 2001/18/EC enabling them to restrict or prohibit the use of a particular GMO on their territory if new evidence since the EFSA’s assessment became available that indicated it constituted a risk to the environment or human health. Following the implementation of the safeguard clause to introduce restrictions on planting in certain member states, EFSA was asked by the European Commission to evaluate if the invocations were scientifically justifiable. The Authority concluded that its previous risk assessment conclusions remained valid in all cases and that the GM crops in question were safe [4].

The EC attempted to resolve the contradiction between the official European safety assurance concerning GM crops and the challenges to it from member states in 2015 by creating an amendment to Directive 2001/18/EC that allows member states to restrict or prohibit cultivation on their territory or part of it.² According to this Directive (EU Directive 2015/412), however, member states can only prohibit the authorised GM crop cultivation on grounds³ that do not “conflict with the environmental risk assessment carried out pursuant to this Directive”, thus ensuring the sovereignty of EFSA’s assessment as authoritative on questions of scientific risk [5].

The role of EFSA in the decision-making process

The role of EFSA’s risk assessment is crucial in the whole decision-making process concerning GMOs. The European Commission bases its draft decision about applications for authorisation of GM food, feed, or growing of GM plants on EFSA’s risk assessment. The draft decision should then be adopted or rejected by a qualified majority of Member States Experts (MSE) Committee. When no qualified majority is reached, the draft is passed on to the Appeal Committee which can again adopt or reject it or issue “no opinion”. In the third case is the draft adopted by the EC. Indeed, final decisions have been adopted by the EC on the basis of positive opinions issued by EFSA, because neither the MSE nor the Appeal Committee reached a qualified majority since 2003 [6, 7]. The concern is that EFSA has been criticised for issuing favourable opinions despite inadequate information for environmental risk assessments (ERA) submitted by applicants and improvements to the Authority’s own ERA have been suggested [6, 8, 9].

The critiques of EFSA and its GMO Panel

EFSA was established in 2002 to provide scientific advice on risks related to the food chain, striving for “scientific excellence” [10]. Notwithstanding, the aim for scientific excellence in both the way EFSA performs risk assessment and the independence of its panel members have been challenged over the years since its establishment.

Conflict of interests

According to EFSA’s own description, one of its key values is independence: “EFSA is committed to safeguarding the independence of its experts, methods and data from any undue external influence and to ensuring that it has the necessary mechanisms in place to achieve this.” [10]. Nevertheless, its independence from economic interests and national and policy influences has been questioned by the European Parliament, NGOs, the media, national governments, and scientists [11,12,13]. Moreover, it has been demonstrated that members of the EFSA Panels have been appointed on the basis of their willingness to reach consensus [13].

The GMO Panel that was responsible for issuing the favourable report on GM maize MON810 has been criticised for more than half of its members having a conflict of interests [14]. Despite this, the Panel states that no member abstained from the discussion on the subject due to any possible conflict of interests [3]. This problem is not specific or restricted to one term of office. The very first GMO Panel, established in 2003, was also criticised on the same grounds and even after the adoption of the new policy on independence in 2011, and its revised version in 2014, the situation of EFSA’s Panels continues to be criticised for containing members with significant conflicts of interests [11, 15, 16].

The existence of conflicts of interests does not necessarily imply malpractice, but may be a source of scientific bias [17,18,19]. An apt illustration might be the influence of the International Life Sciences Institute (ILSI, “a controversial food and biotech industry-funded scientific think tank and lobby group” [14]) on the development of EFSA’s guidance for the risk assessment of food and feed derived from GM crops [20]. The risk assessment of GM crops has been based on a problematic concept of “comparative assessment”, which has been attributed to a network around a chairman of the GMO Panel who was working at the same time for ILSI [20, 21]. The establishment of the industry-friendly concept has been criticised for resulting in a less rigorous investigation of the possible risks posed by GM crops [20].

Quality of science

Another of EFSA’s key values is to provide “high-quality scientific advice” [10]. However, the quality and use of scientific information in its risk assessments has also been criticised both by NGOs and scientists [13, 22].

Despite claiming a strict distinction between the claimed value-free science provided by the Authority and the politically driven management measures taken by the EC, it has been suggested that EFSA’s opinions have been based on a range of normative judgments about what constitutes acceptable or relevant effects [13] and its scientific critique to be laden with social, political, and ethical dimensions [23].

Furthermore, the quality of scientific output produced by the GMO Panel has been criticised in regard to (synthesis drawn from [8, 13, 22,23,24,25,26]):

• Selective use of studies and results of studies

• Inappropriate interpretation of scientific information

• Use of double standards in scrutiny of scientific evidence

• Misleading presentation of study results

• Failure to cite sources of information

• Failure to acknowledge uncertainties of risks

• Failure to call for further research

• Use of double standards in dealing with uncertainties in studies according to the study findings

• Assessments based on untested or questioned assumptions.

In conclusion, the way that EFSA uses scientific information has been significantly challenged and seen to not even meet certain basic scientific standards much less represent “high-quality scientific advice”. The repeated identification of shortcomings in EFSA’s work is of high concern given that the Authority’s outputs serve as a basis for decision-making.

Concept of risk assessment

Although EFSA claims to have developed “good risk assessment practices” [10], the concept of risk assessment that it adopts has been criticised by NGOs, the European Commission and scientists [21, 24, 27, 28].

The point of many critiques has been the concept of “substantial equivalence”, which is the underlying concept for ERA [21]. Specifically, the design of ERA was criticised for solely focusing on the newly expressed protein, insufficient testing of chronic and indirect effects, choice of test organism, and testing procedures [21]. As a result, certain risks are not assessed, e.g., the long-term and combinatorial effects. Furthermore, the effects of herbicide use⁴ associated with cultivation of GM herbicide-resistant crops, and the possibility of uncontrolled spread of GMOs, are disregarded in the risk assessment according to national EU experts on GMOs [28].

The purpose of the study

Against this backdrop, the aim of this study was to critically evaluate how the EFSA GMO Panel performed its environmental risk assessment in a specific case. Given the presumably greater risk to the environment from growing rather than from the import of a GMO, I chose the assessment of risks to non-target organisms from cultivation of Bt maize MON810, the only GM crop grown commercially in the EU over a longer period.

The evaluated EFSA document forms the basis for the EC Draft implementing decision intended to renew the authorisation of MON810 cultivation [29]. Relevantly, the DG environment recommended to progress with the approval process only with updated and complete ERA by EFSA and the European Parliament called on the EC to withdraw its draft [30, 31].

The second aim was to help increase the robustness of advice for future decision-making on GMOs by demonstrating the inherent flexibility in how science is generated and interpreted. The case of GMO risk assessment serving for policy decisions is a perfect example of an issue where “facts [are] uncertain [and] values in dispute”, allowing for different conclusions based on the same results, while “stakes [are] high and decisions urgent” [32]. Under such conditions, an extended peer review can contribute to the quality assurance of the advice used for decision-making [32]. “Extended” here refers to a review that includes other types of experts and stakeholders, and the different quality criteria and modes of reflection of both natural and social sciences [33].

The analysis was guided by the “Reliability Rating and Reflective Questioning” framework developed by Wickson [33] to perform a critical evaluation of Australia’s risk assessment of GM Bt cotton. This method evaluates both the quality of the science used in an assessment, as well as the way that it is used to draw conclusions. In what follows below, I assess both of these components for EFSA’s ERA on MON810.

The main finding of this study is that, based on the critical evaluation of EFSA’s risk assessment, the Authority cannot claim “scientific excellence”, and indeed that the way it treats scientific information leads to the opposite of “high-quality scientific advice”. The analysis confirmed many of the previously identified shortcomings, and, in addition, provides an assessment of the independence and reliability of the source of scientific information on which basis EFSA conducts its work.

This manuscript is an extended version of a conference paper “Science for policy: assessing genetically modified crops in the EU” presented at the STS Conference Graz on Critical Issues in Science, Technology and Society Studies in 2017 [34].

Methods

Subject of the analysis

In this study, I examine EFSA’s “Scientific Opinion on Applications for renewal of authorisation for the continued marketing of food, feed, and seed for cultivation of GM maize MON810” issued in 2009, referred to here as the Opinion [3]. Among other issues, the Opinion deals with “Environmental risk assessment and monitoring”, which consists of chapters “Evaluation of relevant scientific data” and “Post-market environmental monitoring”. Scientific information is cited concerning the assessment of the possible effects of GM maize, including its impacts on human and animal health, abiotic environment, and agricultural techniques, in the former chapter.

Given the nature of the genetic modification, which enables the plant to produce a toxin targeted at a specific pest, the subchapter “Interactions between the GM plant and non-target organisms” forms a crucial part of the environmental risk assessment. This part of the Opinion assesses possible impacts on ten different groups of organisms that are not supposed to be affected by the toxin (Cry1Ab protein or simply Bt protein/toxin).

In this study, I perform an evaluation of the ERA for two groups of non-target organisms: “Pollinating insects: honeybees” and “Soil organisms: earthworms”. I deliberately chose not to assess the groups of organisms that contained studies that have received a lot of publicity and were subjected to EFSA reassessments (natural enemies, non-target Lepidoptera, and water-dwelling organisms). The choice of honeybees as a starting point of the analysis was guided by their importance as pollinators, and by the elevated scientific and popular media attention, they are currently receiving due to their dramatic decline. The results warranted a comparison to another group of organisms to see if the identified shortcomings were only an artefact connected to the work on the honeybee group. Subsequently, I chose earthworms for assessment due to their importance for soil processes and widespread cultural awareness of them as important organisms when compared to other soil organisms considered in the assessment.

Analytical framework

For the purpose of this extended peer review, I adapted the “Reliability Rating and Reflective Questioning” framework for evaluating the quality and use of science for policy [33]. The analysis consists of three steps. In the first step, I assess the reliability of the scientific studies cited in the Opinion. The second step follows with an assessment of the way in which the scientific information was used in the synthesis in the Opinion. By combining these two steps, I then arrive at an evaluation of the adequacy and appropriateness of the Panel’s conclusions. A detailed description of the analysis follows.

Reliability rating

To guide the first step, the critical evaluation of the reliability of the cited studies, Wickson [33] suggests asking four general questions: “Who performed the study? Where was the study performed? How was the study performed? What now?”. After collecting answers to all these questions, I created three specific categories within which I determined scales for ranking the strength of reliability, as outlined in more detail below and summarised in Table 1. The reliability is ranked according to good scientific standards (for the categories affiliation and support, status of the study) and a general principle of ecological relevance (category event). The ranking scale also included examples (e.g., industry in the affiliation and support category) that were not found in this analysis, but which might be identified in other risk assessments. Others performing a reliability rating might choose other categories or rank the values differently reflecting what they consider as relevant and important for the risk assessment. In the following, the categories, the scales, and the rationale for their order are explained.

Table 1 The reliability categories with scales

Category: Affiliation and support

Question: “What was the affiliation of the authors and where did financial support for the study come from?”

Rationale for the order: Reliability rises with increasing independence from the industry producing or marketing GM crops as this poses a conflict of interest. Moreover, industry dossiers have been criticised for an inadequate ERA [6, 9, 35].

• Public universities and public research institutes

• Various including industry and/or applicant mixed affiliation and/or support of the study, which includes a company that produces and/or markets GM crops in general or the specific GM crop assessed in the Opinion

• Industry the study and/or the funding originates from a company which produces and/or markets GM crops

• Applicant the study and/or the funding originates from the company which produces and/or markets the specific GM crop assessed in the Opinion.

Category: Status of the study

Question: “What is the status of the study?”

Rationale for the order: First, I ordered reliability in this category according to the published or unpublished status of a study, with a higher ranking for the former. In general, peer-reviewed studies have a higher reliability than un-reviewed. The last distinction which I made was between original research studies bringing a new body of evidence and meta-analyses or reviews building on results of others. Original studies were assessed as having a higher level of reliability. Although a meta-analysis often has higher credibility than single original studies, it is not the case in the present study. This is because the meta-analysis cited in the Opinion [36] involved various Cry proteins, while only four studies used the Cry1Ab protein expressed in MON810. The use of reviews is questionable as the Opinion itself is a kind of a review and should arguably refer to original literature, where possible.

• Original research, peer-reviewed, published

• Review/meta-analysis, peer-reviewed, published

• Un-reviewed, published

• Un-reviewed, unpublished.

Category: Event⁵

Question: “What event was used?”

Rationale for the order: The properties of the cry protein, and thus the risk it might pose, may differ dramatically between the distinct cry proteins [37]. Similarly, the expression of the protein in different crops and even different maize events can lead to different effects [37]. Therefore, the reliability of studies rises when they use specific Cry1Ab events and the specific event assessed here.

• MON810 the specific Bt maize assessed

• Other Cry1Ab maize other maize event, one that expresses the same protein as the assessed event

• Other Cry1Ab crop other crop than maize that expresses the same protein as the assessed event

• Other Cry crop other crop than maize that expresses other cry protein than Cry1Ab.

For two other categories (location and test material), I did not determine scales for ranking the strength of reliability. However, I noted the proportion of studies within specific categories of methodology cited in the Opinion.

I differentiate among laboratory, semi-field, and field studies within or outside Europe within the category “location” answering the question “Where was the experiment conducted?”. The EFSA guidance for applicants effective at the time of the preparation of the Opinion suggests that field trials may not be required if experiments under controlled laboratory, growth room, or glasshouse conditions do not identify sensitivity in exposed species [38]. Nevertheless, Lang et al. strongly advocate field or semi-field tests regardless of early tier results, because the planting of GM crops in different biogeographical regions can result in different ecological consequences and because they regard most laboratory studies as overly simplistic [39]. Therefore, I note the proportion of references citing laboratory and field studies taking into consideration the reported effects on the NTOs that were tested.

A differentiation among purified Cry1ab protein, plant material, and the whole plant is made within the category “test material” answering the question “What test material was used?”. The use of GM plants and their products is suggested for the first tier in the EFSA guidance [38]. Andow and Hilbeck argue that for an adequate ERA for NTOs, both long-term exposure to the transgene product and whole-plant tests are needed [40]. Spök et al. recommend the use of whole GM plants to account for the “differences between plant and bacterially produced proteins as well as host plant composition” [35]. I, therefore, note the proportion of references citing experiments with bacterially produced proteins, plant material, and whole plants.

Reflective questioning

The second step of the analysis was focused on assessing the ways in which scientific information from the cited studies was used by the Panel. In this procedure, I was inspired by the questions suggested by Wickson [33] and, therefore, reflected on how the scientific study was presented, the accurateness of the references, the representation of the study’s methods and results, if information was selected and how, if critique was applied and how, what uncertainties and assumptions were embedded in the study, and whether they were communicated.

The shortcomings that were identified formed a pattern similar to the one previously found by Wickson [33]; therefore, I adopted her categories of misuse of scientific information. These included: Lack of critique: an uncritical reporting of the study and its results; Misleading presentation: a style of presentation that misleadingly suggests a higher strength in evidence than actually exists; Misquotation: an inaccurate citation of a study in support of a statement; Selection of information: information from a study only being selectively reported in the assessment; Inconsistent presentation: the results of a study being cited differently in various parts of the assessment; Qualifications removed: the study specifically indicating that further research is required, but this is not reported in the assessment; Misrepresentation: an inaccurate citation of a study’s method. I also added a category: Irrelevant critique: study subjected to critique that does not correspond to the content of the original study. Examples of each category will be described in the results.

Evaluation of the adequacy and appropriateness of the conclusions

Finally, I evaluated if the conclusions made by the Panel for each section of NTOs corresponded to the findings from literature through combining the results from the assessment of the reliability of the studies and the way they were used.

In accordance with the aim of the study, the reliability, the use of science, and the adequacy and appropriateness of the conclusions were compared to the state of knowledge at the time of the preparation of the Opinion. All the evaluation criteria are, therefore, based on literature that was available before 2009.

Evaluation of completeness of scientific information

In addition, I examined whether the Opinion omitted relevant scientific information that was available before its promulgation. This was done by searching the Web of Science, Scopus, Proquest Central, and CAB Direct scientific databases. For details of search queries and results, refer to Table 2.

Table 2 Settings and results of literature search for additional relevant studies

Results

Honeybees: The reliability of the cited studies

Here, an assessment of the reliability of the studies cited in the Opinion is provided (the first part of the analysis). In the section dealing with possible risks to honeybees, eight scientific works are cited in the Opinion. Overall, the scientific information stems from reliable sources, but the reliability of the studies themselves varies (Fig. 1).

Fig. 1

Reliability rating in respective categories for studies cited in the section “Pollinating insects: honeybees”. Strength of reliability rises upwards. Columns indicate respective studies. For detailed description, see “Analytical framework” section and Table 1

All of the cited studies were peer-reviewed and published. Five original research studies, two reviews, and one meta-analysis were used. The affiliation of the authors and financial support for the studies is predominantly of a public character. Only two studies were authored by researchers affiliated to public institutions and industry in collaboration. The meta-analysis [36] was written by authors employed by universities and Monsanto, the company which developed markets and holds a patent on the MON810 maize. Various stakeholders including companies developing GM plant applications at the EU level contributed to the review of information in the BEETLE report [41].

The ecological reliability was not assessed for the two reviews and the meta-analysis because of mixed information in the studies that were employed. Four of the five original studies explored the hazard to honeybees. Only one also employed, along with the laboratory experiments, a field trial in USA [42]. Another three studies were performed only in the laboratory [43,44,45]. Intact Bt maize plants and plant material were used only in one study, which was performed with another Cry1Ab maize event (Bt11) [42]. Pollen from MON810 event was used in one study [45], while purified cry toxin was used in three cases [43,44,45]. One original study aimed to estimate the exposure of honeybee larvae to the toxin in pollen by conducting European semi-field trials with non-GM maize plants [46].

In conclusion, the scientific information cited in the Honeybees section comes predominantly from sources of high reliability in terms of authorship and financial support (six out of eight studies) and the status of the study (five out of eight). The ecological reliability of the studies for which the assessment of methods was possible was medium. Although both laboratory and (semi-)field studies were cited, no field or semi-field study that employed the actual event MON810 that was assessed was conducted. Honeybees were exposed both to the purified toxin and whole plants or plant material. However, only one study used plant material from MON810 maize.

Honeybees: The misuse of scientific information in the Opinion

This section describes how the cited scientific information was used in the EFSA’s Opinion [3]. The identified shortcomings are summarised in the use of science categories and indicated in Table 3.

Table 3 EFSA’s use of scientific information from studies cited in the section “Pollinating insects: honeybees”

Misquotation

In the section assessing the risks to honeybees, three studies are cited [42,43,44] by EFSA to support the statement that Bt maize/Cry1Ab protein causes “no direct adverse effects on larvae and adult survival” [3] which is true only for adults as none of the studies used larval stages. Thus, the Opinion makes a broader claim than that which can be supported by the cited evidence.

Selection of information

From other cited works, information is chosen selectively. Only parameters that were not affected are mentioned: “no significant differences were reported in honeybee mortality, syrup consumption and olfactory learning performance” [3], while the one that was negatively influenced is omitted: “foraging activity decreased during and after exposure to Cry1Ab protein” [43]. The authors of this study concluded that “negative effects of Cry1Ab protein on bees cannot be excluded over time.” [43]; however, this is also left out of the EFSA Opinion [3].

Misquotation and qualifications removed

The Opinion continues by concluding that negative effects on foraging behaviour and learning performance are unlikely, despite the fact that the cited study [44] did not explore the effects of the Cry1Ab protein on foraging behaviour. The authors report instead “disturbances in the feeding behaviours” and the “potential effect of sublethal doses of Cry1Ab on learning performance”, and note that this may be unlikely to occur under natural conditions, stressing the need for further research [44].

Selection of information and misrepresentation

Similarly, the study of Rose and colleagues [42] is presented in the Opinion as a field study, which omits laboratory experiments that revealed a difference between the Bt and non-Bt treatment: “More non-Bt than Bt pollen was consumed per bee.” [42]. Although the study authors attribute the difference to pollen quality, the significance of this and the possible negative effect of Bt protein on feeding behaviour cannot be excluded, especially in the context of other cited studies.

From the study of Babendreier et al. [45], only the results for the development of the hypopharyngeal glands are presented in the Opinion, leaving out other relevant information from the study. Another aim of that study was to investigate if the glands could take up ingested transgene products [45]. Traces of Cry1Ab protein were detected in the glands, leading the authors to conclude that the protein is unlikely to be transferred to larvae or, if so, in small amounts [45]. The Opinion thus again uses the information selectively and fails to address other possible routes of exposure.

Earthworms: The reliability of the cited studies

Fourteen scientific works are cited in the section dealing with possible risks to earthworms in the Opinion. They are of reliable origin and overall of medium reliability (Fig. 2). The reliability of one study was not assessed as the content was irrelevant for the risk assessment [47].

Fig. 2

Reliability rating in respective categories for studies cited in the section “Soil organisms: earthworms”. Strength of reliability rises upwards. Columns indicate respective studies. For detailed description, see “Analytical framework” section and Table 1

The vast majority of cited studies were authored by researchers affiliated with universities or public research institutions and financially supported by public sources, making them receive a high reliability rating within this category. Only one review was prepared by various stakeholders, including industry [41]. All the cited studies were subjected to the peer-review process and published. Nine of the studies presented original research results, while four were reviews [37, 41, 48, 49]. The ecological reliability was not assessed for the reviews because of mixed information in the studies that were employed.

Seven cited sources which focused on hazard identification used either whole plants (four out of seven) [50,51,52,53] or plant material (five out of seven) [50, 51, 54,55,56] in European fields (three out of seven) [52, 53, 55] or under laboratory conditions (five out of seven) [50, 51, 54,55,56]. The MON810 event was employed in the majority of the studies (five out of seven) [50, 52,53,54, 56], and other Cry1Ab maize was used too (four out of seven studies) [51, 52, 55, 56].

Two studies elaborating on the release of cry toxin from root exudates and its persistence in soil were performed under laboratory and, in USA, semi-field conditions with whole plants and plant material from MON810 and other Cry1Ab maize [57, 58].

In conclusion, the scientific information cited in the Earthworms section is from predominantly reliable sources: 12 out of 13 relevant studies are of public origin (both authorship and financial support) and 9 out of 13 studies are original research articles. The ecological reliability of the original studies is high owing to the balanced ratio of laboratory and field experiments, use of plant material, and, mostly, the MON810 event that was assessed. Two studies used whole MON810 plants under European field conditions.

Earthworms: The misuse of scientific information in the Opinion

The purpose of this section is to illustrate the categories of respective shortcomings identified in the use of scientific information by EFSA in the risk assessment for earthworms [3]. Therefore, I describe only selected cases here, for further results please refer to Additional file 1. The summary of use of science is provided in Table 4.

Table 4 EFSA’s use of scientific information from studies cited in the section “Soil organisms: earthworms”

Misquotation

The EFSA Opinion cites an irrelevant study [47] to support a statement about how Cry1Ab protein enters soil by plant material. Webster and colleagues [47] investigated the chemical composition and decomposition of birch and conventional maize pollen in soil. No link can be traced to the source of the Cry1Ab protein from plant material on the basis of this study. To make sure that it was not a citation error, the Web of Science database was searched, but did not reveal any study authored by Webster that could be cited to support the statement as it appears in the Opinion.

Inconsistent presentation

The Opinion cites several studies [50, 51, 54,55,56] to support the claim that “laboratory studies performed on some earthworm species (…) did not reveal significant adverse effects” [3]. Nevertheless, among these studies are two [50, 55] that reported negative effects on reproduction and weight in adult earthworms from feeding on Bt maize leaves, respectively. The introductory claim, therefore, is simplistic and misleading.

Lack of critique

The negative findings reported by Vercesi et al. [50] are presented only later in the Opinion text, where each of the cited studies is discussed. It seems, however, that the original interpretation of the results by the authors was uncritically adopted by EFSA. EFSA along with Vercesi et al. [50] consider the concentration of Bt biomass in soil in which the negative effect was observed as relatively high and, therefore, question the ecological significance of the effect under natural conditions. However, this reasoning is based on erroneous assumptions. The estimation of exposure of earthworms to the toxin only considers the amount of the cry protein from Bt maize leaves left in the soil while omitting other routes of exposure to the protein (e.g., other parts of maize plants, root exudates, and wildlife faeces). Furthermore, Vercesi et al. [50] assume a shorter persistence of the protein in soil, although evidence of its accumulation [57] and longer degradation times [59] exists and is cited in the study itself and in the Opinion.

Further examples of lack of critique will be discussed in this papers’ section “Earthworms: The adequacy and appropriateness of the conclusions”.

Misquotation

Although the findings of negative effects on the weight of adult earthworms observed by Zwahlen et al. [55] are discussed further in the Opinion, the results are represented imprecisely. According to EFSA “the growth of adults (…) significantly declined thereafter in Bt-exposed earthworms up to 200 days.” [3], whereas Zwahlen et al. [55] indicate more importantly, that after 200 days, Bt maize-fed earthworms lost weight, whereas non-Bt maize-fed earthworms gained weight.

Furthermore, the study is incorrectly cited to support parameters that were not examined by the authors: “The ingestion of the Cry1Ab protein by earthworms was confirmed through the detection of the protein in their gut and faeces (e.g., Zwahlen et al. [59]).” [3]. Probably, another study [51] was meant instead, where the cry protein was determined in the gut and faeces of L. terrestris. Similarly, Zwahlen et al. [55] did not perform field surveys “during the cultivation of Bt-maize” [3], as the Opinion suggests, but used dried Bt maize leaves instead.

Selection of information

The Opinion lists possible explanations of the observed effect of weight loss, which agree with those discussed by Zwahlen et al. [55]. However, another possible explanation indicated by the authors is omitted. According to them [55], significantly higher lignin content in the Bt maize variety used in their experiments and reported by [60] could lead to “a delayed litter degradation and decomposition by microbes or to a lower nutritional quality for earthworms in the long term.”, an effect deserving further attention.

Critique

Interestingly, it is the study that reported a negative effect [55] where EFSA applies the second of only two critiques of the cited science. EFSA appropriately criticises that the authors did not quantify the reproductive activity recorded during the experiment, which would allow for a potential alternative explanation of the weight loss due to “differences in timing or production of cocoons in the Bt-maize treatment.” [3]. If true, this explanation could have been, however, considered itself as an adverse effect and would have needed yet another explanation.

Qualifications removed

Finally, when discussing the findings reported by Zwahlen et al. [55], the GMO Panel concludes that: “it is not possible to make any inference on long-term effects on natural populations” [3], without calling for any further research. This was despite the fact that the authors of the cited study claimed that longer experiments were necessary and recommended further research to include other life history traits and fitness parameters, the use of newly hatched juveniles, and species common for areas with the respective GM crop.

Misquotation and qualifications removed

In the Opinion, the study of Clark and Coats [56] is incorrectly cited as having found no negative effects on the reproduction of earthworms, an impact that was not actually examined in the study. This parameter was examined in only one cited study [50] and in one species, Aporrectodea caliginosa. The adverse effect on the cocoon hatchability rate, which decreased significantly with increasing Bt concentration, as reported by the authors, is considered by EFSA to be questionable under natural conditions. The Opinion statement citing Clark and Coats [56] that there is no negative impact on reproduction is incorrect and counter to the findings of Vercesi et al. [50], who recommend further research with earthworms in field experiments under long-term cultivation.

Irrelevant critique

The Opinion criticises the assessment of growth in adult earthworms purportedly performed in the study of Saxena and Stotzky [51]. Interestingly, the review by Clark et al. [37] cited in the Opinion also notes that the use of growth in adult earthworms in this study [51] was probably not an appropriate assessment endpoint [37]. It seems that EFSA adopted this critique without referring to the original article [51]. The authors of that article described lack of effects on survival and weight and actually made no conclusions about growth of earthworms [51]. The critique is, therefore, irrelevant and the representation of the study imprecise.

Misleading presentation

The results of the research project “Monitoring of the environmental effects of the Bt gene” [52] are imprecisely represented, suggesting a greater strength in evidence for no effects on the population density or biomass of Lumbricidae. The statement in the Opinion that no differences between Bt and non-Bt-maize soils were reported “(…) at 5 sites during 4 years of maize cultivation in field” [3] could be interpreted as if sampling had been undertaken at multiple places per year in the course of 4 years. However, the sampling was performed once at each site in the second year or third year of the 4-year cultivation period [52].

Discussion

Honeybees: The adequacy and appropriateness of the conclusions

EFSA concludes that “…the likelihood of adverse effects on honeybees is expected to be very low.” [3]. This conclusion is based on information from eight scientific works stemming from sources of high reliability and medium ecological reliability. Nevertheless, taking into account the three studies reporting negative effects, the ecological reliability of the body of evidence is insufficient as a result of the lack of higher tier studies. Although both laboratory and (semi-)field studies using purified Cry1Ab protein, and in one case, plant material from MON810 were cited, no (semi-)field study employed the MON810 event that was assessed.

The Opinion admits only one possible route of exposure of honeybees to the Cry1Ab protein which is through pollen [41]. Other exposure pathways such as nectar [61] and possibly worker jelly [45] are left out.

Five original research studies, one review and one meta-analysis are used to assess the hazards and exposure of honeybee larvae and adults to the Cry1Ab protein. A close look reveals, however, that larval stages are addressed in only two studies, both of which are presented in the meta-analysis by Duan et al. [36]. Interestingly, the Opinion only cites the meta-analysis for these studies’ findings and not the original sources. The meta-analysis focused on various cry proteins and life stages, but only four studies exposed honeybees to the Cry1Ab protein: larvae were used in a peer-reviewed and published study not cited in the Opinion [62] and in an unpublished study by Monsanto (Maggi and Sims, cited in [36]); and adults in a study cited in the Opinion [42] and another unpublished study by Monsanto (Maggi and Sims, cited in [36]). Although meta-analyses are usually attributed a higher credibility than single studies, the use of the study by Duan et al. [36] is of questionable value for the Opinion assessment as it analyses 25 studies that used various cry proteins, but only four studies exposed honeybees to the Cry1Ab protein expressed in MON810. Furthermore, half of the experiments cited are non-peer-reviewed studies issued either by the US Environmental Protection Agency or the Monsanto Company. The first and third authors of the meta-analysis have a conflict of interests, being employed by Monsanto, which produces and markets Bt crops. Though clearly declared in the original paper, none of these limitations are reflected in the Opinion.

Adverse effects were observed in three of four cited original research studies that exposed honeybees to purified Cry1Ab protein and/or plant material. However, in the Opinion the negative effects of the Cry1Ab protein on feeding behaviour and possibly on learning performance are left out [42,43,44]. Only negative impacts presented by Ramirez-Romero et al. [44] as being unlikely to occur under natural conditions are cited in the Opinion. The authors of all the original studies, in contrast, suggest that negative effects cannot be excluded and call for further research [42,43,44]. EFSA’s selective use of information does not fulfil the requirements of good scientific practice.

Selective reporting of results in the Opinion seems to be based on the assumption that adverse effects will not occur in nature. This belief assumes a known and stable concentration of the Cry1Ab protein produced by plants and that pollen is the only source of the protein. Both suppositions were being questioned at the time the assessment was performed, even in some studies cited in the Opinion [45, 61, 63, 64]. Moreover, a negative effect was also observed using pollen from Bt11 maize [42], which expresses the Cry1Ab protein at a comparable concentration to the assessed event MON810 [3]. The assessment is further based on studies using bacterially produced purified Cry1Ab protein. Such studies assume interchangeability of surrogate proteins with plant proteins and assume no differences between the conventional and GM plants, except for the presence of the novel Bt protein. These assumptions are challenged in the scientific literature [35, 65] including a study cited in the Opinion by Clark and Coats [56].

The conclusion continues with a generalisation: “The EFSA GMO Panel has no reason to consider that maize MON810 will cause reductions to pollinating insects…” [3]. The whole section concerns, however, research on one species exclusively: the honeybee. Studies testing other pollinating species that are available are lacking in the Opinion, and thus, the general conclusion on pollinating insects is unwarranted.

A literature search for further relevant works not cited in the Opinion revealed three articles that present additional information or provide further support for claims that are insufficiently backed in the Opinion. For example, the wrongly supported claim of no adverse effects on larvae survival could cite the study of Hanley and colleagues, who report “no effect on honey bee larval mortality, pupal mortality, pupal weight, or haemolymph protein concentration” compared to controls [62]. Babendreier et al. [66] provide further information concerning a potential hazard to honeybees not discussed in the Opinion: “Neither Bt-maize pollen nor high concentrations of Cry1Ab significantly affected bacterial communities in honeybee intestines.”. The third study showed that neither feeding honeybees with pollen from a Cry1Ab maize nor contact with its tassels had an effect on honey bee mortality [67].

The additionally identified studies are of equivalent quality to the cited ones, i.e., original research articles, peer-reviewed, published, and using maize producing the Cry1Ab protein or the protein itself. Two of them were included in the cited review [68] and one in the meta-analysis [36]. In these cases, the Opinion referred to secondary literature instead of using original sources of information.

By omitting relevant studies, misquoting, and selectively citing results, the GMO Panel failed to perform a thorough, high-quality, risk assessment for honeybees. They conclude that: “… the likelihood of adverse effects on honeybees is expected to be very low. The EFSA GMO Panel has no reason to consider that maize MON810 will cause reductions to pollinating insects that are significantly greater from those caused by conventional farming.” [3].

A more nuanced and evidence-based conclusion drawn from the studies that are cited and omitted could read: “The risk assessment for honeybees could not be completed because of the lack of essential data. Further research is needed to confirm the ecological relevance of adverse effects on feeding behaviour and learning performance reported in laboratory studies. The risk assessment for other pollinating insects could not be completed because of the lack of essential data.”.

Earthworms: The adequacy and appropriateness of the conclusions

In the Opinion, the conclusion of the section on earthworms reads: “The EFSA GMO Panel is of the opinion that there is no evidence to indicate that the placing of maize MON810 and derived products on the market is likely to cause adverse effects on earthworms in the context of its proposed use.” [3]. It cites four reviews, nine original research studies, and one study not relevant for the assessment. Although the scientific information stems from generally reliable sources (authors affiliations and financial support) and the ecological reliability of the studies is high, the use of the information by EFSA in the Opinion is problematic. Studies are often misquoted and information selectively used. The Opinion does not require any further research, which contrasts with the recommendation by two cited studies [55, 56] and the cited findings of negative effects from Bt maize on reproduction and weight [50, 55]. The literature search did not reveal any additional relevant studies for the ERA of earthworms.

The Opinion lists a few sources of the Cry1Ab protein in soil, but does not include others from the faeces of wildlife and livestock and effluents from biogas facilities as reported in a cited report [41]. On the other hand, a cited exposure route of earthworms to the protein through plant material is wrongly supported by an irrelevant study [47].

The authors of the Opinion fail to reflect and critique the methods and results of the cited studies. Nevertheless, a critique could be applied, e.g., to the duration of the experiments and choice of the test organisms. In virtually all of the cited studies (except for [55] which performed a 200-day experiment), the exposure time was short (ranging from 28 to 98 days) compared to the life spans of earthworms, which generally live for more than 4 years in the laboratory and approximately 1 year in natural surroundings (Edwards, as cited in [69]).

Moreover, the relevance of particular test species for arable land could be questioned. Only two of the cited studies [50, 54] used the most appropriate species, A. caliginosa, which is abundant in European agricultural soils [52, 70]. Others used Eisenia fetida [56], a species scarce in agricultural soils preferring areas with high amounts of organic matter or Lumbricus terrestris [51, 54, 55] for which the evidence of occurrence in these soils is ambiguous.

Criticism was applied only to that study which reported an adverse effect [55] or adopted from a review in which the study’s method was criticised, even though the critique was irrelevant [51]. The widespread lack of critique or its inappropriate application does not inspire confidence in the scientific rigour of the EFSA.

Four studies examined the effects of Bt maize plant material on adults and three on juveniles in laboratory and/or field conditions. The exposure had no effect on earthworm survival and growth, and the population density, number of species, and biomass of Lumbricidae. However, according to Vercesi et al. [50], Bt maize residues had a negative impact on cocoon hatchability. The relevance of this effect under natural conditions is questioned by the authors and EFSA and field experiments to confirm the results which are recommended by the authors but not by EFSA. Significant weight loss in Bt maize-fed earthworms was observed by Zwahlen et al. [55]. These authors conclude that sublethal long-term effects cannot be excluded and further research is needed. In the Opinion, studies reporting negative effects are either criticised or the ecological relevance thereof is questioned, diminishing the results. The findings of adverse effects thus do not find their way into the conclusion.

The report by Bartsch et al. [41] cited several times in the Opinion concludes that the likelihood of effects on non-target organisms due to accumulation of toxic compounds is “low for Bt maize, but uncertainties remain concerning long-term effects on soil NTOs and soil ecological functions and concerning effects of specific Bt proteins if Bt maize is cultivated continuously on same fields” [41]. Although earthworms are not explicitly mentioned there, EFSA communicates the uncertainties about soil organisms in neither any section relating to soil organisms nor in the overall conclusion of the Opinion.

By failing to indicate uncertainties, to apply critique to all studies’ methods, misquoting and selectively citing results, the GMO Panel did not perform a thorough or reliable risk assessment for earthworms. Their conclusion in the Opinion reads: “The EFSA GMO Panel is of the opinion that there is no evidence to indicate that the placing of maize MON810 and derived products on the market is likely to cause adverse effects on earthworms in the context of its proposed use.” [3].

A more appropriate conclusion based on the studies that are cited could read: “The risk assessment for earthworms could not be completed because of the lack of essential data. Further research is needed to confirm the ecological relevance of adverse effects on reproduction and weight in adult earthworms reported in laboratory studies.”

Overall discussion

The critical evaluation of the risk assessment for the two groups of non-target organisms shows that the scientific information that is cited comes predominantly from reliable sources and that the ecological reliability of the studies is medium. However, the body of evidence in both groups is insufficient to draw conclusions on risk, particularly in the context of reported adverse effects in lower tier experiments. The GMO Panel, nonetheless, arrived at the conclusion that “…the likelihood of adverse effects on non-target organisms or on ecological functions [is expected to be] very low…” [3]. Furthermore, the results of my research indicate the following failures by the EFSA GMO Panel: selective use of information, neglect of relevant available studies, misquotation of studies, failure to acknowledge uncertainties, to call for further research where needed, and to critically interpret studies. These results are consistent with the current literature [8, 13, 22,23,24,25,26].

Limitations of the study

This study examined the risk assessment for two out of ten groups of non-target organisms performed by EFSA. The results of the two selected groups showed a similar pattern regarding the reliability of scientific sources and the use of scientific information, allowing me to conclude that the identified shortcomings were not accidental. It has been argued that EFSA subjected studies reporting adverse effects on other NTOs (e.g., lady beetles, lacewing larvae, Lepidoptera, and Daphnia magna) to a higher level of scrutiny [26, 71]. Therefore, as stated in the Methods, I did not choose to evaluate other groups of NTOs assessed in the Opinion that contain highly debated studies (natural enemies, non-target Lepidoptera, and water-dwelling organisms). However, I encourage others to evaluate other elements of the ERA. Various reviewers can have different views and priorities guiding their analysis, and thus enrich the knowledge base of the ERA used for decision-making.

The reliability of the studies was ranked according to a general principle of ecological relevance and good scientific standards. As mentioned in the Methods section, others performing a reliability rating might choose other categories or rank the values differently based on what they see as relevant and important for a high-quality ERA. However, in my view, it would not invalidate the current analysis, but would rather broaden the risk assessment and further illustrate that science-based decision-making is inherently interwoven with values and judgments that affect, for example, judgements concerning what constitutes quality and reliability in science for policy.

The aim of this critical evaluation was not to perform a “true risk assessment”, but rather to help increase the robustness of advice for future decision-making on GMOs by demonstrating the inherent flexibility in how science is interpreted. Further research would reveal if the identified shortcomings were specific to this case or characterise a more widespread pattern of behaviour.

Conclusions

According to EFSA’s own words, the Authority aims to “provide high-quality scientific advice based on the expertise of its network of scientists and staff and the quality of its science-based information and methodologies, which are grounded in internationally recognised standards.” [10].

The results of this extended review of a section of the Scientific Opinion issued by the EFSA GMO Panel show that the science-based information used in the Opinion originates predominantly from reliable sources, with authors affiliated mostly to public sector and financial support for the studies coming from public resources. All cited studies have been peer reviewed and published, two-thirds are original research articles and one-third is constituted by reviews and a meta-analysis. The overall ecological reliability of the studies is medium. Although the results from experiments under laboratory and (semi-)field conditions with purified toxin and whole plants or plant material were reported, no (semi-)field honeybee study employed the MON810 event that was assessed and only two earthworm studies used whole MON810 plants under European field conditions. The duration of the experiments and choice of the test organisms, particularly in the earthworms’ section, make these studies less reliable.

However, how the Panel uses the scientific information was particularly concerning and indicates the opposite of “high-quality scientific advice”. Studies are often misquoted, using a wrong citation or representing the results and methods imprecisely. Information from the studies is used selectively, in particular leaving out negative effects and further research requirements. Critique is only selectively applied, once to a study reporting a negative sublethal effect on earthworms and in the second case borrowed from another source and applied inadequately. Critical reflection on the methods used in all cited studies would allow for more balanced assessment. Similarly, EFSA does not communicate any uncertainties, although these are sometimes explicitly stated in the cited literature, and does not provide its own reflection on the relevance of the results under natural conditions. Moreover, the cited information is not comprehensive, as evidenced by additional relevant articles found through a literature search but not used in the Opinion.

The body of referenced evidence is insufficient to draw conclusions on risk, especially taking into account the reported adverse effects on honeybees and earthworms in lower tier experiments. Thus, the following conclusions of the GMO Panel are inappropriate and misleading: “… the likelihood of adverse effects on honeybees is expected to be very low. The EFSA GMO Panel has no reason to consider that maize MON810 will cause reductions to pollinating insects that are significantly greater from those caused by conventional farming.” and “…there is no evidence to indicate that the placing of maize MON810 and derived products on the market is likely to cause adverse effects on earthworms in the context of its proposed use.” [3]. More robust and evidence-based conclusions drawn from the cited studies and additional relevant ones could read: “The risk assessment for honeybees could not be completed because of the lack of essential data. Further research is needed to confirm the ecological relevance of adverse effects on feeding behaviour and learning performance reported in laboratory studies. The risk assessment for other pollinating insects could not be completed because of the lack of essential data. The risk assessment for earthworms could not be completed because of the lack of essential data. Further research is needed to confirm the ecological relevance of adverse effects on reproduction and weight in adult earthworms reported in laboratory studies.”.

My results add to the substantial body of research that has demonstrated that EFSA’s claim to scientific excellence is seriously compromised [8, 13, 22,23,24,25,26, 71]. As a result, EFSA’s credibility in scientific assessment of GM crops can no longer be safely relied upon.

Those responsible for food safety, and indeed food security, within the EU would be well advised to reconsider the scientific reliability of the advice given by EFSA in this area and to seek alternative and, perhaps, less complacent sources of advice.

Relevantly, these results would support the call on the EC to withdraw its Draft implementing decision to renew the authorisation of MON810 cultivation voiced by the European Parliament and NGOs, criticising among others the incomplete environmental risk assessment by EFSA [31, 72, 73].