‘In the Line of Fire’: Science news survey results (COVID-19 researchers)
Cathleen O’Grady and Tim Errington
25 March 2022
Methods
To survey researchers who had published on COVID-19, we extracted a list of corresponding author email addresses from Web of Science, drawing from all research articles, editorials, and review papers that mentioned COVID as a keyword. We then limited the pool to 35,830 email addresses that appeared more than once (indicating multiple publications) and emailed a random selection of 9955 researchers. This was the limit set by our institutional subscription to Alchemer, the survey platform we used. Recruitment and reminder emails, and a printout of the survey from Alchemer, are available at osf.io/3bn9u.
Because of the sensitive nature of the subject, Science applied for ethical review of the survey through BRANY (Biomedical Research Alliance of New York), an independent review board. The IRB protocol (BRANY SBER IRB #: 21-141-972) is available at osf.io/3bn9u.
Cathleen O’Grady, a contributing correspondent for Science, designed and conducted the survey, analyzed the data and published a story about it in the news section of Science magazine, March 25, 2022. Meta-scientist Tim Errington advised on the IRB process, survey methods, and statistical analysis. Martin Enserink, International News Editor at Science, provided editorial input.
Of the 9955 emails sent, 9585 were delivered, with some bounced emails presumably due to researchers changing institutions. We received 511 responses and removed one we questioned because it had identical answers across multiple questions. A similar survey was sent to 59,653 members of the American Association for the Advancement of Science, Science’s publisher, representing a wide range of disciplines; 1281 responded. In order to allow participants to share only the information they were comfortable sharing, and in line with the IRB-approved consent procedure, all questions were optional. This means that many questions were not answered by all participants, and have some “NA” (empty) responses.
Below, we show summary results for each question in the survey. As with any survey, it is very likely that certain kinds of people responded more than others. Targets of abuse may have been more likely to respond, and in order to try to limit the effect of this, our recruitment email specifically asked people with no experience of harassment to fill in the survey. However, the skew could also operate in the opposite direction: People who have experienced harassment could also be more worried about anonymity or less willing to discuss their experiences. Either way, the fact that respondents self-select into filling in the survey mean that the results cannot be taken to represent all COVID-19 scientists, or scientists more generally. These results also rely on self-report and on invididuals’ interpretations of our questions, which further undermine the quality of all survey data. Results should all be interpreted with caution.
Because of the number of different questions and because we did not have specific hypotheses about our results, we have treated this as an exploratory dataset. We have not used significance testing, which could lend undue weight to tentative findings, instead calculating only descriptive statistics and effect sizes. For a discussion of the inappropriate use of null hypothesis significance testing, see: Szucs, D., & Ioannidis, J. (2017). When null hypothesis significance testing is unsuitable for research: a reassessment. Frontiers in human neuroscience, 11, 390.
For the sake of transparency, and to allow further analysis of the dataset, data and analysis code are available at osf.io/3bn9u. However, given the sensitive nature of the subject, the public dataset has been anonymized, removing all potentially identifying details (including answers related to location, discipline, and demographics). We are open to collaboration in further analyzing the data, but access to the full dataset will be dependent on additional ethical review. Please contact us at intimidationsurvey@cathleenogrady.com if you are interested in this.
Section 1: Discipline, publications and public attention
Have you read and understood the above consent statements and do you agree to participate in the survey?*
All rows without consent were removed during data cleaning.
Please choose the discipline(s) that most closely describe(s) your COVID-19 publication(s)
Participants could select multiple discipline options. Free-text “other” responses covered a wide range of disciplines, including ethics, history of medicine, nutrition, and aerosol physics.
Approximately how many papers and preprints have you published or co-authored on COVID-19 since the start of the pandemic?
Participants could choose only one option. Despite the list of email addresses from Web of Science being limited to those that appeared more than once, three respondents selected “0 to 1”.
