Metascience (also known as meta-research or evidence-based research) is the use of scientific methodology to study science itself. Metascience seeks to increase the quality of scientific research while reducing waste. It is also known as "research on research" and "the science of science", as it uses research methods to study how research is done and where improvements can be made. Metascience concerns itself with all fields of research and has been described as "a bird's eye view of science." In the words of John Ioannidis, "Science is the best thing that has happened to human beings ... but we can do it better."
in 1966, an early meta-research paper examined the statistical methods of 295 papers published in ten high-profile medical journals. It found that, "in almost 73% of the reports read [...] conclusions were drawn when the justification for these conclusions was invalid." Later meta-research identified widespread difficulty in replicating results in many scientific fields, including psychology and medicine. This problem was termed "the replication crisis". Metascience has grown as a reaction to the replication crisis and to concerns about waste in research.
Many prominent publishers are interested in meta-research and in improving the quality of their publications. Top journals such as Science, The Lancet, and Nature, provide ongoing coverage of meta-research and problems with reproducibility. In 2012 PLOS ONE launched a Reproducibility Initiative. In 2015 Biomed Central introduced a minimum-standards-of-reporting checklist to four titles.
The first international conference in the broad area of meta-research was the Research Waste/EQUATOR conference held in Edinburgh in 2015; the first international conference on peer review was the Peer Review Congress held in 1989. In 2016, Research Integrity and Peer Review was launched. The journal's opening editorial called for "research that will increase our understanding and suggest potential solutions to issues related to peer review, study reporting, and research and publication ethics".
Areas of meta-researchEdit
Metascience can be categorize into five major areas of interest: Methods, Reporting, Reproducibility, Evaluation, and Incentives. These correspond, respectively, with how to perform, communicate, verify, evaluate, and reward research.
Metascience seeks to identify poor research practices, including biases in research, poor study design, abuse of statistics, and to find methods to reduce these practices. Meta-research has identified numerous biases in scientific literature. Of particular note is the widespread misuse of p-values and abuse of statistical significance.
Meta-research has identified poor practices in reporting, explaining, disseminating and popularizing research, particularly withing the social and health sciences. Poor reporting makes it difficult to accurately interpret the results of scientific studies and to identify biases and conflicts of interest in the authors. It is thought to contribute to problems in reproducability. Solutions include study registration, implementation of reporting standards, and greater transparency in scientific studies (including better requirements for disclosure of conflicts of interest).
The replication crisis is an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. While the crisis has roots in the meta-research of the mid to late 1900s, the phrase "replication crisis" was coined in the early 2010s as part of a growing awareness of the problem.
Metascience seeks to create a scientific foundation for peer review. Meta-research evalutes peer review systems including pre-publication peer review, post-publication peer review, and open peer review. It also seeks to develop better research funding criteria.
Metascience seeks to promote better research through better incentive systems. This includes an evaluation of the accuracy, effectiveness, costs, and benefits of old and new approaches to ranking and evaluating the performance, quality, value of research, individuals, teams, and institutions.
Clinical research in medicine is often of low quality, and many studies cannot be replicated. An estimated 85% of research funding is wasted. Additionally, the presence of bias affects research quality. The pharmaceutical industry exerts substantial influence on the design and execution of medical research. Conflicts of interest are common among authors of medical literature and among editors of medical journals. While almost all medical journals require their authors to disclose conflicts of interest, editors are not required to do so. Financial conflicts of interest have been linked to higher rates of positive study results. In antidepressant trials, pharmaceutical sponsorship is the best predictor of trial outcome.
Blinding is another focus of meta-research, as error caused by poor blinding is a source of experimental bias. Blinding is not well reported in medical literature, and widespread misunderstanding of the subject has resulted in poor implementation of blinding in clinical trials. Furthermore, the success or failure of a blind is rarely measured or reported. Research showing the failure of blinding in antidepressant trials has led some scientists to argue that antidepressants are no better than placebo. In light of meta-research showing failures of blinding, CONSORT standards recommend that all clinical trials assess and report the quality of blinding.
Studies have shown that systematic reviews of existing research evidence are sub-optimally used in planning a new research or summarizing the results. Cumulative meta-analyses of studies evaluating the effectiveness of medical interventions have shown that many clinical trials could have been avoided if a systematic review of existing evidence was done prior to conducting a new trial. For example, Lau et al. analyzed 33 clinical trials (involving 36974 patients) evaluating the effectiveness of intravenous streptokinase for acute myocardial infarction. Their cumulative meta-analysis demonstrated that 25 of 33 trials could have been avoided if a systematic review was conducted prior to conducting a new trial. In other words, randomizing 34542 patients was potentially unnecessary.
One study found that only 2 of 25 randomized clinical trials published in five major general medical journals during May 1997 included an updated systematic review in the discussion of results. Subsequent reports showed confirmed this result. These reports also showed that the majority of randomized clinical trials published in five major general medical journals did not present any systematic reviews of existing evidence to justify the research. Robinson et al. analyzed 1523 clinical trials included in 227 meta-analyses and concluded that "less than one quarter of relevant prior studies" were cited. They also confirmed the earlier findings that clinical trial reports do not present systematic review to justify the research or summarize the results.
Metascience has revealed significant problems in psychological research. The field suffers from high bias, low reproducibility, and widespread misuse of statistics. The replication crisis affects psychology more strongly than any other field; as many as two-thirds of highly publicized findings may be impossible to replicate. Meta-research finds that 80-95% of psychological studies support their initial hypotheses, which strongly implies the existence of publication bias.
The replication crisis has led to renewed efforts to re-test important findings. In response to concerns about publication bias and p-hacking, more than 140 psychology journals have adopted result-blind peer review, in which studies are pre-registered and published without regard for their outcome. An analysis of these reforms estimated that 61 percent of result-blind studies produce null results, in contrast with 5 to 20 percent in earlier research. This analysis shows that result-blind peer review substantially reduces publication bias.
Psychologists routinely confuse statistical significance with practical importance, enthusiastically reporting great certainty in unimportant facts. Some psychologists have responded with an increased use of effect size statistics, rather than sole reliance on the p values.
Richard Feynman noted that estimates of physical constants were closer to published values than would be expected by chance. This was believed to be the result of confirmation bias: results that agreed with existing literature were more likely to be believed, and therefore published. Physicists now implement blinding to prevent this kind of bias.
Scientometrics concerns itself with measuring bibliographic data in scientific publications. Major research issues include the measurement of the impact of research papers and academic journals, the understanding of scientific citations, and the use of such measurements in policy and management contexts.
Scientific data scienceEdit
Scientific data science is the use of data science to analyse research papers. It encompasses both qualitative and quantitative methods. Research in scientific data science includes fraud detection and citation network analysis.
- Ioannidis, John P. A.; Fanelli, Daniele; Dunne, Debbie Drake; Goodman, Steven N. (2015-10-02). "Meta-research: Evaluation and Improvement of Research Methods and Practices". PLOS Biology. 13 (10): –1002264. doi:10.1371/journal.pbio.1002264. ISSN 1545-7885. PMC 4592065. PMID 26431313.
- Bach, Author Becky (8 December 2015). "On communicating science and uncertainty: A podcast with John Ioannidis". Scope. Retrieved 20 May 2019.
- Schor, Stanley (1966). "Statistical Evaluation of Medical Journal Manuscripts". JAMA: The Journal of the American Medical Association. 195 (13): 1123. doi:10.1001/jama.1966.03100130097026. ISSN 0098-7484.
- "Researching the researchers". Nature Genetics. 46 (5): 417. 2014. doi:10.1038/ng.2972. ISSN 1061-4036. PMID 24769715.
- Enserink, Martin (2018). "Research on research". Science. 361 (6408): 1178–1179. Bibcode:2018Sci...361.1178E. doi:10.1126/science.361.6408.1178. ISSN 0036-8075. PMID 30237336.
- Rennie, Drummond (1990). "Editorial Peer Review in Biomedical Publication". JAMA. 263 (10): 1317–1441. doi:10.1001/jama.1990.03440100011001. ISSN 0098-7484. PMID 2304208.
- Harriman, Stephanie L.; Kowalczuk, Maria K.; Simera, Iveta; Wager, Elizabeth (2016). "A new forum for research on research integrity and peer review". Research Integrity and Peer Review. 1 (1): 5. doi:10.1186/s41073-016-0010-y. ISSN 2058-8615. PMC 5794038. PMID 29451544.
- Ioannidis, John P. A.; Fanelli, Daniele; Dunne, Debbie Drake; Goodman, Steven N. (2 October 2015). "Meta-research: Evaluation and Improvement of Research Methods and Practices". PLoS Biology. 13 (10). doi:10.1371/journal.pbio.1002264. ISSN 1544-9173. PMID 26431313. Retrieved 11 June 2019.
- Fanelli, Daniele; Costas, Rodrigo; Ioannidis, John P. A. (2017). "Meta-assessment of bias in science". Proceedings of the National Academy of Sciences of the United States of America. 114 (14): 3714–3719. doi:10.1073/pnas.1618569114. ISSN 1091-6490. PMID 28320937. Retrieved 11 June 2019.
- Check Hayden, Erika (2013). "Weak statistical standards implicated in scientific irreproducibility". Nature. doi:10.1038/nature.2013.14131. Retrieved 9 May 2019.
- Schooler, J. W. (2014). "Metascience could rescue the 'replication crisis'". Nature. 515 (7525): 9. Bibcode:2014Natur.515....9S. doi:10.1038/515009a. PMID 25373639.
- Smith, Noah. "Why 'Statistical Significance' Is Often Insignificant". Bloomberg. Retrieved 7 November 2017.
- Pashler, Harold; Wagenmakers, Eric Jan (2012). "Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?". Perspectives on Psychological Science. 7 (6): 528–530. doi:10.1177/1745691612465253. PMID 26168108.
- Ioannidis, JPA (2016). "Why Most Clinical Research Is Not Useful". PLoS Med. 13 (6): e1002049. doi:10.1371/journal.pmed.1002049. PMC 4915619. PMID 27328301.
- Ioannidis JA (13 July 2005). "Contradicted and initially stronger effects in highly cited clinical research". JAMA. 294 (2): 218–228. doi:10.1001/jama.294.2.218. PMID 16014596.
- Chalmers, Iain; Glasziou, Paul (2009). "Avoidable waste in the production and reporting of research evidence". The Lancet. 374 (9683): 86–89. doi:10.1016/S0140-6736(09)60329-9. ISSN 0140-6736. PMID 19525005.
- June 24, Jeremy Hsu; ET, Jeremy Hsu. "Dark Side of Medical Research: Widespread Bias and Omissions". Live Science. Retrieved 24 May 2019.
- "Confronting conflict of interest". Nature Medicine. 24 (11): 1629. November 2018. doi:10.1038/s41591-018-0256-7. ISSN 1546-170X. PMID 30401866.
- Haque, Waqas; Minhajuddin, Abu; Gupta, Arjun; Agrawal, Deepak (2018). "Conflicts of interest of editors of medical journals". PLOS ONE. 13 (5): e0197141. Bibcode:2018PLoSO..1397141H. doi:10.1371/journal.pone.0197141. ISSN 1932-6203. PMC 5959187. PMID 29775468.
- Moncrieff, J (March 2002). "The antidepressant debate". The British Journal of Psychiatry : The Journal of Mental Science. 180 (3): 193–4. doi:10.1192/bjp.180.3.193. ISSN 0007-1250. PMID 11872507. Retrieved 22 May 2019.
- Bello, S; Moustgaard, H; Hróbjartsson, A (October 2014). "The risk of unblinding was infrequently and incompletely reported in 300 randomized clinical trial publications". Journal of Clinical Epidemiology. 67 (10): 1059–69. doi:10.1016/j.jclinepi.2014.05.007. ISSN 1878-5921. PMID 24973822.
- Tuleu, Catherine; Legay, Helene; Orlu-Gul, Mine; Wan, Mandy (1 September 2013). "Blinding in pharmacological trials: the devil is in the details". Archives of Disease in Childhood. 98 (9): 656–659. doi:10.1136/archdischild-2013-304037. ISSN 0003-9888. PMC 3833301. PMID 23898156. Retrieved 8 May 2019.
- Kirsch, I (2014). "Antidepressants and the Placebo Effect". Zeitschrift fur Psychologie. 222 (3): 128–134. doi:10.1027/2151-2604/a000176. ISSN 2190-8370. PMID 25279271.
- Ioannidis, John PA (27 May 2008). "Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials?". Philosophy, Ethics, and Humanities in Medicine : PEHM. 3: 14. doi:10.1186/1747-5341-3-14. ISSN 1747-5341. PMC 2412901. PMID 18505564.
- Moher, David; Altman, Douglas G.; Schulz, Kenneth F. (24 March 2010). "CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials". BMJ. 340: c332. doi:10.1136/bmj.c332. ISSN 0959-8138. PMC 2844940. PMID 20332509. Retrieved 24 April 2019.
- Clarke, Michael; Chalmers, Iain (1998). "Discussion Sections in Reports of Controlled Trials Published in General Medical Journals". JAMA. 280 (3): 280–2. doi:10.1001/jama.280.3.280. PMID 9676682.
- Lau, Joseph; Antman, Elliott M; Jimenez-Silva, Jeanette; Kupelnick, Bruce; Mosteller, Frederick; Chalmers, Thomas C (1992). "Cumulative Meta-Analysis of Therapeutic Trials for Myocardial Infarction". New England Journal of Medicine. 327 (4): 248–54. doi:10.1056/NEJM199207233270406. PMID 1614465.
- Fergusson, Dean; Glass, Kathleen Cranley; Hutton, Brian; Shapiro, Stan (2016). "Randomized controlled trials of aprotinin in cardiac surgery: Could clinical equipoise have stopped the bleeding?". Clinical Trials: Journal of the Society for Clinical Trials. 2 (3): 218–29, discussion 229–32. doi:10.1191/1740774505cn085oa. PMID 16279145.
- Clarke, Mike; Brice, Anne; Chalmers, Iain (2014). "Accumulating Research: A Systematic Account of How Cumulative Meta-Analyses Would Have Provided Knowledge, Improved Health, Reduced Harm and Saved Resources". PLOS ONE. 9 (7): e102670. Bibcode:2014PLoSO...9j2670C. doi:10.1371/journal.pone.0102670. PMC 4113310. PMID 25068257.
- Clarke, Mike; Alderson, P; Chalmers, I (2002). "Discussion Sections in Reports of Controlled Trials Published in General Medical Journals". JAMA. 287 (21): 2799–801. doi:10.1001/jama.287.21.2799. PMID 12038916.
- Clarke, M; Hopewell, S; Chalmers, I (2007). "Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: A status report". Journal of the Royal Society of Medicine. 100 (4): 187–190. doi:10.1258/jrsm.100.4.187. PMC 1847744. PMID 17404342.
- Clarke, Mike; Hopewell, Sally; Chalmers, Iain (2010). "Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting". The Lancet. 376 (9734): 20–1. doi:10.1016/s0140-6736(10)61045-8. PMID 20609983.
- Clarke, Michael; Hopewell, Sally (2013). "Many reports of randomised trials still don't begin or end with a systematic review of the relevant evidence". Journal of the Bahrain Medical Society. 24 (3): 145–148.
- Robinson, Karen A; Goodman, Steven N (2011). "A Systematic Examination of the Citation of Prior Research in Reports of Randomized, Controlled Trials". Annals of Internal Medicine. 154 (1): 50–5. doi:10.7326/0003-4819-154-1-201101040-00007. PMID 21200038.
- Franco, Annie; Malhotra, Neil; Simonovits, Gabor (1 January 2016). "Underreporting in Psychology Experiments: Evidence From a Study Registry". Social Psychological and Personality Science. 7 (1): 8–12. doi:10.1177/1948550615598377. ISSN 1948-5506.
- Munafò, Marcus (29 March 2017). "Metascience: Reproducibility blues". Nature. 543 (7647): 619–620. Bibcode:2017Natur.543..619M. doi:10.1038/543619a. ISSN 1476-4687.
- StokstadSep. 20, Erik (19 September 2018). "This research group seeks to expose weaknesses in science—and they'll step on some toes if they have to". Science | AAAS. Retrieved 24 May 2019.
- Open Science Collaboration (2015). "Estimating the reproducibility of psychological science" (PDF). Science. 349 (6251): aac4716. doi:10.1126/science.aac4716. PMID 26315443.
- Allen, Christopher P G.; Mehler, David Marc Anton. "Open Science challenges, benefits and tips in early career and beyond". doi:10.31234/osf.io/3czyt.
- Simmons, Joseph P.; Nelson, Leif D.; Simonsohn, Uri (2011). "False-Positive Psychology". Psychological Science. 22 (11): 1359–1366. doi:10.1177/0956797611417632. PMID 22006061.
- Stroebe, Wolfgang; Strack, Fritz (2014). "The Alleged Crisis and the Illusion of Exact Replication". Perspectives on Psychological Science. 9 (1): 59–71. doi:10.1177/1745691613514450. PMID 26173241.
- Aschwanden, Christie (6 December 2018). "Psychology's Replication Crisis Has Made The Field Better". FiveThirtyEight. Retrieved 19 December 2018.
- Cohen, Jacob (1994). "The earth is round (p < .05)". American Psychologist. 49 (12): 997–1003. doi:10.1037/0003-066X.49.12.997.
- MacCoun, Robert; Perlmutter, Saul (8 October 2015). "Blind analysis: Hide results to seek the truth". Nature. 526 (7572): 187–189. Bibcode:2015Natur.526..187M. doi:10.1038/526187a. PMID 26450040.
- Leydesdorff, L. and Milojevic, S., "Scientometrics" arXiv:1208.4566 (2013), forthcoming in: Lynch, M. (editor), International Encyclopedia of Social and Behavioral Sciences subsection 85030. (2015)
- Markowitz, David M.; Hancock, Jeffrey T. (2016). "Linguistic obfuscation in fraudulent science". Journal of Language and Social Psychology. 35 (4): 435–445. doi:10.1177/0261927X15614605.
- Ding, Y. (2010). "Applying weighted PageRank to author citation networks". Journal of the American Society for Information Science and Technology. 62 (2): 236–245. arXiv:1102.1760. doi:10.1002/asi.21452.
- Harris, Richard (2017). Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hopes, and Wastes Billions. Basic Books. ISBN 9780465097913.