Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer; Felix Holzmeister; Colin F. Camerer; Anna Dreber; Juergen Huber; Magnus Johannesson; Michael Kirchler; Roni Iwanir; Jeanette A. Mumford; R. Alison Adcock; Paolo Avesani; Blazej M. Baczkowski; Aahana Bajracharya; Leah Bakst; Sheryl Ball; Marco Barilari; Nadège Bault; Derek Beaton; Julia Beitner; Roland G. Benoit; Ruud M.W.J. Berkers; Jamil P. Bhanji; Bharat B. Biswal; Sebastian Bobadilla-Suarez; Tiago Bortolini; Katherine L. Bottenhorn; Alexander Bowring; Senne Braem; Hayley R. Brooks; Emily G. Brudner; Cristian B. Calderon; Julia A. Camilleri; Jaime J. Castrellon; Luca Cecchetti; Edna C. Cieslik; Zachary J. Cole; Olivier Collignon; Robert W. Cox; William A. Cunningham; Stefan Czoschke; Kamalaker Dadi; Charles P. Davis; Alberto De Luca; Mauricio R. Delgado; Lysia Demetriou; Jeffrey B. Dennison; Xin Di; Erin W. Dickie; Ekaterina Dobryakova; Claire L. Donnat; Juergen Dukart; Niall W. Duncan; Joke Durnez; Amr Eed; Simon B. Eickhoff; Andrew Erhart; Laura Fontanesi; G. Matthew Fricke; Shiguang Fu; Adriana Galván; Remi Gau; Sarah Genon; Tristan Glatard; Enrico Glerean; Jelle J. Goeman; Sergej A.E. Golowin; Carlos González-García; Krzysztof J. Gorgolewski; Cheryl L. Grady; Mikella A. Green; João F. Guassi Moreira; Olivia Guest; Shabnam Hakimi; J. Paul Hamilton; Roeland Hancock; Giacomo Handjaras; Bronson B. Harry; Colin Hawco; Peer Herholz; Gabrielle Herman; Stephan Heunis; Felix Hoffstaedter; Jeremy Hogeveen; Susan Holmes; Chuan Peng Hu; Scott A. Huettel; Matthew E. Hughes; Vittorio Iacovella; Alexandru D. Iordan; Peder M. Isager; Ayse I. Isik; Andrew Jahn; Matthew R. Johnson; Tom Johnstone; Michael J.E. Joseph; Anthony C. Juliano; Joseph W. Kable; Michalis Kassinopoulos; Cemal Koba; Xiang Zhen Kong; Timothy R. Koscik; Nuri Erkut Kucukboyaci; Brice A. Kuhl; Sebastian Kupek; Angela R. Laird; Claus Lamm; Robert Langner; Nina Lauharatanahirun; Hongmi Lee; Sangil Lee; Alexander Leemans; Andrea Leo; Elise Lesage; Flora Li; Monica Y.C. Li; Phui Cheng Lim; Evan N. Lintz; Schuyler W. Liphardt; Annabel B. Losecaat Vermeer; Bradley C. Love; Michael L. Mack; Norberto Malpica; Theo Marins; Camille Maumet; Kelsey McDonald; Joseph T. McGuire; Helena Melero; Adriana S. Méndez Leal; Benjamin Meyer; Kristin N. Meyer; Glad Mihai; Georgios D. Mitsis; Jorge Moll; Dylan M. Nielson; Gustav Nilsonne; Michael P. Notter; Emanuele Olivetti; Adrian I. Onicas; Paolo Papale; Kaustubh R. Patil; Jonathan E. Peelle; Alexandre Pérez; Doris Pischedda; Jean Baptiste Poline; Yanina Prystauka; Shruti Ray; Patricia A. Reuter-Lorenz; Richard C. Reynolds; Emiliano Ricciardi; Jenny R. Rieck; Anais M. Rodriguez-Thompson; Anthony Romyn; Taylor Salo; Gregory R. Samanez-Larkin; Emilio Sanz-Morales; Margaret L. Schlichting; Douglas H. Schultz; Qiang Shen; Margaret A. Sheridan; Jennifer A. Silvers; Kenny Skagerlund; Alec Smith; David V. Smith; Peter Sokol-Hessner; Simon R. Steinkamp; Sarah M. Tashjian; Bertrand Thirion; John N. Thorp; Gustav Tinghög; Loreen Tisdall; Steven H. Tompson; Claudio Toro-Serey; Juan Jesus Torre Tresols; Leonardo Tozzi; Vuong Truong; Luca Turella; Anna E. van ‘t Veer; Tom Verguts; Jean M. Vettel; Sagana Vijayarajah; Khoi Vo; Matthew B. Wall; Wouter D. Weeda; Susanne Weis; David J. White; David Wisniewski; Alba Xifra-Porxas; Emily A. Yearling; Sangsuk Yoon; Rui Yuan; Kenneth S.L. Yuen; Lei Zhang; Xu Zhang; Joshua E. Zosky; Thomas E. Nichols; Russell A. Poldrack; Tom Schonberg

doi:https://doi.org/10.1038/s41586-020-2314-9

Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, R. Alison Adcock, Paolo Avesani, Blazej M. Baczkowski, Aahana Bajracharya, Leah Bakst, Sheryl Ball, Marco Barilari, Nadège Bault, Derek Beaton, Julia Beitner, Roland G. BenoitRuud M.W.J. Berkers, Jamil P. Bhanji, Bharat B. Biswal, Sebastian Bobadilla-Suarez, Tiago Bortolini, Katherine L. Bottenhorn, Alexander Bowring, Senne Braem, Hayley R. Brooks, Emily G. Brudner, Cristian B. Calderon, Julia A. Camilleri, Jaime J. Castrellon, Luca Cecchetti, Edna C. Cieslik, Zachary J. Cole, Olivier Collignon, Robert W. Cox, William A. Cunningham, Stefan Czoschke, Kamalaker Dadi, Charles P. Davis, Alberto De Luca, Mauricio R. Delgado, Lysia Demetriou, Jeffrey B. Dennison, Xin Di, Erin W. Dickie, Ekaterina Dobryakova, Claire L. Donnat, Juergen Dukart, Niall W. Duncan, Joke Durnez, Amr Eed, Simon B. Eickhoff, Andrew Erhart, Laura Fontanesi, G. Matthew Fricke, Shiguang Fu, Adriana Galván, Remi Gau, Sarah Genon, Tristan Glatard, Enrico Glerean, Jelle J. Goeman, Sergej A.E. Golowin, Carlos González-García, Krzysztof J. Gorgolewski, Cheryl L. Grady, Mikella A. Green, João F. Guassi Moreira, Olivia Guest, Shabnam Hakimi, J. Paul Hamilton, Roeland Hancock, Giacomo Handjaras, Bronson B. Harry, Colin Hawco, Peer Herholz, Gabrielle Herman, Stephan Heunis, Felix Hoffstaedter, Jeremy Hogeveen, Susan Holmes, Chuan Peng Hu, Scott A. Huettel, Matthew E. Hughes, Vittorio Iacovella, Alexandru D. Iordan, Peder M. Isager, Ayse I. Isik, Andrew Jahn, Matthew R. Johnson, Tom Johnstone, Michael J.E. Joseph, Anthony C. Juliano, Joseph W. Kable, Michalis Kassinopoulos, Cemal Koba, Xiang Zhen Kong, Timothy R. Koscik, Nuri Erkut Kucukboyaci, Brice A. Kuhl, Sebastian Kupek, Angela R. Laird, Claus Lamm, Robert Langner, Nina Lauharatanahirun, Hongmi Lee, Sangil Lee, Alexander Leemans, Andrea Leo, Elise Lesage, Flora Li, Monica Y.C. Li, Phui Cheng Lim, Evan N. Lintz, Schuyler W. Liphardt, Annabel B. Losecaat Vermeer, Bradley C. Love, Michael L. Mack, Norberto Malpica, Theo Marins, Camille Maumet, Kelsey McDonald, Joseph T. McGuire, Helena Melero, Adriana S. Méndez Leal, Benjamin Meyer, Kristin N. Meyer, Glad Mihai, Georgios D. Mitsis, Jorge Moll, Dylan M. Nielson, Gustav Nilsonne, Michael P. Notter, Emanuele Olivetti, Adrian I. Onicas, Paolo Papale, Kaustubh R. Patil, Jonathan E. Peelle, Alexandre Pérez, Doris Pischedda, Jean Baptiste Poline, Yanina Prystauka, Shruti Ray, Patricia A. Reuter-Lorenz, Richard C. Reynolds, Emiliano Ricciardi, Jenny R. Rieck, Anais M. Rodriguez-Thompson, Anthony Romyn, Taylor Salo, Gregory R. Samanez-Larkin, Emilio Sanz-Morales, Margaret L. Schlichting, Douglas H. Schultz, Qiang Shen, Margaret A. Sheridan, Jennifer A. Silvers, Kenny Skagerlund, Alec Smith, David V. Smith, Peter Sokol-Hessner, Simon R. Steinkamp, Sarah M. Tashjian, Bertrand Thirion, John N. Thorp, Gustav Tinghög, Loreen Tisdall, Steven H. Tompson, Claudio Toro-Serey, Juan Jesus Torre Tresols, Leonardo Tozzi, Vuong Truong, Luca Turella, Anna E. van ‘t Veer, Tom Verguts, Jean M. Vettel, Sagana Vijayarajah, Khoi Vo, Matthew B. Wall, Wouter D. Weeda, Susanne Weis, David J. White, David Wisniewski, Alba Xifra-Porxas, Emily A. Yearling, Sangsuk Yoon, Rui Yuan, Kenneth S.L. Yuen, Lei Zhang, Xu Zhang, Joshua E. Zosky, Thomas E. Nichols, Russell A. Poldrack, Tom Schonberg

Tel Aviv University

Research output: Contribution to journal › Article › peer-review

Abstract

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses¹. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset^2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Original language	English
Pages (from-to)	84-88
Number of pages	5
Journal	Nature
Volume	582
Issue number	7810
DOIs	https://doi.org/10.1038/s41586-020-2314-9
State	Published - 4 Jun 2020

All Science Journal Classification (ASJC) codes

General

Access to Document

https://doi.org/10.1038/s41586-020-2314-9

Cite this

Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., Kirchler, M., Iwanir, R., Mumford, J. A., Adcock, R. A., Avesani, P., Baczkowski, B. M., Bajracharya, A., Bakst, L., Ball, S., Barilari, M., Bault, N., Beaton, D., Beitner, J., ... Schonberg, T. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582(7810), 84-88. https://doi.org/10.1038/s41586-020-2314-9

@article{9709b83a868b4a77b0e9aad9421e806d,

title = "Variability in the analysis of a single neuroimaging dataset by many teams",

abstract = "Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.",

author = "Rotem Botvinik-Nezer and Felix Holzmeister and Camerer, {Colin F.} and Anna Dreber and Juergen Huber and Magnus Johannesson and Michael Kirchler and Roni Iwanir and Mumford, {Jeanette A.} and Adcock, {R. Alison} and Paolo Avesani and Baczkowski, {Blazej M.} and Aahana Bajracharya and Leah Bakst and Sheryl Ball and Marco Barilari and Nad{\`e}ge Bault and Derek Beaton and Julia Beitner and Benoit, {Roland G.} and Berkers, {Ruud M.W.J.} and Bhanji, {Jamil P.} and Biswal, {Bharat B.} and Sebastian Bobadilla-Suarez and Tiago Bortolini and Bottenhorn, {Katherine L.} and Alexander Bowring and Senne Braem and Brooks, {Hayley R.} and Brudner, {Emily G.} and Calderon, {Cristian B.} and Camilleri, {Julia A.} and Castrellon, {Jaime J.} and Luca Cecchetti and Cieslik, {Edna C.} and Cole, {Zachary J.} and Olivier Collignon and Cox, {Robert W.} and Cunningham, {William A.} and Stefan Czoschke and Kamalaker Dadi and Davis, {Charles P.} and Luca, {Alberto De} and Delgado, {Mauricio R.} and Lysia Demetriou and Dennison, {Jeffrey B.} and Xin Di and Dickie, {Erin W.} and Ekaterina Dobryakova and Donnat, {Claire L.} and Juergen Dukart and Duncan, {Niall W.} and Joke Durnez and Amr Eed and Eickhoff, {Simon B.} and Andrew Erhart and Laura Fontanesi and Fricke, {G. Matthew} and Shiguang Fu and Adriana Galv{\'a}n and Remi Gau and Sarah Genon and Tristan Glatard and Enrico Glerean and Goeman, {Jelle J.} and Golowin, {Sergej A.E.} and Carlos Gonz{\'a}lez-Garc{\'i}a and Gorgolewski, {Krzysztof J.} and Grady, {Cheryl L.} and Green, {Mikella A.} and {Guassi Moreira}, {Jo{\~a}o F.} and Olivia Guest and Shabnam Hakimi and Hamilton, {J. Paul} and Roeland Hancock and Giacomo Handjaras and Harry, {Bronson B.} and Colin Hawco and Peer Herholz and Gabrielle Herman and Stephan Heunis and Felix Hoffstaedter and Jeremy Hogeveen and Susan Holmes and Hu, {Chuan Peng} and Huettel, {Scott A.} and Hughes, {Matthew E.} and Vittorio Iacovella and Iordan, {Alexandru D.} and Isager, {Peder M.} and Isik, {Ayse I.} and Andrew Jahn and Johnson, {Matthew R.} and Tom Johnstone and Joseph, {Michael J.E.} and Juliano, {Anthony C.} and Kable, {Joseph W.} and Michalis Kassinopoulos and Cemal Koba and Kong, {Xiang Zhen} and Koscik, {Timothy R.} and Kucukboyaci, {Nuri Erkut} and Kuhl, {Brice A.} and Sebastian Kupek and Laird, {Angela R.} and Claus Lamm and Robert Langner and Nina Lauharatanahirun and Hongmi Lee and Sangil Lee and Alexander Leemans and Andrea Leo and Elise Lesage and Flora Li and Li, {Monica Y.C.} and Lim, {Phui Cheng} and Lintz, {Evan N.} and Liphardt, {Schuyler W.} and {Losecaat Vermeer}, {Annabel B.} and Love, {Bradley C.} and Mack, {Michael L.} and Norberto Malpica and Theo Marins and Camille Maumet and Kelsey McDonald and McGuire, {Joseph T.} and Helena Melero and {M{\'e}ndez Leal}, {Adriana S.} and Benjamin Meyer and Meyer, {Kristin N.} and Glad Mihai and Mitsis, {Georgios D.} and Jorge Moll and Nielson, {Dylan M.} and Gustav Nilsonne and Notter, {Michael P.} and Emanuele Olivetti and Onicas, {Adrian I.} and Paolo Papale and Patil, {Kaustubh R.} and Peelle, {Jonathan E.} and Alexandre P{\'e}rez and Doris Pischedda and Poline, {Jean Baptiste} and Yanina Prystauka and Shruti Ray and Reuter-Lorenz, {Patricia A.} and Reynolds, {Richard C.} and Emiliano Ricciardi and Rieck, {Jenny R.} and Rodriguez-Thompson, {Anais M.} and Anthony Romyn and Taylor Salo and Samanez-Larkin, {Gregory R.} and Emilio Sanz-Morales and Schlichting, {Margaret L.} and Schultz, {Douglas H.} and Qiang Shen and Sheridan, {Margaret A.} and Silvers, {Jennifer A.} and Kenny Skagerlund and Alec Smith and Smith, {David V.} and Peter Sokol-Hessner and Steinkamp, {Simon R.} and Tashjian, {Sarah M.} and Bertrand Thirion and Thorp, {John N.} and Gustav Tingh{\"o}g and Loreen Tisdall and Tompson, {Steven H.} and Claudio Toro-Serey and {Torre Tresols}, {Juan Jesus} and Leonardo Tozzi and Vuong Truong and Luca Turella and {van {\textquoteleft}t Veer}, {Anna E.} and Tom Verguts and Vettel, {Jean M.} and Sagana Vijayarajah and Khoi Vo and Wall, {Matthew B.} and Weeda, {Wouter D.} and Susanne Weis and White, {David J.} and David Wisniewski and Alba Xifra-Porxas and Yearling, {Emily A.} and Sangsuk Yoon and Rui Yuan and Yuen, {Kenneth S.L.} and Lei Zhang and Xu Zhang and Zosky, {Joshua E.} and Nichols, {Thomas E.} and Poldrack, {Russell A.} and Tom Schonberg",

note = "Publisher Copyright: {\textcopyright} 2020, The Author(s), under exclusive licence to Springer Nature Limited.",

year = "2020",

month = jun,

day = "4",

doi = "https://doi.org/10.1038/s41586-020-2314-9",

language = "الإنجليزيّة",

volume = "582",

pages = "84--88",

journal = "Nature",

issn = "0028-0836",

publisher = "Nature Research",

number = "7810",

}

Botvinik-Nezer, R, Holzmeister, F, Camerer, CF, Dreber, A, Huber, J, Johannesson, M, Kirchler, M, Iwanir, R, Mumford, JA, Adcock, RA, Avesani, P, Baczkowski, BM, Bajracharya, A, Bakst, L, Ball, S, Barilari, M, Bault, N, Beaton, D, Beitner, J, Benoit, RG, Berkers, RMWJ, Bhanji, JP, Biswal, BB, Bobadilla-Suarez, S, Bortolini, T, Bottenhorn, KL, Bowring, A, Braem, S, Brooks, HR, Brudner, EG, Calderon, CB, Camilleri, JA, Castrellon, JJ, Cecchetti, L, Cieslik, EC, Cole, ZJ, Collignon, O, Cox, RW, Cunningham, WA, Czoschke, S, Dadi, K, Davis, CP, Luca, AD, Delgado, MR, Demetriou, L, Dennison, JB, Di, X, Dickie, EW, Dobryakova, E, Donnat, CL, Dukart, J, Duncan, NW, Durnez, J, Eed, A, Eickhoff, SB, Erhart, A, Fontanesi, L, Fricke, GM, Fu, S, Galván, A, Gau, R, Genon, S, Glatard, T, Glerean, E, Goeman, JJ, Golowin, SAE, González-García, C, Gorgolewski, KJ, Grady, CL, Green, MA, Guassi Moreira, JF, Guest, O, Hakimi, S, Hamilton, JP, Hancock, R, Handjaras, G, Harry, BB, Hawco, C, Herholz, P, Herman, G, Heunis, S, Hoffstaedter, F, Hogeveen, J, Holmes, S, Hu, CP, Huettel, SA, Hughes, ME, Iacovella, V, Iordan, AD, Isager, PM, Isik, AI, Jahn, A, Johnson, MR, Johnstone, T, Joseph, MJE, Juliano, AC, Kable, JW, Kassinopoulos, M, Koba, C, Kong, XZ, Koscik, TR, Kucukboyaci, NE, Kuhl, BA, Kupek, S, Laird, AR, Lamm, C, Langner, R, Lauharatanahirun, N, Lee, H, Lee, S, Leemans, A, Leo, A, Lesage, E, Li, F, Li, MYC, Lim, PC, Lintz, EN, Liphardt, SW, Losecaat Vermeer, AB, Love, BC, Mack, ML, Malpica, N, Marins, T, Maumet, C, McDonald, K, McGuire, JT, Melero, H, Méndez Leal, AS, Meyer, B, Meyer, KN, Mihai, G, Mitsis, GD, Moll, J, Nielson, DM, Nilsonne, G, Notter, MP, Olivetti, E, Onicas, AI, Papale, P, Patil, KR, Peelle, JE, Pérez, A, Pischedda, D, Poline, JB, Prystauka, Y, Ray, S, Reuter-Lorenz, PA, Reynolds, RC, Ricciardi, E, Rieck, JR, Rodriguez-Thompson, AM, Romyn, A, Salo, T, Samanez-Larkin, GR, Sanz-Morales, E, Schlichting, ML, Schultz, DH, Shen, Q, Sheridan, MA, Silvers, JA, Skagerlund, K, Smith, A, Smith, DV, Sokol-Hessner, P, Steinkamp, SR, Tashjian, SM, Thirion, B, Thorp, JN, Tinghög, G, Tisdall, L, Tompson, SH, Toro-Serey, C, Torre Tresols, JJ, Tozzi, L, Truong, V, Turella, L, van ‘t Veer, AE, Verguts, T, Vettel, JM, Vijayarajah, S, Vo, K, Wall, MB, Weeda, WD, Weis, S, White, DJ, Wisniewski, D, Xifra-Porxas, A, Yearling, EA, Yoon, S, Yuan, R, Yuen, KSL, Zhang, L, Zhang, X, Zosky, JE, Nichols, TE, Poldrack, RA & Schonberg, T 2020, 'Variability in the analysis of a single neuroimaging dataset by many teams', Nature, vol. 582, no. 7810, pp. 84-88. https://doi.org/10.1038/s41586-020-2314-9

TY - JOUR

T1 - Variability in the analysis of a single neuroimaging dataset by many teams

AU - Botvinik-Nezer, Rotem

AU - Holzmeister, Felix

AU - Camerer, Colin F.

AU - Dreber, Anna

AU - Huber, Juergen

AU - Johannesson, Magnus

AU - Kirchler, Michael

AU - Iwanir, Roni

AU - Mumford, Jeanette A.

AU - Adcock, R. Alison

AU - Avesani, Paolo

AU - Baczkowski, Blazej M.

AU - Bajracharya, Aahana

AU - Bakst, Leah

AU - Ball, Sheryl

AU - Barilari, Marco

AU - Bault, Nadège

AU - Beaton, Derek

AU - Beitner, Julia

AU - Benoit, Roland G.

AU - Berkers, Ruud M.W.J.

AU - Bhanji, Jamil P.

AU - Biswal, Bharat B.

AU - Bobadilla-Suarez, Sebastian

AU - Bortolini, Tiago

AU - Bottenhorn, Katherine L.

AU - Bowring, Alexander

AU - Braem, Senne

AU - Brooks, Hayley R.

AU - Brudner, Emily G.

AU - Calderon, Cristian B.

AU - Camilleri, Julia A.

AU - Castrellon, Jaime J.

AU - Cecchetti, Luca

AU - Cieslik, Edna C.

AU - Cole, Zachary J.

AU - Collignon, Olivier

AU - Cox, Robert W.

AU - Cunningham, William A.

AU - Czoschke, Stefan

AU - Dadi, Kamalaker

AU - Davis, Charles P.

AU - Luca, Alberto De

AU - Delgado, Mauricio R.

AU - Demetriou, Lysia

AU - Dennison, Jeffrey B.

AU - Di, Xin

AU - Dickie, Erin W.

AU - Dobryakova, Ekaterina

AU - Donnat, Claire L.

AU - Dukart, Juergen

AU - Duncan, Niall W.

AU - Durnez, Joke

AU - Eed, Amr

AU - Eickhoff, Simon B.

AU - Erhart, Andrew

AU - Fontanesi, Laura

AU - Fricke, G. Matthew

AU - Fu, Shiguang

AU - Galván, Adriana

AU - Gau, Remi

AU - Genon, Sarah

AU - Glatard, Tristan

AU - Glerean, Enrico

AU - Goeman, Jelle J.

AU - Golowin, Sergej A.E.

AU - González-García, Carlos

AU - Gorgolewski, Krzysztof J.

AU - Grady, Cheryl L.

AU - Green, Mikella A.

AU - Guassi Moreira, João F.

AU - Guest, Olivia

AU - Hakimi, Shabnam

AU - Hamilton, J. Paul

AU - Hancock, Roeland

AU - Handjaras, Giacomo

AU - Harry, Bronson B.

AU - Hawco, Colin

AU - Herholz, Peer

AU - Herman, Gabrielle

AU - Heunis, Stephan

AU - Hoffstaedter, Felix

AU - Hogeveen, Jeremy

AU - Holmes, Susan

AU - Hu, Chuan Peng

AU - Huettel, Scott A.

AU - Hughes, Matthew E.

AU - Iacovella, Vittorio

AU - Iordan, Alexandru D.

AU - Isager, Peder M.

AU - Isik, Ayse I.

AU - Jahn, Andrew

AU - Johnson, Matthew R.

AU - Johnstone, Tom

AU - Joseph, Michael J.E.

AU - Juliano, Anthony C.

AU - Kable, Joseph W.

AU - Kassinopoulos, Michalis

AU - Koba, Cemal

AU - Kong, Xiang Zhen

AU - Koscik, Timothy R.

AU - Kucukboyaci, Nuri Erkut

AU - Kuhl, Brice A.

AU - Kupek, Sebastian

AU - Laird, Angela R.

AU - Lamm, Claus

AU - Langner, Robert

AU - Lauharatanahirun, Nina

AU - Lee, Hongmi

AU - Lee, Sangil

AU - Leemans, Alexander

AU - Leo, Andrea

AU - Lesage, Elise

AU - Li, Flora

AU - Li, Monica Y.C.

AU - Lim, Phui Cheng

AU - Lintz, Evan N.

AU - Liphardt, Schuyler W.

AU - Losecaat Vermeer, Annabel B.

AU - Love, Bradley C.

AU - Mack, Michael L.

AU - Malpica, Norberto

AU - Marins, Theo

AU - Maumet, Camille

AU - McDonald, Kelsey

AU - McGuire, Joseph T.

AU - Melero, Helena

AU - Méndez Leal, Adriana S.

AU - Meyer, Benjamin

AU - Meyer, Kristin N.

AU - Mihai, Glad

AU - Mitsis, Georgios D.

AU - Moll, Jorge

AU - Nielson, Dylan M.

AU - Nilsonne, Gustav

AU - Notter, Michael P.

AU - Olivetti, Emanuele

AU - Onicas, Adrian I.

AU - Papale, Paolo

AU - Patil, Kaustubh R.

AU - Peelle, Jonathan E.

AU - Pérez, Alexandre

AU - Pischedda, Doris

AU - Poline, Jean Baptiste

AU - Prystauka, Yanina

AU - Ray, Shruti

AU - Reuter-Lorenz, Patricia A.

AU - Reynolds, Richard C.

AU - Ricciardi, Emiliano

AU - Rieck, Jenny R.

AU - Rodriguez-Thompson, Anais M.

AU - Romyn, Anthony

AU - Salo, Taylor

AU - Samanez-Larkin, Gregory R.

AU - Sanz-Morales, Emilio

AU - Schlichting, Margaret L.

AU - Schultz, Douglas H.

AU - Shen, Qiang

AU - Sheridan, Margaret A.

AU - Silvers, Jennifer A.

AU - Skagerlund, Kenny

AU - Smith, Alec

AU - Smith, David V.

AU - Sokol-Hessner, Peter

AU - Steinkamp, Simon R.

AU - Tashjian, Sarah M.

AU - Thirion, Bertrand

AU - Thorp, John N.

AU - Tinghög, Gustav

AU - Tisdall, Loreen

AU - Tompson, Steven H.

AU - Toro-Serey, Claudio

AU - Torre Tresols, Juan Jesus

AU - Tozzi, Leonardo

AU - Truong, Vuong

AU - Turella, Luca

AU - van ‘t Veer, Anna E.

AU - Verguts, Tom

AU - Vettel, Jean M.

AU - Vijayarajah, Sagana

AU - Vo, Khoi

AU - Wall, Matthew B.

AU - Weeda, Wouter D.

AU - Weis, Susanne

AU - White, David J.

AU - Wisniewski, David

AU - Xifra-Porxas, Alba

AU - Yearling, Emily A.

AU - Yoon, Sangsuk

AU - Yuan, Rui

AU - Yuen, Kenneth S.L.

AU - Zhang, Lei

AU - Zhang, Xu

AU - Zosky, Joshua E.

AU - Nichols, Thomas E.

AU - Poldrack, Russell A.

AU - Schonberg, Tom

PY - 2020/6/4

Y1 - 2020/6/4

N2 - Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

AB - Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2–5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

UR - http://www.scopus.com/inward/record.url?scp=85085279867&partnerID=8YFLogxK

U2 - https://doi.org/10.1038/s41586-020-2314-9

DO - https://doi.org/10.1038/s41586-020-2314-9

M3 - مقالة

C2 - 32483374

SN - 0028-0836

VL - 582

SP - 84

EP - 88

JO - Nature

JF - Nature

IS - 7810

ER -

Variability in the analysis of a single neuroimaging dataset by many teams

Abstract

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Cite this