Please wait a minute...
Data Analysis and Knowledge Discovery  2021, Vol. 5 Issue (11): 1-12    DOI: 10.11925/infotech.2096-3467.2021.0515
Current Issue | Archive | Adv Search |
Review of Application and Evolution of Meta-Analysis in Social Sciences
Li Xiao1,2,Qu Jiansheng2,3()
1Northwest Institute of Eco-Environment and Resources, Chinese Academy of Sciences, Lanzhou 730000, China
2University of Chinese Academy of Sciences, Beijing 100049, China
3Chengdu Library and Information Center, Chinese Academy of Sciences, Chengdu 610041, China
Download: PDF (991 KB)   HTML ( 32 ) 
Export: BibTeX | EndNote (RIS)      
Abstract  

[Objective] This paper reviews the latest application and evolution of meta-analysis in social sciences. [Methods] First, we summarized the main characteristics of meta-analysis as well as the key problems facing the application of meta-analysis in social sciences. Then, we conducted case studies with the MetaBUS and CoDa databases. Finally, we exmined the meta-analysis from different perspectives. [Results] The meta-analysis in social sciences mainly studied the aggregated data meta with traditional methods, such as standardized mean differences and correlation coefficients. At present, the key issues are effect size deviation, lack of transparency and quality assessment, as well as time and manpower consuming, etc. Meta-analytics database and meta-analytic research can benefit each other. Data repository, open science movement and artificial intelligence technology all posed various significant impacts on meta-analytical research. [Limitations] The content analysis is mainly based on sampling samples, so there are potential limitations in comprehensively revealing the characteristics and problems of meta-analysis. [Conclusions] There are still many problems to be addressed for meta-analysis in social sciences, and all parties need to work together to improve these research and draw better conclusions.
Key words： Meta-Analysis      Social Science      Main Features      Key Problems      Meta-Analysis Database     
Received: 24 May 2021      Published: 23 December 2021
ZTFLH:  G350  
Fund:National Key Research and Development Project of China(2018YFC1509007)
Corresponding Authors: Qu Jiansheng,ORCID：0000-0002-2806-3447     E-mail: jsqu@lzb.ac.cn
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Xiao Li
Jiansheng Qu

Cite this article:

Li Xiao, Qu Jiansheng. Review of Application and Evolution of Meta-Analysis in Social Sciences. Data Analysis and Knowledge Discovery, 2021, 5(11): 1-12.

URL:

https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/10.11925/infotech.2096-3467.2021.0515     OR     https://manu44.magtech.com.cn/Jwk_infotech_wk3/EN/Y2021/V5/I11/1

Top 10 Disciplines Based on Meta-Analysis of WoS in Social Science
Number of Meta-Analysis by Year in Social Science
名称 作者或机构 适用学科 主要贡献
PRISMA声明（系统评价和元分析的首选报告）[10] PRISMA小组（2009） 最初专注于医学领域随机试验,后来被各个领域广泛采用 提供了系统评价或元分析报告要包括的27个项目清单以及描述文献检索过程及结果的流程图
MARS （元分析报告标准）[11] 美国心理协会
（APA 2008,2010）		 心理学相关学科以及其他社会科学学科 所提供的元分析研究报告格式包括标题、摘要、引言、方法、结果、讨论,方法部分包括样本纳入与排除标准、调节变量和中介变量分析、检索方式、编码程序、统计方法
Campbell协作网系统评价方法[12] Campbell协作网方法小组 犯罪与司法、教育、国际发展以及社会福利等领域 为Campbell协作网干预效果系统评价的实施提供了详细的方法指南。其所列大部分方法指南适用于所有综述主题,但有些可能并不完全适用于非干预综述
Standards or Guidelines Used for Meta-analysis in Social Science
Data Extraction by Two Types of Meta-Analysis
Manual Literature Screening Process
级别 元数据
文献级别信息 期刊名称、卷号、发行号、起始页和结束页、出版年份,以及财政支持情况（如拨款资助：是或否）
变量级别信息 报告的变量名、样本量、确切的样本量、分类节点、分类节点名、概念反转、变量均值、标准差、可靠性值、可靠性值是否为系数α（是/否）、时间点、回复率、样本编号、数据来源和数据对象、分析单位、国家、编码置信度
Main Data for MetaBUS Database
One Annotated Instance in CoDa’s Model
Some Meta-Analysis Extracting Data from Data Repositories
[1] Glass G V. Primary, Secondary, and Meta-Analysis of Research[J]. Educational Researcher, 1976, 5(10):3-8.
[2] Higgins J P T, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions[M]. The 2nd Edition. Chichester, UK: John Wiley & Sons, 2019.
[3] Koricheva J, Gurevitch J, Mengersen K. Handbook of Meta-Analysis in Ecology and Evolution[M]. Princeton, New Jersey: Princeton University Press, 2013.
[4] Coope H. Research Synjournal and Meta-Analysis a Step-by-Step Approach[J]. Journal of Chemical Information and Modeling, 2016, 53(9):1689-1699.
doi: 10.1021/ci400128m
[5] Cooper H, Hedges L V, Valentine J C. The Handbook of Research Synbook and Meta-Analysis[M]. New York: Russell Sage Foundation, 2019.
[6] Lipsey M W, Wilson D B. Practical Meta-Analysis[M]. London: Sage Publications, 2001.
[7] Borenstein M, Hedges L V, Higgins J P T, et al. Introduction to Meta-Analysis[M]. Chichester, UK: John Wiley & Sons, 2009.
[8] Hedges L V, Olkin I. Statistical Methods for Meta-Analysis[M]. Academic Press, 1985.
[9] Hunter J E, Schmidt F L. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings[M]. SAGE Publications, Inc, 1990.
[10] Moher D, Liberati A, Tetzlaff J, et al. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement[J]. PLoS Medicine, 2009, 6(7):e1000097.
doi: 10.1371/journal.pmed.1000097
[11] APA Publications and Communications Board Working Group on Journal Article Reporting Standards. Reporting Standards for Research in Psychology: Why do We Need Them? What Might They Be?[J]. The American Psychologist, 2008, 63(9):839-851.
doi: 10.1037/0003-066X.63.9.839
[12] The Methods Group of the Campbell Collaboration. Methodological Expectations of Campbell Collaboration Intervention Reviews: Conduct Standards[S]. The Campbell Collaboration, 2016.
[13] Sterne J A, Egger M, Moher D. Addressing Reporting Biases[A]//Cochrane Handbook for Systematic Reviews of Interventions[M]. Chichester, UK: John Wiley & Sons, 2008: 297-333.
[14] Kalter J, Verdonck-de Leeuw I M, Sweegers M G, et al. Effects and Moderators of Psychosocial Interventions on Quality of Life, and Emotional and Social Function in Patients with Cancer: An Individual Patient Data Meta-Analysis of 22 RCTS[J]. Psycho-Oncology, 2018, 27(4):1150-1161.
doi: 10.1002/pon.4648 pmid: 29361206
[15] van Wijk R, Jansen J J P, Lyles M A. Inter- and Intra-Organizational Knowledge Transfer: A Meta-Analytic Review and Assessment of Its Antecedents and Consequences[J]. Journal of Management Studies, 2008, 45(4):830-853.
doi: 10.1111/j.1467-6486.2008.00771.x
[16] Cohen J. Statistical Power Analysis for the Behavioral Sciences[M]. The 2nd Edition. Hillsdale, NJ: Lawrence Erlbaum Associates Publishers, 1988.
[17] Hedges L V. Unbiased Estimation of Effect Size[J]. Evaluation in Education, 1980, 4:25-27.
doi: 10.1016/0191-765X(80)90005-1
[18] Bergh D D, Aguinis H, Heavey C, et al. Using Meta-Analytic Structural Equation Modeling to Advance Strategic Management Research: Guidelines and an Empirical Illustration via the Strategic Leadership-Performance Relationship[J]. Strategic Management Journal, 2016, 37(3):477-497.
doi: 10.1002/smj.2016.37.issue-3
[19] Simmons J P, Nelson L D, Simonsohn U. False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant[J]. Psychological Science, 2011, 22(11):1359-1366.
doi: 10.1177/0956797611417632 pmid: 22006061
[20] Motulsky H J. Common Misconceptions about Data Analysis and Statistics[J]. The Journal of Pharmacology and Experimental Therapeutics, 2014, 351(1):200-205.
doi: 10.1124/jpet.114.219170
[21] Polanin J R, Tanner-Smith E E, Hennessy E A. Estimati. the Difference Between Published and Unpublished Effect Sizes[J]. Review of Educational Research, 2016, 86(1):207-236.
doi: 10.3102/0034654315582067
[22] Gage N A, Cook B G, Reichow B. Publication Bias in Special Education Meta-Analyses[J]. Exceptional Children, 2017, 83(4):428-445.
doi: 10.1177/0014402917691016
[23] Friese M, Frankenbach J. p-Hacking and Publication Bias Interact to Distort Meta-Analytic Effect Size Estimates[J]. Psychological Methods, 2020, 25(4):456-471.
doi: 10.1037/met0000246 pmid: 31789538
[24] Egger M, Smith G D, Schneider M, et al. Bias in Meta-Analysis Detected by a Simple, Graphical Test[J]. BMJ, 1997, 315(7109):629-634.
pmid: 9310563
[25] Orwin R G. A Fail-Safe N for Effect Size in Meta-Analysis[J]. Journal of Educational Statistics, 1983, 8(2):157.
[26] Duval S, Tweedie R. Trim and Fill: A Simple Funnel-Plot-Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis[J]. Biometrics, 2000, 56(2):455-463.
pmid: 10877304
[27] Simonsohn U, Nelson L D, Simmons J P. p-Curve and Effect Size[J]. Perspectives on Psychological Science, 2014, 9(6):666-681.
doi: 10.1177/1745691614553988 pmid: 26186117
[28] van Assen M A, van Aert R C, Wicherts J M. Meta-Analysis Using Effect Size Distributions of Only Statistically Significant Studies[J]. Psychological Methods, 2015, 20(3):293-309.
doi: 10.1037/met0000025
[29] Hedges L V. Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences[J]. Journal of Educational Statistics, 1984, 9(1):61-85.
doi: 10.3102/10769986009001061
[30] Carter E C, Schönbrodt F D, Gervais W M, et al. Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods[J]. Advances in Methods and Practices in Psychological Science, 2019, 2(2):115-144.
doi: 10.1177/2515245919847196
[31] McShane B B, Böckenholt U, Hansen K T. Adjusting for Publication Bias in Meta-Analysis[J]. Perspectives on Psychological Science, 2016, 11(5):730-749.
doi: 10.1177/1745691616662243
[32] Becker B J. Multivariate Meta-Analysis[A]//Handbook of Applied Multivariate Statistics and Mathematical Modeling[M]. Amsterdam: Elsevier, 2000: 499-525.
[33] Hedges L V. Stochastically Dependent Effect Sizes[A]//The Handbook of Research Synbook and Meta-Analysis[M]. New York: Russell Sage Foundation, 2019: 281-298.
[34] Mengersen K, Jennions M D, Schmid C H. Statistical Models for the Meta-Analysis of Nonindependent Data[A]//Handbook of Meta-Analysis in Ecology and Evolution[M]. Princeton, New Jersey: Princeton University Press, 2013: 255-283.
[35] Lajeunesse M J. Meta-Analysis and the Comparative Phylogenetic Method[J]. The American Naturalist, 2009, 174(3):369-381.
doi: 10.1086/603628 pmid: 19637963
[36] Noble D W A, Lagisz M, O’Dea R E, et al. Nonindependence and Sensitivity Analyses in Ecological and Evolutionary Meta-Analyses[J]. Molecular Ecology, 2017, 26(9):2410-2425.
doi: 10.1111/mec.2017.26.issue-9
[37] Rodgers M A, Pustejovsky J E. Evaluating Meta-Analytic Methods to Detect Selective Reporting in the Presence of Dependent Effect Sizes[J]. Psychological Methods, 2021, 26(2):141-160.
doi: 10.1037/met0000300
[38] Kalaian H A, Raudenbush S W. A Multivariate Mixed Linear Model for Meta-Analysis[J]. Psychological Methods, 1996, 1(3):227-235.
doi: 10.1037/1082-989X.1.3.227
[39] van den Noortgate W, López-López J A, Marín-Martínez F, et al. Three-Level Meta-Analysis of Dependent Effect Sizes[J]. Behavior Research Methods, 2013, 45(2):576-594.
doi: 10.3758/s13428-012-0261-6 pmid: 23055166
[40] Hedges L V, Tipton E, Johnson M C. Robust Variance Estimation in Meta-Regression with Dependent Effect Size Estimates[J]. Research Synjournal Methods, 2010, 1(1):39-65.
[41] Banks G C, Field J G, Oswald F L, et al. Answers to 18 Questions About Open Science Practices[J]. Journal of Business and Psychology, 2019, 34(3):257-270.
doi: 10.1007/s10869-018-9547-8
[42] O’Boyle E H, Banks G C, Gonzalez-Mulé E. The Chrysalis Effect: How Ugly Initial Results Metamorphosize into Beautiful Articles[J]. Journal of Management, 2017, 43(2):376-399.
doi: 10.1177/0149206314527133
[43] Schwab A, Starbuck W H. A Call for Openness in Research Reporting: How to Turn Covert Practices into Helpful Tools[J]. Academy of Management Learning & Education, 2017, 16(1):125-141.
[44] Christensen G, Miguel E. Transparency, Reproducibility, and the Credibility of Economics Research[J]. Journal of Economic Literature, 2018, 56(3):920-980.
doi: 10.1257/jel.20171350
[45] Polanin J R, Hennessy E A, Tsuji S. Transparency and Reproducibility of Meta-Analyses in Psychology: A Meta-Review[J]. Perspectives on Psychological Science, 2020, 15(4):1026-1041.
doi: 10.1177/1745691620906416 pmid: 32516081
[46] Moreau D, Gamble B. Conducting a Meta-Analysis in the Age of Open Science: Tools, Tips, and Practical Recommendations[J]. Psychological Methods, 2020. DOI: 10.1037/met0000351.
doi: 10.1037/met0000351
[47] van de Schoot R, de Bruin J, Schram R, et al. An Open Source Machine Learning Framework for Efficient and Transparent Systematic Reviews[J]. Nature Machine Intelligence, 2021, 3(2):125-133.
doi: 10.1038/s42256-020-00287-7
[48] Harrison H, Griffin S J, Kuhn I, et al. Software Tools to Support Title and Abstract Screening for Systematic Reviews in Healthcare: An Evaluation[J]. BMC Medical Research Methodology, 2020, 20(1):7.
doi: 10.1186/s12874-020-0897-3 pmid: 31931747
[49] Coyne J C, Hagedoorn M, Thombs B. Most Published and Unpublished Dissertations Should be Excluded from Meta-Analyses: Comment on Moyer et al[J]. Psycho-Oncology, 2011, 20(2):224-225.
doi: 10.1002/pon.1788
[50] Ioannidis J P A. The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-Analyses[J]. The Milbank Quarterly, 2016, 94(3):485-514.
[51] Sutton A J, Abrams K R, Jones D R, et al. Methods for Meta-Analysis in Medical Research[M]. Chichester: Wiley, 2000.
[52] Roberts P D, Stewart G B, Pullin A S. Are Review Articles a Reliable Source of Evidence to Support Conservation and Environmental Management? A Comparison with Medicine[J]. Biological Conservation, 2006, 132(4):409-423.
doi: 10.1016/j.biocon.2006.04.034
[53] Higgins J P T, Lane P W, Anagnostelis B, et al. A Tool to Assess the Quality of a Meta-Analysis[J]. Research Synjournal Methods, 2013, 4(4):351-366.
[54] Philibert A, Loyce C, Makowski D. Assessment of the Quality of Meta-Analysis in Agronomy[J]. Agriculture, Ecosystems & Environment, 2012, 148:72-82.
doi: 10.1016/j.agee.2011.12.003
[55] Bosco F A, Uggerslev K L, Steel P. MetaBUS as a Vehicle for Facilitating Meta-Analysis[J]. Human Resource Management Review, 2017, 27(1):237-254.
doi: 10.1016/j.hrmr.2016.09.013
[56] Bosco F A, Uggerslev K L. MetaBUS[EB/OL]. [2021-05-10]. http://metabus.org/.
[57] Bosco F A, Aguinis H, Singh K, et al. Correlational Effect Size Benchmarks[J]. Journal of Applied Psychology, 2015, 100(2):431-449.
doi: 10.1037/a0038047
[58] Singh K, Bosco F, Field J G. Mapping I-O Psychology: Content and Trends from 1980 to 2009[C]// Proceedings of Society for Industrial and Organizational Psychology Conference. 2014.
[59] Chamberlin M, Newton D W, Lepine J A. A Meta-Analysis of Voice and Its Promotive and Prohibitive Forms: Identification of Key Associations, Distinctions, and Future Research Directions[J]. Personnel Psychology, 2017, 70(1):11-71.
doi: 10.1111/peps.2017.70.issue-1
[60] Rudolph C W, Kooij D T A M, Rauvola R S, et al. Occupational Future Time Perspective: A Meta-Analysis of Antecedents and Outcomes[J]. Journal of Organizational Behavior, 2018, 39(2):229-248.
doi: 10.1002/job.2264
[61] Schmidt J A, Pohler D M. Making Stronger Causal Inferences: Accounting for Selection Bias in Associations Between High Performance Work Systems, Leadership, and Employee and Customer Satisfaction[J]. The Journal of Applied Psychology, 2018, 103(9):1001-1018.
doi: 10.1037/apl0000315
[62] Spadaro G, Tiddi I, Columbus S, et al. The Cooperation Databank[EB/OL]. [2021-05-10]. https://app.cooperationdatabank.org/.
[63] Spadaro G, Tiddi I, Columbus S, et al. The Cooperation Databank[OL]. https://doi.org/10.31234/osf.io/rveh3.
[64] Wilkinson M D, Dumontier M, Aalbersberg I J, et al. The FAIR Guiding Principles for Scientific Data Management and Stewardship[J]. Scientific Data, 2016, 3:160018.
doi: 10.1038/sdata.2016.18 pmid: 26978244
[65] Centre for Open Science. The Centre for Open Science 2018[EB/OL]. [2021-05-10]. https://www.cos.io/.
[66] Kraker P, Leony D, Reinhardt W, et al. The Case for an Open Science in Technology Enhanced Learning[J]. International Journal of Technology Enhanced Learning, 2011, 3(6):643-654.
doi: 10.1504/IJTEL.2011.045454
[1] Wu Shengnan, Pu Hongjun, Tian Ruonan, Liang Wenqi, Yu Qi. Network Structure’s Impacts on Link Prediction Algorithm from Meta-Analysis Perspective[J]. 数据分析与知识发现, 2021, 5(11): 102-113.
[2] Hong Pan,Li Tang. Qualitative Data Analysis in Chinese Social Science Studies——The Case of Nvivo[J]. 数据分析与知识发现, 2020, 4(1): 51-62.
[3] Liu Hongxu,Qu Jiansheng. Using Meta-analysis Software for Domain Knowledge Discovery[J]. 现代图书情报技术, 2016, 32(5): 9-21.
[4] Yang Bo,Hu Liyun. Metadata Standard for Social Science Information Organization——DDI[J]. 现代图书情报技术, 2005, 21(8): 7-11.
[5] Ma Yu. The Organization and Deploitation of Chinese Social Science Academic Network Resource[J]. 现代图书情报技术, 2001, 17(3): 43-45.
  Copyright © 2016 Data Analysis and Knowledge Discovery   Tel/Fax:(010)82626611-6626，82624938   E-mail:jishu@mail.las.ac.cn