Quantitative Evaluation Model in Large Scale Innovation Competition Based on Statistical Theory
Abstract
Large-scale innovation contests are becoming increasingly popular globally, with many innovators and enterprises actively participating. Work allocation optimization and fair judging are two significant concerns in innovation contests. This academic paper aims to recommend an optimized "cross- assignment" program by leveraging the optimal objective function method to improve the comparability of scores given by various judges. We conducted data-based descriptive statistical analysis and concluded that the two-stage and weighted evaluation schemes are more beneficial than the traditional judging scheme. Nonetheless, there are still some shortcomings that require addressing. To enhance fairness, we propose an improved two-stage evaluation scheme. In the first stage, we normalize the scores with a normal distribution. In the second stage of the process, we implement a system utilizing the Borda sorting technique to categorize submissions into five distinct groups for judges to evaluate based on their perceptions. We also detail a method for weighting tied scores to determine the final rankings. Testing indicates that this approach yields a Normalized Discounted Cumulative Gain (NDCG) of 0.8667, implying greater fairness and precision in the assessment of submissions.
Keywords
Evaluation program, Normalization of a normal distributionHow to Cite
Downloads
References
Chesbrough H W, Vanhaverbeke W, West J, et al. Open Innovation: Researching a New Paradigm[J]. Wim Vanhaverbeke, 2006, 84(1):1259-1262.
Poetz M K, Schreier M. The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas [J]? Journal of Product Innovation Management, 2012, 29(2):245-256.
You Qinggen. Research and Implementation of Expert Selection Model for WISCO Research Project Review. Huazhong University of Science and Technology, 2014,32(7):56-60 (in Chinese)
Kim Y, Kim Y, Kim J, et al. Boosting on the functional ANOVA decomposition[J]. Statistics & Its Interface, 2009, 2(3):361-368.
Kannemann K. The Exact Evaluation of 2-way Cross-classifications: An Algorithmic Solution[J]. Biometrical Journal, 2010,24(2):157-169.
Song Yonghao, Shi Xiao, Hu Bin, et al. Team Building Algorithm Based on Multi-Objective Greedy Strategy for Gain Maximization[J]. High Technology Communications, 2018, 28(4): 279-290. (in Chinese)
Mielik Inen T, Ukkonen E. The Complexity of Maximum Matroid-Greedoid Intersection and Weighted Greedoid Maximization[J]. Discrete Applied Mathematics, 2006, 154(4):684-691.
Jia Tianli. Hypothesis Analysis Model of the Causes of Mean Differences[J]. Mathematics Practice and Understanding, 2005,64(9): 212-215. (in Chinese)
Daoji S. An analysis of variance test for the extreme value distribution[J]. Tianjin Daxue Xuebao, 1988, 1988(2):116-121.
Shapiro S S, Wilk M B. An analysis of variance test for normality (complete samples)[J]. Biometrika, 1975, 67(3):215-216.
Guohong S, Zhenhui Z. Application of Gray Correlation Analysis in Fault Tree Diagnosis[J]. China Safety Science Journal, 1999, 30(2):1505-1507.
Liu G, Yu J. Gray correlation analysis and prediction models of living refuse generation in Shanghai city[J]. Waste Management, 2007, 27(3):345-351.
Victor G, D. J T M. The Eurovision Song Contest: voting rules, biases and rationality[J]. Journal of Cultural Economics, 2022,47(2):247-277.
Looney, Marilyn A. Evaluating Judge Performance in Sport[J]. Journal of Applied Measurement,2004,5(1):31-47.
Margaret A. Cliff & Marjorie C. King A proposed approach for evaluating expert wine judge performance using descriptive statistics, Journal of Wine Research, 1996, 7(2), 83-90.
Wang Yonglin. Subjective Indicators in Educational Evaluation and Factors Affecting Their Judgment[J]. Education Science, 2017, 33(3): 14-19. (in Chinese)
Xu Kai. Exploration and Research on Retrospective Evaluation Selection Method Based on Peer Review Experts[J]. Beijing University of Chinese Medicine, 2017,7(4):47-50 (in Chinese)
Ryo I, Kazumasa O. Borda Count Method for Fiscal Policy: A Political Economic Analysis[J]. The Institute of Comparative Economic Studies, Hosei University, 2022, 36(7):25-40.
Chen Yuan. Research on Decision-Making Methods for Scientific Fund Project Review and Selection. Northeastern University, 2011,9(5):23-26 (in Chinese)
Lee S M, Kim K H, Kim E J. Selection and Classification of Bacterial Strains Using Standardization and Cluster Analysis[J]. Journal of Animal Science and Technology, 2012,32(6):54-56.
Zheng Rui. Ranking Risk with Borda Method. Enterprise Reform and Management, 2015,32(11): 154-156. (in Chinese)
Orouskhani M, Shi D, Cheng X. A Fuzzy Adaptive Dynamic NSGA-II With Fuzzy-Based Borda Ranking Method and its Application to Multimedia Data Analysis[J]. IEEE Transactions on Fuzzy Systems, 2021,29(8):118-128.
Tang Huanling, Wang Jingdong, Lu Yuchang. A New Weight Allocation Strategy to Reduce Similar Topic Classification Errors. Computer Engineering and Applications, 2004,58(13): 185- 188. (in Chinese)
Wang Zeyan, Gu Hongfang, Yi Xiaoxin, Zhang Shenru.A Linear Combination Weighting Method Based on Entropy. Systems Engineering Theory and Practice, 2003, 69(3): 112-116. (in Chinese)
Kilgour, Marc D, Grégoire , et al. Correction to: Weighted scoring elections: is Borda best?[J]. Social Choice and Welfare,2021,58(2):1-2.
Pierrat L, Samuel K. Standardization of the Logistic Distribution based on Entropy[J]. International Journal of Performability Engineering, 2013, 9(3):352-354.
Jian W, Qianqian M, Hu-Chen L. A meta-evaluation model on science and technology project review experts using IVIF-BWM and MULTIMOORA[J]. Expert Systems With Applications,2021,36(3),68-70.
Ke Nianri, Zhang Chaomo, Wang Yingwei. Application of Normal Distribution Method in Well Logging Curve Standardization in M Oilfield. Journal of Yangtze University (Natural Science Edition), 2010, 7(4): 76-78. (in Chinese)
Aroian, Leo A. Handbook of the Normal Distribution[J]. Technometrics, 1996, 25(1):112-115.
Taylor M. ABSTRACT SoftRank: Optimising Non-Smooth Rank Metrics[J]. Inproceedings, 2008(Volume 7):77-86.
Copyright (c) 2024 Mengyao Zhang, Ling Deng, Jiaxing Wang, Bin Zhao (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain the copyright of their manuscripts, and all Open Access articles are disseminated under the terms of the Creative Commons Attribution License 4.0 (CC-BY), which licenses unrestricted use, distribution, and reproduction in any medium, provided that the original work is appropriately cited. The use of general descriptive names, trade names, trademarks, and so forth in this publication, even if not specifically identified, does not imply that these names are not protected by the relevant laws and regulations.
