Mengyao Zhang
1
,
Ling Deng
1
,
Jiaxing Wang
1
,
Bin Zhao
1
,
4
School of Science, Wuhan University of Science and Technology, Wuhan, Hubei, China
4
School of Science, Wuhan University of Science and Technology, Wuhan, Hubei, China
4
School of Science, Wuhan University of Science and Technology, Wuhan, Hubei, China
4
School of Science, Hubei University of Technology, Wuhan, Hubei, China
Abstract
Large-scale innovation contests are becoming increasingly popular globally, with many innovators and enterprises actively participating. Work allocation optimization and fair judging are two significant concerns in innovation contests. This academic paper aims to recommend an optimized "cross- assignment" program by leveraging the optimal objective function method to improve the comparability of scores given by various judges. We conducted data-based descriptive statistical analysis and concluded that the two-stage and weighted evaluation schemes are more beneficial than the traditional judging scheme. Nonetheless, there are still some shortcomings that require addressing. To enhance fairness, we propose an improved two-stage evaluation scheme. In the first stage, we normalize the scores with a normal distribution. In the second stage of the process, we implement a system utilizing the Borda sorting technique to categorize submissions into five distinct groups for judges to evaluate based on their perceptions. We also detail a method for weighting tied scores to determine the final rankings. Testing indicates that this approach yields a Normalized Discounted Cumulative Gain (NDCG) of 0.8667, implying greater fairness and precision in the assessment of submissions.
ย
Keywords
Evaluation program, Normalization of a normal distribution
References
๐Chesbrough H W, Vanhaverbeke W, West J, et al. Open Innovation: Researching a New Paradigm[J]. Wim Vanhaverbeke, 2006, 84(1):1259-1262.
๐Poetz M K, Schreier M. The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas [J]? Journal of Product Innovation Management, 2012, 29(2):245-256.
๐You Qinggen. Research and Implementation of Expert Selection Model for WISCO Research Project Review. Huazhong University of Science and Technology, 2014,32(7):56-60 (in Chinese)
๐Kim Y, Kim Y, Kim J, et al. Boosting on the functional ANOVA decomposition[J]. Statistics & Its Interface, 2009, 2(3):361-368.
๐Kannemann K. The Exact Evaluation of 2-way Cross-classifications: An Algorithmic Solution[J]. Biometrical Journal, 2010,24(2):157-169.
๐Song Yonghao, Shi Xiao, Hu Bin, et al. Team Building Algorithm Based on Multi-Objective Greedy Strategy for Gain Maximization[J]. High Technology Communications, 2018, 28(4): 279-290. (in Chinese)
๐Mielik Inen T, Ukkonen E. The Complexity of Maximum Matroid-Greedoid Intersection and Weighted Greedoid Maximization[J]. Discrete Applied Mathematics, 2006, 154(4):684-691.
๐Jia Tianli. Hypothesis Analysis Model of the Causes of Mean Differences[J]. Mathematics Practice and Understanding, 2005,64(9): 212-215. (in Chinese)
๐Daoji S. An analysis of variance test for the extreme value distribution[J]. Tianjin Daxue Xuebao, 1988, 1988(2):116-121.
๐Shapiro S S, Wilk M B. An analysis of variance test for normality (complete samples)[J]. Biometrika, 1975, 67(3):215-216.
๐Guohong S, Zhenhui Z. Application of Gray Correlation Analysis in Fault Tree Diagnosis[J]. China Safety Science Journal, 1999, 30(2):1505-1507.
๐Liu G, Yu J. Gray correlation analysis and prediction models of living refuse generation in Shanghai city[J]. Waste Management, 2007, 27(3):345-351.
๐Victor G, D. J T M. The Eurovision Song Contest: voting rules, biases and rationality[J]. Journal of Cultural Economics, 2022,47(2):247-277.
๐Looney, Marilyn A. Evaluating Judge Performance in Sport[J]. Journal of Applied Measurement,2004,5(1):31-47.
๐Margaret A. Cliff & Marjorie C. King A proposed approach for evaluating expert wine judge performance using descriptive statistics, Journal of Wine Research, 1996, 7(2), 83-90.
๐Wang Yonglin. Subjective Indicators in Educational Evaluation and Factors Affecting Their Judgment[J]. Education Science, 2017, 33(3): 14-19. (in Chinese)
๐Xu Kai. Exploration and Research on Retrospective Evaluation Selection Method Based on Peer Review Experts[J]. Beijing University of Chinese Medicine, 2017,7(4):47-50 (in Chinese)
๐Ryo I, Kazumasa O. Borda Count Method for Fiscal Policy: A Political Economic Analysis[J]. The Institute of Comparative Economic Studies, Hosei University, 2022, 36(7):25-40.
๐Chen Yuan. Research on Decision-Making Methods for Scientific Fund Project Review and Selection. Northeastern University, 2011,9(5):23-26 (in Chinese)
๐Lee S M, Kim K H, Kim E J. Selection and Classification of Bacterial Strains Using Standardization and Cluster Analysis[J]. Journal of Animal Science and Technology, 2012,32(6):54-56.
๐Zheng Rui. Ranking Risk with Borda Method. Enterprise Reform and Management, 2015,32(11): 154-156. (in Chinese)
๐Orouskhani M, Shi D, Cheng X. A Fuzzy Adaptive Dynamic NSGA-II With Fuzzy-Based Borda Ranking Method and its Application to Multimedia Data Analysis[J]. IEEE Transactions on Fuzzy Systems, 2021,29(8):118-128.
๐Tang Huanling, Wang Jingdong, Lu Yuchang. A New Weight Allocation Strategy to Reduce Similar Topic Classification Errors. Computer Engineering and Applications, 2004,58(13): 185- 188. (in Chinese)
๐Wang Zeyan, Gu Hongfang, Yi Xiaoxin, Zhang Shenru.A Linear Combination Weighting Method Based on Entropy. Systems Engineering Theory and Practice, 2003, 69(3): 112-116. (in Chinese)
๐Kilgour, Marc D, Grรฉgoire , et al. Correction to: Weighted scoring elections: is Borda best?[J]. Social Choice and Welfare,2021,58(2):1-2.
๐Pierrat L, Samuel K. Standardization of the Logistic Distribution based on Entropy[J]. International Journal of Performability Engineering, 2013, 9(3):352-354.
๐Jian W, Qianqian M, Hu-Chen L. A meta-evaluation model on science and technology project review experts using IVIF-BWM and MULTIMOORA[J]. Expert Systems With Applications,2021,36(3),68-70.
๐Ke Nianri, Zhang Chaomo, Wang Yingwei. Application of Normal Distribution Method in Well Logging Curve Standardization in M Oilfield. Journal of Yangtze University (Natural Science Edition), 2010, 7(4): 76-78. (in Chinese)
๐Aroian, Leo A. Handbook of the Normal Distribution[J]. Technometrics, 1996, 25(1):112-115.