Open Access
NAVIGATING ALGORITHMIC EQUITY: UNCOVERING DIVERSITY AND INCLUSION INCIDENTS IN ARTIFICIAL INTELLIGENCE
4
Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST), South Korea
4
Department of Artificial Intelligence, Pohang University of Science and Technology (POSTECH)
Abstract
As artificial intelligence (AI) systems increasingly shape decision-making in critical domains—ranging from healthcare to criminal justice—their societal impact demands careful scrutiny. Despite advancements in algorithmic performance, growing evidence points to systemic issues of bias, exclusion, and inequity embedded within AI models and datasets. This paper offers a comprehensive investigation into documented incidents where AI systems have adversely affected marginalized populations due to a lack of diversity and inclusion considerations. We explore the underlying causes, including biased data, non-representative training sets, and opaque algorithmic design. By analyzing real-world case studies and evaluating mitigation strategies, we assess the effectiveness of existing fairness frameworks and ethical guidelines. Our findings underscore the need for more robust socio-technical interventions, interdisciplinary collaboration, and proactive governance to ensure equitable AI development and deployment. This work contributes to the growing discourse on algorithmic accountability and provides practical recommendations for fostering inclusive and responsible AI systems.
Keywords
Algorithmic Fairness,
Artificial Intelligence Ethics,
Bias in AI
References
2014. Australian Human Rights Commission: A quick guide to Australian discrimination laws. (2014).
Rusul Abduljabbar, Hussein Dia, Sohani Liyanage, and Saeed Asadi Bagloee. 2019. Applications of artificial intelligence in transport: An overview. Sustainability 11, 1 (2019), 189.
Lavisha Aggarwal and Shruti Bhargava. 2023. Fairness in AI Systems: Mitigating gender bias from language-vision models. arXiv preprint arXiv:2305.01888 (2023).
Alba Soriano Arnanz et al. 2023. Creating non-discriminatory Artificial Intelligence systems: balancing the tensions between code granularity and the general nature of legal rules. IDP. Revista de Internet, Derecho y Política 38 (2023), 1–12.
Katie Atkinson, Trevor Bench-Capon, and Danushka Bollegala. 2020. Explanation in AI and law: Past, present and future. Artificial Intelligence 289 (2020), 103387.
Muneera Bano, Didar Zowghi, Fernando Mourao, Sarah Kaur, and Tao Zhang. 2024. Diversity and Inclusion in AI for Recruitment: Lessons from Industry Workshop. arXiv preprint arXiv:2411.06066 (2024).
Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and machine learning: Limitations and opportunities. MIT press.
J. Brodkin. 2023. Black man wrongfully jailed for a week after face recognition error, report says. https://arstechnica.com/tech-policy/2023/01/facial-recognition-error-led-to-wrongful-arrest-of-black-man-report-says/.
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77–91.
Gaelle Cachat-Rosset and Alain Klarsfeld. 2023. Diversity, equity, and inclusion in artificial intelligence: An evaluation of guidelines. Applied Artificial Intelligence 37, 1 (2023), 2176618.
Chiara Cavaglieri. 2022. Tinder’s unfair pricing algorithm exposed. https://www.which.co.uk/news/article/tinders-unfair-pricing-algorithm-exposed-adCwG8b7VRYo.
Nicole Chi, Emma Lurie, and Deirdre K Mulligan. 2021. Reconfiguring diversity and inclusion for AI ethics. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 447–457.
J. Cook. 2018. Amazon ditches AI recruitment tool that ’learnt to be sexist’. https://www.afr.com/world/europe/amazon-ditches-ai-recruitment-tool-that-learnt-to-be-sexist-20181011-h16h8p.
Daniela S Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement. IEEE, 275–284.
Andrew Van Dam. 2019. Searching for images of CEOs or managers? The results almost always show men. https://www.washingtonpost.com/business/2019/01/03/searching-images-ceos-or-managers-results-almost-always-show-men/.
Liya Ding and Aleix M Martinez. 2010. Features versus context: An approach for precise and detailed detection and delineation of faces and facial features. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 11 (2010), 2022–2038.
Sina Fazelpour and Maria De-Arteaga. 2022. Diversity in sociotechnical machine learning systems. Big Data & Society 9, 1 (2022), 20539517221082027.
Michael Feffer, Nikolas Martelaro, and Hoda Heidari. 2023. The AI Incident Database as an Educational Tool to Raise Awareness of AI Harms: A Classroom Exploration of Efficacy, Limitations, & Future Improvements. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1–11.
Xavier Ferrer, Tom Van Nuenen, Jose M Such, Mark Coté, and Natalia Criado. 2021. Bias and discrimination in AI: a cross-disciplinary perspective. IEEE Technology and Society Magazine 40, 2 (2021), 72–80.
Derek Fisk, Ben Clendenning, Philip St. John, and Jose Francois. 2024. Multi-stakeholder validation of entrustable professional activities for a family medicine care of the elderly residency program: A focus group study. Gerontology & Geriatrics Education 45, 1 (2024), 12–25.
Eduard Fosch-Villaronga and Adam Poulsen. 2022. Diversity and inclusion in artificial intelligence. Law and Artificial Intelligence: Regulating AI and Applying AI in Legal Practice (2022), 109–134.
Alex Hern. 2018. Google’s solutions to accidental algorithmic racism: ban gorillas. https://www.theguardian.com/technology/2018/jan/12/google-racism-ban-gorilla-black-people.
Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–16.
J Kavitha, J Sasi Kiran, Srisailapu D Vara Prasad, Krushima Soma, G Charles Babu, and S Sivakumar. 2022. Prediction and Its Impact on Its Attributes While Biasing Machine Learning Training Data. In 2022 Third International Conference on Smart Technologies in Computing, Electrical and Electronics (ICSTCEE). IEEE, 1–7.
Joanna Kerr, Katerina Hilari, and Lia Litosseliti. 2010. Information needs after stroke: What to include and how to structure it on a website. A qualitative study using focus groups and card sorting. Aphasiology 24, 10 (2010), 1170–1196.
Peter A Lichtenberg and Susan E Macneill. 2000. Prospective validity study of a triaging method for mental health problems: The MacNeill-Lichtenberg Decision Tree (MLDT). Clinical gerontologist 21, 1 (2000), 11–19.
Sean McGregor. 2021. Preventing repeated real world AI failures by cataloging incidents: The AI incident database. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 15458–15463.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447–453.
OECD. 2024. Defining AI Incidents and Related Terms. OECD Artificial Intelligence Papers, No. 16, Paris (2024).
Kevin Paeth, Daniel Atherton, Nikiforos Pittaras, Heather Frase, and Sean McGregor. 2024. Lessons for Editors of AI Incidents from the AI Incident Database. arXiv preprint arXiv:2409.16425 (2024).
J. Pepitone. 2024. AI is being built on dated, flawed motion-capture data, IEEE Spectrum. https://spectrum.ieee.org/motion-capture-standards.
Lorenzo Porcaro, Carlos Castillo, Emilia Gómez, and João Vinagre. 2023. Fairness and diversity in information access systems. arXiv preprint arXiv:2305.09319 (2023).
Valentina Pyatkin, Frances Yung, Merel CJ Scholman, Reut Tsarfaty, Ido Dagan, and Vera Demberg. 2023. Design choices for crowdsourcing implicit discourse relations: Revealing the biases introduced by task design. Transactions of the Association for Computational Linguistics 11 (2023), 1014–1032.
Pranav Rajpurkar, Emma Chen, Oishi Banerjee, and Eric J Topol. 2022. AI in health and medicine. Nature medicine 28, 1 (2022), 31–38.
Jeremy Sear. 2001. The ARL ‘Black Box’Flight Recorder–Invention and Memory. Bachelor of Arts (Honours). The University of Melbourne (2001).
Rifat Ara Shams, Didar Zowghi, and Muneera Bano. 2023. AI and the quest for diversity and inclusion: a systematic literature review. AI and Ethics (2023), 1–28.
Rifat Ara Shams, Didar Zowghi, and Muneera Bano. 2024. Diversity and Inclusion (DI)- Related AI Incidents Repository. https://doi.org/10.5281/zenodo.11639709
Donghee Shin and Emily Y Shin. 2023. Data’s Impact on Algorithmic Bias. Computer 56, 6 (2023), 90–94.
Jan Simson, Alessandro Fabris, and Christoph Kern. 2024. Lazy data practices harm fairness research. In The 2024 ACM Conference on Fairness, Accountability, and Transparency. 642–659.
RM Singari and PK Kankar. 2022. Contemporary Evolution of Artificial Intelligence (AI): An Overview and Applications. Advanced Production and Industrial Engineering: Proceedings of ICAPIE 2022 27 (2022), 130.
Roger Søraa. 2023. AI for Diversity. CRC Press.
Donna Spencer and Todd Warfel. 2004. Card sorting: a definitive guide. Boxes and arrows 2, 2004 (2004), 1–23.
Jeff C Stanley and Stephen L Dorton. 2023. Exploring Trust With the AI Incident Database. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 67. SAGE Publications Sage CA: Los Angeles, CA, 489–494.
C. Stokel-Walker. 2023. ChatGPT replicates gender bias in recommendation letters. https://www.scientificamerican.com/article/chatgpt-replicates-gender-bias-in-recommendation-letters/.
Beki Subaeki, Aedah Abd Rahman, Khaerul Manaf, Riffa Haviani Laluma, Agung Wahana, and Nur Lukman. 2022. Assessing Tax Online System Success: A Validation of Success Model with Focus Group Study. In 2022 IEEE 8th International Conference on Computing, Engineering and Design (ICCED). IEEE, 1–646.
Xianghui Tao, Lujia Li, Jianjun He, et al. 2022. Research on discrimination and regulation of artificial intelligence algorithm. In 2nd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2022), Vol. 12348. SPIE, 79–84.
Violet Turri and Rachel Dzombak. 2023. Why We Need to Know More: Exploring the State of AI Incident Documentation Practices. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 576–583.
Antje Von Ungern-Sternberg. 2021. Discriminatory AI and the Law–Legal Standards for Algorithmic Profiling. Draft Chapter, in: Silja Vöneky, Philipp Kellmeyer, Oliver Müller and Wolfram Burgard (ed.) Responsible AI, Cambridge University Press (Forthcoming) (2021).
Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2021. Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI. Computer Law & Security Review 41 (2021), 105567.
Mengyi Wei and Zhixuan Zhou. 2022. Ai ethics issues in real world: Evidence from ai incident database. arXiv preprint arXiv:2206.07635 (2022).
George Williams. 2006. The Victorian charter of human rights and responsibilities: Origins and scope. Melb. UL Rev. 30 (2006), 880.
Xuesong Zhai, Xiaoyan Chu, Ching Sing Chai, Morris Siu Yung Jong, Andreja Istenic, Michael Spector, Jia-Bao Liu, Jing Yuan, and Yan Li. 2021. A Review of Artificial Intelligence (AI) in Education from 2010 to 2020. Complexity 2021 (2021), 1–18.
Jianlong Zhou, Fang Chen, Adam Berry, Mike Reed, Shujia Zhang, and Siobhan Savage. 2020. A survey on ethical principles of AI and implementations. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 3010–3017.
Didar Zowghi and Muneera Bano. 2024. AI for all: Diversity and Inclusion in AI. https://doi.org/10.1007/s43681-024-00485-8. , 4 pages.
Didar Zowghi and Francesca da Rimini. 2023. Diversity and Inclusion in Artificial Intelligence. arXiv preprint arXiv:2305.12728 (2023).
Frederik J Zuiderveen Borgesius. 2020. Strengthening legal protection against discrimination by algorithms and artificial intelligence. The International Journal of Human Rights 24, 10 (2020), 1572–1593.
Most read articles by the same author(s)
- Sashank Siwakoti, Bhaskar Chaganti, Human-in-the-Loop Control Planes for Cortex Agents: Policy-Driven Escalation, Approval, and Evidence Capture , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Mohammed Imran Choudhary, AI-Augmented Network-Forensics: Leveraging LLMs for Real-Time Threat Detection and Automated Response in Enterprise Environments , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Priya Sharma, A Data-Centric Approach to Transforming Digital Retail Through Artificial Intelligence-Based Shopping Systems , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Suprajyotsna Dasari , Automated Testing Techniques for Enterprise Software Systems with GenAI Integration , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Sonam Kumari, Enhancing Clinical Decision-Making Using Generative AI-Powered Knowledge Retrieval Systems: A Review of Emerging Approaches and Challenges , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 08 (2026): Volume 03 Issue 08
- Myroslav Mishov, Autonomous Threat Remediation in Localized AI Environments: A Review of Security-as-Code Execution Models , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 06 (2026): Volume 03 Issue 06
- Nimal Perera, Anjali Fernando, Robust Browser Fingerprinting Under Adversarial Conditions: An AI-Driven Detection and Defense Architecture , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Dr. Eleni Markou, Narrative Intelligence In The Age Of Generative Ai: Integrating Computational Storytelling, Transformer Architectures, Ethical Governance, And Consumer Impact , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 03 (2026): Volume 03 Issue 03
- Nguyen Minh Anh, Tran Quoc Bao, Unsupervised Learning Framework for Country Clustering Based on Agricultural Import Patterns , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Dr. Leila K. Moreno, Integrated Real-Time Fraud Detection and Response: A Streaming Analytics Framework for Financial Transaction Security , International Journal of Advanced Artificial Intelligence Research: Vol. 2 No. 11 (2025): Volume 02 Issue 11
Similar Articles
- Dr. Mei-Ling Zhou, Dr. Haojie Xu, LEARNING RICH FEATURES WITHOUT LABELS: CONTRASTIVE APPROACHES IN MULTIMODAL ARTIFICIAL INTELLIGENCE SYSTEMS , International Journal of Advanced Artificial Intelligence Research: Vol. 2 No. 04 (2025): Volume 02 Issue 04
- Marko Petrovic, Intelligent Failure Prediction Techniques for Modern Electricity Distribution Infrastructure , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 03 (2026): Volume 03 Issue 03
- Dwi Jatmiko, Huu Nguyen, AI-Guided Policy Learning For Hyperdimensional Sampling: Exploiting Expert Human Demonstrations From Interactive Virtual Reality Molecular Dynamics , International Journal of Advanced Artificial Intelligence Research: Vol. 2 No. 10 (2025): Volume 02 Issue 10
- Dr Chintal Kumar Patel, Survey of Artificial Intelligence Approaches for Traffic Accident Analysis, Prediction, And Prevention , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 07 (2026): Volume 03 Issue 07
- Angelo soriano, Sheila Ann Mercado, The Convergence of AI And UVM: Advanced Methodologies for the Verification of Complex Low-Power Semiconductor Architectures , International Journal of Advanced Artificial Intelligence Research: Vol. 2 No. 11 (2025): Volume 02 Issue 11
- Mr. Raman Kumar, An Analysis of Explainable Artificial Intelligence for Intelligent Cybersecurity Applications , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 08 (2026): Volume 03 Issue 08
- Rizky Pratama, Dinda Maharani, Computational Representation and Structural Enhancement of Nature-Derived Collective Monitoring Behaviors , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 04 (2026): Volume 03 Issue 04
- Nguyen Minh Anh, Tran Quoc Bao, Unsupervised Learning Framework for Country Clustering Based on Agricultural Import Patterns , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Kwame Asare, AI-Powered Predictive Exception Management and Supply Network Coordination in SAP S/4HANA Manufacturing , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 10 (2026): Volume 03 Issue 10
- Dr. Erion Hoxha, Dr. Elira Dervishi, Global Firefly Optimization Model for IoT Attack Detection , International Journal of Advanced Artificial Intelligence Research: Vol. 3 No. 08 (2026): Volume 03 Issue 08
You may also start an advanced similarity search for this article.