Open Access
An Analytical Study of a Governed Lakehouse Framework in Multi-State Medicare Claims and Provider Data for Fraud, Waste, and Abuse Detection
Chandra Obul Reddy Nagi Reddy
1
,
,
4
Organization: Gainwell Technologies, Designation: Advisor Database Architect
Abstract
Medicare fraud detection, waste, and abuse is hampered by the high volume of claims, highly fragmented data-analysis systems, limited visibility into data quality, and significant class imbalance. This study assessed the potential of an Original Medicare Part B (OMB) Bronze-Silver-Gold lakehouse to improve data quality, traceability, analytical efficiency, and provider-level risk identification across OMB-covered populations in California, New York, Florida, Texas, and Pennsylvania in 2024. Using Apache Spark/PySpark with Delta-format tables, the study sampled 1,500,000 provider-service records in proportion to state Traditional Medicare enrollment, representing 214,873 unique NPIs, consolidated them with CMS provider files and HHS-OIG provider-exclusion data, interpreted 30 features, and used four models: Logistic Regression, Random Forest, XGBoost, and Isolation Forest. During Silver-layer screening, 27,362 records (1.82%) triggered one or more quality rules and were quarantined; NPI enrichment had 99.7% compliance, and Gold-layer data quality had 99.2% compliance. This reduced query time from a median of 18.4 seconds to 5.7 seconds (approximately 69%). XGBoost performed best among the supervised models, with ROC-AUC = 0.91, PR-AUC = 0.34, and F1 = 0.66. State elevated/high-risk flags were not confirmed as fraud; they were a priority for investigation and ranged from 2.2%–3.4% of unique providers within each state. Manually managed, repeatable pipelines provide defensible Medicare FWA prioritization, and repeatable, source-to-feature, auditable pipelines can support multi-state, analytic workflows where needed. Areas to explore include longitudinal claims, Part D, DMEPOS, enforcement outcomes, graph analytics, and prospective investigator validation.
Keywords
Medicare fraud,
waste and abuse,
Governed data lakehouse,
Medicare Part B claims
References
S. A. Maluleke, “Optometrists’ experience of healthcare fraud, waste and abuse (FWA) and strategies to combat it,” African Vision and Eye Health, vol. 83, no. 1, pp. 1–10, 2024. https://www.scielo.org.za/scielo.php?pid=S2410-15162024000100032&script=sci_arttext
Takefuji, “Generative AI for analysis and identification of Medicare improper payments by provider type and HCPC code,” Exploratory Research in Clinical and Social Pharmacy, vol. 12, Art. no. 100387, 2023. https://www.sciencedirect.com/science/article/pii/S2667276623001683
A. Teymourifar, “Technological advancements and the use of key performance indicators in hospital management: A critical-conceptual review,” Frontiers in Health Services, vol. 6, Art. no. 1780458, 2026. https://www.frontiersin.org/journals/health-services/articles/10.3389/frhs.2026.1780458/pdf
Centers for Medicare & Medicaid Services, “Medicare Physician & Other Practitioners - by Provider and Service,” CMS Data, 2024 data. https://data.cms.gov/provider-summary-by-type-of-service/medicare-physician-other-practitioners/medicare-physician-other-practitioners-by-provider-and-service
M. Suesserman, S. Gorny, D. Lasaga, J. Helms, D. Olson, E. Bowen, and S. Bhattacharya, “Procedure code overutilization detection from healthcare claims using unsupervised deep learning methods,” BMC Medical Informatics and Decision Making, vol. 23, no. 1, Art. no. 196, 2023. https://link.springer.com/content/pdf/10.1186/s12911-023-02268-3.pdf
A. du Preez, S. Bhattacharya, P. Beling, and E. Bowen, “Fraud detection in healthcare claims using machine learning: A systematic review,” Artificial Intelligence in Medicine, vol. 160, Art. no. 103061, 2025. https://www.sciencedirect.com/science/article/pii/S0933365724003038
A. A. Harby and F. H. Zulkernine, “Data Lakehouse: A survey and experimental study,” Information Systems, vol. 127, Art. no. 102460, 2025, doi: 10.1016/j.is.2024.102460. https://doi.org/10.1016/j.is.2024.102460
Sardana, R. Hariharan, and R. S. Pudi, “AI-driven secure and resilient healthcare data platforms with intelligent search and personalized communication,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–6. https://ieeexplore.ieee.org/document/11374370
A. Fannouch, J. Gharib, and Y. Gahi, “A governed Lakehouse DataOps architecture: Design and evaluation in healthcare,” HighTech and Innovation Journal, vol. 7, no. 2, pp. 704–731, Jun. 2026, doi: 10.28991/HIJ-2026-07-02-021. https://doi.org/10.28991/HIJ-2026-07-02-021
N. M. K. Koneru, “AI-driven optimization of CI/CD pipelines for multi-cloud infrastructure deployment,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–6. https://ieeexplore.ieee.org/abstract/document/11374441/
Centers for Medicare & Medicaid Services, “Medicare Physician & Other Practitioner Look-up Tool,” CMS Data, 2024 data. https://data.cms.gov/tools/medicare-physician-other-practitioner-look-up-tool
Centers for Medicare & Medicaid Services, “Medicare monthly enrollment,” CMS Data, 2024. https://data.cms.gov/summary-statistics-on-beneficiary-enrollment/medicare-and-medicaid-reports/medicare-monthly-enrollment
C. Bonthu, G. Malik, and M. R. Dhanagari, “AI-driven secure and scalable data architecture with multi-domain MDM and cybersecurity for real-time FinTech decision-making,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374416
T. Knezevic Ivanovski, S. Honap, R. Matic, S. Markovic, and L. Peyrin-Biroulet, “Building a healthcare data warehouse: Considerations, opportunities, and challenges,” Frontiers in Digital Health, vol. 7, Art. no. 1691142, 2025. https://www.frontiersin.org/journals/digital-health/articles/10.3389/fdgth.2025.1691142/pdf
M. Schlegel and K.-U. Sattler, “Capturing end-to-end provenance for machine learning pipelines,” Information Systems, vol. 132, Art. no. 102495, 2025. https://www.sciencedirect.com/science/article/pii/S0306437924001534
Z. Hamid, F. Khalique, S. Mahmood, A. Daud, A. Bukhari, and B. Alshemaimri, “Healthcare insurance fraud detection using data mining,” BMC Medical Informatics and Decision Making, vol. 24, no. 1, Art. no. 112, 2024. https://link.springer.com/content/pdf/10.1186/s12911-024-02512-4.pdf
S. Sepuri, “Comparative analysis of artificial intelligence test generation methods for compliance validation in regulated enterprise workflows,” Journal of Intelligent Decision Making and Information Science, vol. 3, no. 2s, pp. 107–123, 2026. https://doi.org/10.59543/jidmis.v3.632
J. Debener, V. Heinke, and J. Kriebel, “Detecting insurance fraud using supervised and unsupervised machine learning,” Journal of Risk and Insurance, vol. 90, no. 3, pp. 743–768, 2023. https://onlinelibrary.wiley.com/doi/pdf/10.1111/jori.12427
A. Cejudo, Y. Tellechea, A. Calvo, A. Almeida, C. Martín, and A. Beristain, “Scalable big data platform with end-to-end traceability for health data monitoring in older adults: Development and performance evaluation,” JMIR Medical Informatics, vol. 13, Art. no. e81701, 2025. https://medinform.jmir.org/2025/1/e81701
C. Flynn, T. Murphy, J. Walsh, and D. Riordan, “An open industrial energy dataset with asset-level measurements and high-coverage 15-minute aggregates from a manufacturing facility,” Data, vol. 11, no. 5, Art. no. 101, 2026. https://www.mdpi.com/2306-5729/11/5/101
L. L. Motta, W. S. Alvarez, H. R. Carvajal, F. B. Neto, E. R. de Lima, and L. G. P. Meloni, “A cloud architecture for home energy management systems: A conceptual model,” Energy Informatics, vol. 8, no. 1, Art. no. 142, 2025. https://link.springer.com/content/pdf/10.1186/s42162-025-00599-1.pdf
S. Rangu, K. S. Chadha, and S. Samala, “AI-powered cloud architecture, agile analytics, and enterprise DevOps-enabled framework for healthcare and insurance ecosystem optimisation,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374414
S. Scheider and M. K. Mallick, “Exploring metadata catalogs in health care data ecosystems: Taxonomy development study,” JMIR Formative Research, vol. 9, Art. no. e63396, 2025. https://formative.jmir.org/2025/1/e63396
N. Cheimarios, “Scientific software development in the AI era: Reproducibility, MLOps, and applications in soft matter physics,” Frontiers in Physics, vol. 13, Art. no. 1711356, 2025. https://www.frontiersin.org/journals/physics/articles/10.3389/fphy.2025.1711356/pdf
M. Armbrust et al., “Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores,” Proceedings of the VLDB Endowment, vol. 13, no. 12, pp. 3411–3424, 2020, doi: 10.14778/3415478.3415560. https://doi.org/10.14778/3415478.3415560
H. De Meulemeester, F. De Smet, J. van Dorst, E. Derroitte, and B. De Moor, “Explainable unsupervised anomaly detection for healthcare insurance data,” BMC Medical Informatics and Decision Making, vol. 25, no. 1, Art. no. 14, 2025. https://link.springer.com/content/pdf/10.1186/s12911-024-02823-6.pdf
D. J. Hekman et al., “Dashboarding to monitor machine-learning-based clinical decision support interventions,” Applied Clinical Informatics, vol. 15, no. 1, pp. 164–169, 2024. https://www.thieme-connect.com/products/ejournals/pdf/10.1055/a-2219-5175.pdf
J. M. Johnson and T. M. Khoshgoftaar, “Data-centric AI for healthcare fraud detection,” SN Computer Science, vol. 4, no. 4, Art. no. 389, 2023. https://pmc.ncbi.nlm.nih.gov/articles/PMC10173919/pdf/42979_2023_Article_1809.pdf
Y. Shahsavari, Y. Baseri, A. Hafid, O. A. Dambri, and D. Makrakis, “Integration of federated learning and blockchain in health care: Tutorial on medical data, architectures, privacy, security, and regulatory compliance,” Journal of Medical Internet Research, vol. 28, Art. no. e80178, 2026. https://doi.org/10.2196/80178
K. Lekadir et al., “FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare,” BMJ, vol. 388, 2025. https://www.bmj.com/content/bmj/388/bmj-2024-081554.full.pdf
K. H. Hohman, M. Klompas, B. Zambarano, H. K. Wall, S. L. Jackson, and E. M. Kraus, “Validation of Multi-State EHR-Based Network for Disease Surveillance (MENDS) data and implications for improving data quality and representativeness,” Preventing Chronic Disease, vol. 21, Art. no. E43, 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11192496/pdf/PCD-21-E43.pdf
S. Nagababu, “Towards intelligent data testing: Integrating rule-based and adaptive validation techniques in healthcare systems,” International Journal of Computer Information Systems and Industrial Management Applications, vol. 18, no. 13s, pp. 758–766, 2026. https://cspub-ijcisim.org/index.php/ijcisim/article/download/4119/3301
S. K. R. Vanama, “Integrating Site Reliability Engineering SRE principles into enterprise architecture for predictive resilience,” International Journal of Emerging Trends in Computer Science and Information Technology, vol. 4, no. 3, pp. 164–170, 2023. https://www.ijetcsit.org/index.php/ijetcsit/article/download/514/462
X. Tang, L. Chen, and L. Zhang, “HF-DIF: Hierarchical feature-selected deep isolation forest for unsupervised anomaly detection on tabular data,” Journal of King Saud University Computer and Information Sciences, 2026. https://link.springer.com/content/pdf/10.1007/s44443-026-00706-6_reference.pdf
Z. Wang, X. Chen, Y. Wu, L. Jiang, S. Lin, and G. Qiu, “A robust and interpretable ensemble machine learning model for predicting healthcare insurance fraud,” Scientific Reports, vol. 15, no. 1, Art. no. 218, 2025. https://www.nature.com/articles/s41598-024-82062-x.pdf
J. T. Hancock, R. A. Bauder, H. Wang, and T. M. Khoshgoftaar, “Explainable machine learning models for Medicare fraud detection,” Journal of Big Data, vol. 10, Art. no. 154, 2023. https://link.springer.com/content/pdf/10.1186/s40537-023-00821-5.pdf
V. Nagaraj, S. R. Gundla, and P. R. Vennamaneni, “AI-driven secure and cloud-native microservice architectures for high-performance financial and hardware accelerator systems,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374740
Shekhar, J. Leder-Luis, and L. Akoglu, “Can machine learning target health care fraud? Evidence from Medicare hospitalizations,” Journal of Policy Analysis and Management, vol. 45, no. 1, Art. no. e70078, 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC13026505/
L. Grout, B. Anderson, A. Hannah, T. MacKenzie, S. Moonaz, and J. Whedon, “Geographic variation in utilization of chiropractic and primary care for new onset neck pain among Medicare fee-for-service beneficiaries in the United States in 2021,” BMC Health Services Research, 2026. https://link.springer.com/article/10.1186/s12913-026-15531-w
H. Shi, M. A. Tayebi, J. Pei, and J. Cao, “Cost-sensitive learning for medical insurance fraud detection with temporal information,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 10, pp. 10451–10463, 2023. https://www.sfu.ca/science/stat/cao/Papers/Fraud.pdf
M. A. A. El-Enen et al., “Fraud detection in medical insurance claims using majority voting of multiple unsupervised algorithms,” Procedia Computer Science, vol. 244, pp. 9–22, 2024. https://doi.org/10.1016/j.procs.2024.10.173
K. Razzaq and M. Shah, “Next-generation machine learning in healthcare fraud detection: Current trends, challenges, and future research directions,” Information, vol. 16, no. 9, Art. no. 730, 2025. https://www.mdpi.com/2078-2489/16/9/730
Most read articles by the same author(s)
- Engr. Mohammed Al-Farsi, Fatima Al Mansoori, Ahmed Zaki El-Sayed, OPTIMIZED POWER MANAGEMENT IN RESIDENTIAL SYSTEMS: AN IOT-DRIVEN APPROACH , International Journal of Intelligent Data and Machine Learning: Vol. 1 No. 01 (2024): Volume 01 Issue 01
- Abdullah Khalid Al-Harbi, Noor Fahad Al-Qahtani, A Quantum-Inspired Computational Architecture for Efficient High-Dimensional Data Processing , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 10 (2026): Volume 03 Issue 10
- Alexander V. Korovin, Optimizing Zero-Downtime Microservice Deployments: Integrating DevOps Principles in .NET Core Environments , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 01 (2026): Volume 03 Issue 01
- Yuki Nakamura, Hiroshi Tanaka, A SEMANTIC METRIC LEARNING APPROACH FOR ENHANCED MALWARE SIMILARITY SEARCH , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 01 (2025): Volume 02 Issue 01
- Yuki Nakamura, Isabella Romano, HYBRID DEEP LEARNING FOR TEXT CLASSIFICATION: INTEGRATING BIDIRECTIONAL GATED RECURRENT UNITS WITH CONVOLUTIONAL NEURAL NETWORKS , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 04 (2025): Volume 02 Issue 04
- Dr. Tanay Deshpande, Dr. Kavita Sharma, ADVANCING ARTIFICIAL INTELLIGENCE: AN IN-DEPTH LOOK AT MACHINE LEARNING AND DEEP LEARNING ARCHITECTURES, METHODOLOGIES, APPLICATIONS, AND FUTURE TRENDS , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 01 (2025): Volume 02 Issue 01
- Vaibhav Tummalapalli, Cohort-Based Segmentation Framework for Machine Learning: Structuring Temporal Data for Enhanced Feature Engineering , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 03 (2026): Volume 03 Issue 03
- Vaibhav Tummalapalli, A Framework for Adjusting Oversampling Bias in Machine Learning Models , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 04 (2026): Volume 03 Issue 04
- Chandra Obul Reddy Nagi Reddy , An Analytical Study of a Governed Lakehouse Framework in Multi-State Medicare Claims and Provider Data for Fraud, Waste, and Abuse Detection , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 10 (2026): Volume 03 Issue 10
Similar Articles
- Igor Litovsky, A Systematic Review of Machine Learning Approaches For AI-Driven Fraud Detection in Loyalty Programs , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 04 (2026): Volume 03 Issue 04
- Inna Simonova, Ai In Dispute Management: Automating Resolution and Reducing False Claims in E-Commerce , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 02 (2026): Volume 03 Issue 02
- Tristan K. Rowell, Real Time Event Streaming Architectures in Digital Finance: A Theoretical and Infrastructural Analysis of Kafka Based Financial Systems , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 10 (2025): Volume 02 Issue 10
- Dr Adrian Morrow, Dynamic AI Based Credit Scoring and Alternative Data Driven Risk Governance in Digital Lending Platforms , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 11 (2025): Volume 02 Issue 11
- Dr. Lucas Vermeulen, Sophie De Smet, Dr. Thomas Dubois, Integrated Temporal Analytics and AI-Based Approaches for Predicting Culinary Ingredient Consumption Patterns: Evidence from Thai Markets , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 04 (2026): Volume 03 Issue 04
- Dr. Julian E. Vance, Prof. Anya S. Petrova, Advancing Artificial Intelligence: An In-Depth Look at Machine Learning and Deep Learning Architectures, Methodologies, Applications, and Future Trends , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 09 (2025): Volume 02 Issue 09
- Dr. Amir Reza Khosravi, Distributed Stream Processing Models for Financial Markets: A Theoretical Investigation of Kafka-Based Infrastructure in High-Frequency Digital Finance Systems , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 06 (2026): Volume 03 Issue 06
- Dr. Minh Quang Tran, Dr. Lan Anh Nguyen, An Advanced Analytical Architecture for Leveraging Big Data in Artificial Intelligence Systems: Techniques, Optimization Strategies, and Case-Based Evaluation , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 08 (2026): Volume 03 Issue 08
- Bima Satria Nugraha, Professor Anindya larasati, Dr. Huỳnh Chí Dũng, Assessing The Interoperability And Semantic Readiness Of BIM And IFC Data For AI Integration In The Architecture, Engineering, And Construction Industry: A Systematic Review , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 11 (2025): Volume 02 Issue 11
- Dr. Natalia V. Smirnova, Elena Baranova, ADAPTIVE LINEAR MODELS FOR REGRESSION IN EVOLVING DATA STREAMS , International Journal of Intelligent Data and Machine Learning: Vol. 1 No. 01 (2024): Volume 01 Issue 01
1-10 of 60
Next
You may also start an advanced similarity search for this article.