Open Access
An Analytical Study of a Governed Lakehouse Framework in Multi-State Medicare Claims and Provider Data for Fraud, Waste, and Abuse Detection
Chandra Obul Reddy Nagi Reddy
1
,
,
4
Organization: Gainwell Technologies, Designation: Advisor Database Architect
Abstract
Medicare fraud detection, waste, and abuse is hampered by the high volume of claims, highly fragmented data-analysis systems, limited visibility into data quality, and significant class imbalance. This study assessed the potential of an Original Medicare Part B (OMB) Bronze-Silver-Gold lakehouse to improve data quality, traceability, analytical efficiency, and provider-level risk identification across OMB-covered populations in California, New York, Florida, Texas, and Pennsylvania in 2024. Using Apache Spark/PySpark with Delta-format tables, the study sampled 1,500,000 provider-service records in proportion to state Traditional Medicare enrollment, representing 214,873 unique NPIs, consolidated them with CMS provider files and HHS-OIG provider-exclusion data, interpreted 30 features, and used four models: Logistic Regression, Random Forest, XGBoost, and Isolation Forest. During Silver-layer screening, 27,362 records (1.82%) triggered one or more quality rules and were quarantined; NPI enrichment had 99.7% compliance, and Gold-layer data quality had 99.2% compliance. This reduced query time from a median of 18.4 seconds to 5.7 seconds (approximately 69%). XGBoost performed best among the supervised models, with ROC-AUC = 0.91, PR-AUC = 0.34, and F1 = 0.66. State elevated/high-risk flags were not confirmed as fraud; they were a priority for investigation and ranged from 2.2%–3.4% of unique providers within each state. Manually managed, repeatable pipelines provide defensible Medicare FWA prioritization, and repeatable, source-to-feature, auditable pipelines can support multi-state, analytic workflows where needed. Areas to explore include longitudinal claims, Part D, DMEPOS, enforcement outcomes, graph analytics, and prospective investigator validation.
Keywords
Medicare fraud,
waste and abuse,
Governed data lakehouse,
Medicare Part B claims
References
S. A. Maluleke, “Optometrists’ experience of healthcare fraud, waste and abuse (FWA) and strategies to combat it,” African Vision and Eye Health, vol. 83, no. 1, pp. 1–10, 2024. https://www.scielo.org.za/scielo.php?pid=S2410-15162024000100032&script=sci_arttext
Takefuji, “Generative AI for analysis and identification of Medicare improper payments by provider type and HCPC code,” Exploratory Research in Clinical and Social Pharmacy, vol. 12, Art. no. 100387, 2023. https://www.sciencedirect.com/science/article/pii/S2667276623001683
A. Teymourifar, “Technological advancements and the use of key performance indicators in hospital management: A critical-conceptual review,” Frontiers in Health Services, vol. 6, Art. no. 1780458, 2026. https://www.frontiersin.org/journals/health-services/articles/10.3389/frhs.2026.1780458/pdf
Centers for Medicare & Medicaid Services, “Medicare Physician & Other Practitioners - by Provider and Service,” CMS Data, 2024 data. https://data.cms.gov/provider-summary-by-type-of-service/medicare-physician-other-practitioners/medicare-physician-other-practitioners-by-provider-and-service
M. Suesserman, S. Gorny, D. Lasaga, J. Helms, D. Olson, E. Bowen, and S. Bhattacharya, “Procedure code overutilization detection from healthcare claims using unsupervised deep learning methods,” BMC Medical Informatics and Decision Making, vol. 23, no. 1, Art. no. 196, 2023. https://link.springer.com/content/pdf/10.1186/s12911-023-02268-3.pdf
A. du Preez, S. Bhattacharya, P. Beling, and E. Bowen, “Fraud detection in healthcare claims using machine learning: A systematic review,” Artificial Intelligence in Medicine, vol. 160, Art. no. 103061, 2025. https://www.sciencedirect.com/science/article/pii/S0933365724003038
A. A. Harby and F. H. Zulkernine, “Data Lakehouse: A survey and experimental study,” Information Systems, vol. 127, Art. no. 102460, 2025, doi: 10.1016/j.is.2024.102460. https://doi.org/10.1016/j.is.2024.102460
Sardana, R. Hariharan, and R. S. Pudi, “AI-driven secure and resilient healthcare data platforms with intelligent search and personalized communication,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–6. https://ieeexplore.ieee.org/document/11374370
A. Fannouch, J. Gharib, and Y. Gahi, “A governed Lakehouse DataOps architecture: Design and evaluation in healthcare,” HighTech and Innovation Journal, vol. 7, no. 2, pp. 704–731, Jun. 2026, doi: 10.28991/HIJ-2026-07-02-021. https://doi.org/10.28991/HIJ-2026-07-02-021
N. M. K. Koneru, “AI-driven optimization of CI/CD pipelines for multi-cloud infrastructure deployment,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–6. https://ieeexplore.ieee.org/abstract/document/11374441/
Centers for Medicare & Medicaid Services, “Medicare Physician & Other Practitioner Look-up Tool,” CMS Data, 2024 data. https://data.cms.gov/tools/medicare-physician-other-practitioner-look-up-tool
Centers for Medicare & Medicaid Services, “Medicare monthly enrollment,” CMS Data, 2024. https://data.cms.gov/summary-statistics-on-beneficiary-enrollment/medicare-and-medicaid-reports/medicare-monthly-enrollment
C. Bonthu, G. Malik, and M. R. Dhanagari, “AI-driven secure and scalable data architecture with multi-domain MDM and cybersecurity for real-time FinTech decision-making,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374416
T. Knezevic Ivanovski, S. Honap, R. Matic, S. Markovic, and L. Peyrin-Biroulet, “Building a healthcare data warehouse: Considerations, opportunities, and challenges,” Frontiers in Digital Health, vol. 7, Art. no. 1691142, 2025. https://www.frontiersin.org/journals/digital-health/articles/10.3389/fdgth.2025.1691142/pdf
M. Schlegel and K.-U. Sattler, “Capturing end-to-end provenance for machine learning pipelines,” Information Systems, vol. 132, Art. no. 102495, 2025. https://www.sciencedirect.com/science/article/pii/S0306437924001534
Z. Hamid, F. Khalique, S. Mahmood, A. Daud, A. Bukhari, and B. Alshemaimri, “Healthcare insurance fraud detection using data mining,” BMC Medical Informatics and Decision Making, vol. 24, no. 1, Art. no. 112, 2024. https://link.springer.com/content/pdf/10.1186/s12911-024-02512-4.pdf
S. Sepuri, “Comparative analysis of artificial intelligence test generation methods for compliance validation in regulated enterprise workflows,” Journal of Intelligent Decision Making and Information Science, vol. 3, no. 2s, pp. 107–123, 2026. https://doi.org/10.59543/jidmis.v3.632
J. Debener, V. Heinke, and J. Kriebel, “Detecting insurance fraud using supervised and unsupervised machine learning,” Journal of Risk and Insurance, vol. 90, no. 3, pp. 743–768, 2023. https://onlinelibrary.wiley.com/doi/pdf/10.1111/jori.12427
A. Cejudo, Y. Tellechea, A. Calvo, A. Almeida, C. Martín, and A. Beristain, “Scalable big data platform with end-to-end traceability for health data monitoring in older adults: Development and performance evaluation,” JMIR Medical Informatics, vol. 13, Art. no. e81701, 2025. https://medinform.jmir.org/2025/1/e81701
C. Flynn, T. Murphy, J. Walsh, and D. Riordan, “An open industrial energy dataset with asset-level measurements and high-coverage 15-minute aggregates from a manufacturing facility,” Data, vol. 11, no. 5, Art. no. 101, 2026. https://www.mdpi.com/2306-5729/11/5/101
L. L. Motta, W. S. Alvarez, H. R. Carvajal, F. B. Neto, E. R. de Lima, and L. G. P. Meloni, “A cloud architecture for home energy management systems: A conceptual model,” Energy Informatics, vol. 8, no. 1, Art. no. 142, 2025. https://link.springer.com/content/pdf/10.1186/s42162-025-00599-1.pdf
S. Rangu, K. S. Chadha, and S. Samala, “AI-powered cloud architecture, agile analytics, and enterprise DevOps-enabled framework for healthcare and insurance ecosystem optimisation,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374414
S. Scheider and M. K. Mallick, “Exploring metadata catalogs in health care data ecosystems: Taxonomy development study,” JMIR Formative Research, vol. 9, Art. no. e63396, 2025. https://formative.jmir.org/2025/1/e63396
N. Cheimarios, “Scientific software development in the AI era: Reproducibility, MLOps, and applications in soft matter physics,” Frontiers in Physics, vol. 13, Art. no. 1711356, 2025. https://www.frontiersin.org/journals/physics/articles/10.3389/fphy.2025.1711356/pdf
M. Armbrust et al., “Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores,” Proceedings of the VLDB Endowment, vol. 13, no. 12, pp. 3411–3424, 2020, doi: 10.14778/3415478.3415560. https://doi.org/10.14778/3415478.3415560
H. De Meulemeester, F. De Smet, J. van Dorst, E. Derroitte, and B. De Moor, “Explainable unsupervised anomaly detection for healthcare insurance data,” BMC Medical Informatics and Decision Making, vol. 25, no. 1, Art. no. 14, 2025. https://link.springer.com/content/pdf/10.1186/s12911-024-02823-6.pdf
D. J. Hekman et al., “Dashboarding to monitor machine-learning-based clinical decision support interventions,” Applied Clinical Informatics, vol. 15, no. 1, pp. 164–169, 2024. https://www.thieme-connect.com/products/ejournals/pdf/10.1055/a-2219-5175.pdf
J. M. Johnson and T. M. Khoshgoftaar, “Data-centric AI for healthcare fraud detection,” SN Computer Science, vol. 4, no. 4, Art. no. 389, 2023. https://pmc.ncbi.nlm.nih.gov/articles/PMC10173919/pdf/42979_2023_Article_1809.pdf
Y. Shahsavari, Y. Baseri, A. Hafid, O. A. Dambri, and D. Makrakis, “Integration of federated learning and blockchain in health care: Tutorial on medical data, architectures, privacy, security, and regulatory compliance,” Journal of Medical Internet Research, vol. 28, Art. no. e80178, 2026. https://doi.org/10.2196/80178
K. Lekadir et al., “FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare,” BMJ, vol. 388, 2025. https://www.bmj.com/content/bmj/388/bmj-2024-081554.full.pdf
K. H. Hohman, M. Klompas, B. Zambarano, H. K. Wall, S. L. Jackson, and E. M. Kraus, “Validation of Multi-State EHR-Based Network for Disease Surveillance (MENDS) data and implications for improving data quality and representativeness,” Preventing Chronic Disease, vol. 21, Art. no. E43, 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11192496/pdf/PCD-21-E43.pdf
S. Nagababu, “Towards intelligent data testing: Integrating rule-based and adaptive validation techniques in healthcare systems,” International Journal of Computer Information Systems and Industrial Management Applications, vol. 18, no. 13s, pp. 758–766, 2026. https://cspub-ijcisim.org/index.php/ijcisim/article/download/4119/3301
S. K. R. Vanama, “Integrating Site Reliability Engineering SRE principles into enterprise architecture for predictive resilience,” International Journal of Emerging Trends in Computer Science and Information Technology, vol. 4, no. 3, pp. 164–170, 2023. https://www.ijetcsit.org/index.php/ijetcsit/article/download/514/462
X. Tang, L. Chen, and L. Zhang, “HF-DIF: Hierarchical feature-selected deep isolation forest for unsupervised anomaly detection on tabular data,” Journal of King Saud University Computer and Information Sciences, 2026. https://link.springer.com/content/pdf/10.1007/s44443-026-00706-6_reference.pdf
Z. Wang, X. Chen, Y. Wu, L. Jiang, S. Lin, and G. Qiu, “A robust and interpretable ensemble machine learning model for predicting healthcare insurance fraud,” Scientific Reports, vol. 15, no. 1, Art. no. 218, 2025. https://www.nature.com/articles/s41598-024-82062-x.pdf
J. T. Hancock, R. A. Bauder, H. Wang, and T. M. Khoshgoftaar, “Explainable machine learning models for Medicare fraud detection,” Journal of Big Data, vol. 10, Art. no. 154, 2023. https://link.springer.com/content/pdf/10.1186/s40537-023-00821-5.pdf
V. Nagaraj, S. R. Gundla, and P. R. Vennamaneni, “AI-driven secure and cloud-native microservice architectures for high-performance financial and hardware accelerator systems,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374740
Shekhar, J. Leder-Luis, and L. Akoglu, “Can machine learning target health care fraud? Evidence from Medicare hospitalizations,” Journal of Policy Analysis and Management, vol. 45, no. 1, Art. no. e70078, 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC13026505/
L. Grout, B. Anderson, A. Hannah, T. MacKenzie, S. Moonaz, and J. Whedon, “Geographic variation in utilization of chiropractic and primary care for new onset neck pain among Medicare fee-for-service beneficiaries in the United States in 2021,” BMC Health Services Research, 2026. https://link.springer.com/article/10.1186/s12913-026-15531-w
H. Shi, M. A. Tayebi, J. Pei, and J. Cao, “Cost-sensitive learning for medical insurance fraud detection with temporal information,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 10, pp. 10451–10463, 2023. https://www.sfu.ca/science/stat/cao/Papers/Fraud.pdf
M. A. A. El-Enen et al., “Fraud detection in medical insurance claims using majority voting of multiple unsupervised algorithms,” Procedia Computer Science, vol. 244, pp. 9–22, 2024. https://doi.org/10.1016/j.procs.2024.10.173
K. Razzaq and M. Shah, “Next-generation machine learning in healthcare fraud detection: Current trends, challenges, and future research directions,” Information, vol. 16, no. 9, Art. no. 730, 2025. https://www.mdpi.com/2078-2489/16/9/730
Most read articles by the same author(s)
- Mr. Madhav Sharma, Prediction of Heart Disease Using Ensemble Machine Learning Techniques , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Chinedu Emmanuel Okafor, Intelligent Healthcare Systems Powered by Large Language Models: Applications, Limitations, and Emerging Research Perspectives , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Muhammad Rizky Pratama, Siti Aulia Rahma, Interpretable Predictive Analytics Approach for Robust Financial Risk Assessment and Forecasting , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 09 (2026): Volume 03 Issue 09
- Igor Litovsky, A Systematic Review of Machine Learning Approaches For AI-Driven Fraud Detection in Loyalty Programs , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 04 (2026): Volume 03 Issue 04
- Bima Satria Nugraha, Professor Anindya larasati, Dr. Huỳnh Chí Dũng, Assessing The Interoperability And Semantic Readiness Of BIM And IFC Data For AI Integration In The Architecture, Engineering, And Construction Industry: A Systematic Review , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 11 (2025): Volume 02 Issue 11
- Elias J. Vance, Clara M. Soto, High-Frequency Data Driven Network Learning for Systemic Risk Analysis in Financial Markets , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 09 (2025): Volume 02 Issue 09
- Dr. Nguyen Minh Anh, Dr. Tran Hoang Nam, A Scalable Multi-Tenant Framework for AI-Driven Big Data Lake Management and Processing , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 08 (2026): Volume 03 Issue 08
- Dr. Hannah Brown, Ahmed Al-Farsi, BRIDGING DEEP LEARNING AND ADAPTIVE SYSTEMS: A PERFORMANCE STUDY ON CIFAR-10 IMAGE CLASSIFICATION , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 03 (2025): Volume 02 Issue 03
- Eko Purnomo, Rendra Alfiansyah, A Dynamic Nexus: Integrating Big Data Analytics and Distributed Computing for Real-Time Risk Management of Derivatives Portfolios , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 10 (2025): Volume 02 Issue 10
- Ali Hassan Kareem, Mariam Saad, Machine Learning-Based SAP Production Planning Framework for Demand Prediction, Inventory Reduction, and Throughput Optimization , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 10 (2026): Volume 03 Issue 10
Similar Articles
- Mateo Laurent Dufour, Architecting Secure and Scalable Production Machine Learning Systems: Integrating Model Management, High Performance Computing, and Cloud Native Infrastructure , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 03 (2026): Volume 03 Issue 03
- Dr. Elena Petrova, Prof. David J. Hernandez, MACHINE LEARNING MODEL IMPLEMENTATION STRATEGIES AND PREDICTIVE FACTORS FOR PREECLAMPSIA FORECASTING: A REVIEW , International Journal of Intelligent Data and Machine Learning: Vol. 1 No. 01 (2024): Volume 01 Issue 01
- Dr. Tashi Wangchuk, Karma Lhendup, Data-Driven Model Supporting Defect Analysis through Vision Techniques in Press-Formed Vehicle Components , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 04 (2026): Volume 03 Issue 04
- Ananya Patel (Ph.D. Candidate), ADVANCING FINANCIAL PREDICTION THROUGH QUANTUM MACHINE LEARNING , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 02 (2025): Volume 02 Issue 02
- Kartik Tandon, Dr. Priya Menon, LEVERAGING MACHINE LEARNING TO IDENTIFY MATERNAL RISK FACTORS FOR CONGENITAL HEART DISEASE IN OFFSPRING , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 05 (2025): Volume 02 Issue 05
- Isabella Rossi, Elena Petrova, LEVERAGING QUANTUM CONVOLUTIONAL LAYERS FOR ENHANCED IMAGE CLASSIFICATION: AN EXAMINATION OF QUANVOLUTIONAL NEURAL NETWORK CHARACTERISTICS , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 06 (2025): Volume 02 Issue 06
- Dr. Jianhong Liu, Dr. Meilin Zhou, A Machine Learning–Driven Framework for Multi-Temporal Flood Inundation Mapping and Spatial Analysis in Kolhapur, India Using SAR Remote Sensing Observations , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 07 (2026): Volume 03 Issue 07
- Mr. Ram Pratap Singh, An Intelligent Machine Learning Framework for Customer Churn Prediction in CRM Systems , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 07 (2026): Volume 03 Issue 07
- Dr. Larian D. Venorth, Prof. Maevis K. Durand, The Transformative Trajectory Of Large Language Models: Societal Impact, Predictive Limitations, And The Unforeseen Geohazard Nexus , International Journal of Intelligent Data and Machine Learning: Vol. 2 No. 10 (2025): Volume 02 Issue 10
- Dr. Kwame Mensah, Dr. Abena Owusu, An Interpretable Visual Analytics Framework for Machine Learning–Based Multichannel Time Series Classification and Performance Evaluation , International Journal of Intelligent Data and Machine Learning: Vol. 3 No. 08 (2026): Volume 03 Issue 08
You may also start an advanced similarity search for this article.