Open Access

An Analytical Study of a Governed Lakehouse Framework in Multi-State Medicare Claims and Provider Data for Fraud, Waste, and Abuse Detection

https://doi.org/10.55640/ijidml-v03i10-03

PDF

Chandra Obul Reddy Nagi Reddy ¹

⁴ Organization: Gainwell Technologies, Designation: Advisor Database Architect

Abstract

Medicare fraud detection, waste, and abuse is hampered by the high volume of claims, highly fragmented data-analysis systems, limited visibility into data quality, and significant class imbalance. This study assessed the potential of an Original Medicare Part B (OMB) Bronze-Silver-Gold lakehouse to improve data quality, traceability, analytical efficiency, and provider-level risk identification across OMB-covered populations in California, New York, Florida, Texas, and Pennsylvania in 2024. Using Apache Spark/PySpark with Delta-format tables, the study sampled 1,500,000 provider-service records in proportion to state Traditional Medicare enrollment, representing 214,873 unique NPIs, consolidated them with CMS provider files and HHS-OIG provider-exclusion data, interpreted 30 features, and used four models: Logistic Regression, Random Forest, XGBoost, and Isolation Forest. During Silver-layer screening, 27,362 records (1.82%) triggered one or more quality rules and were quarantined; NPI enrichment had 99.7% compliance, and Gold-layer data quality had 99.2% compliance. This reduced query time from a median of 18.4 seconds to 5.7 seconds (approximately 69%). XGBoost performed best among the supervised models, with ROC-AUC = 0.91, PR-AUC = 0.34, and F1 = 0.66. State elevated/high-risk flags were not confirmed as fraud; they were a priority for investigation and ranged from 2.2%–3.4% of unique providers within each state. Manually managed, repeatable pipelines provide defensible Medicare FWA prioritization, and repeatable, source-to-feature, auditable pipelines can support multi-state, analytic workflows where needed. Areas to explore include longitudinal claims, Part D, DMEPOS, enforcement outcomes, graph analytics, and prospective investigator validation.

Keywords

Medicare fraud, waste and abuse, Governed data lakehouse, Medicare Part B claims

References

S. A. Maluleke, “Optometrists’ experience of healthcare fraud, waste and abuse (FWA) and strategies to combat it,” African Vision and Eye Health, vol. 83, no. 1, pp. 1–10, 2024. https://www.scielo.org.za/scielo.php?pid=S2410-15162024000100032&script=sci_arttext

Takefuji, “Generative AI for analysis and identification of Medicare improper payments by provider type and HCPC code,” Exploratory Research in Clinical and Social Pharmacy, vol. 12, Art. no. 100387, 2023. https://www.sciencedirect.com/science/article/pii/S2667276623001683

A. Teymourifar, “Technological advancements and the use of key performance indicators in hospital management: A critical-conceptual review,” Frontiers in Health Services, vol. 6, Art. no. 1780458, 2026. https://www.frontiersin.org/journals/health-services/articles/10.3389/frhs.2026.1780458/pdf

Centers for Medicare & Medicaid Services, “Medicare Physician & Other Practitioners - by Provider and Service,” CMS Data, 2024 data. https://data.cms.gov/provider-summary-by-type-of-service/medicare-physician-other-practitioners/medicare-physician-other-practitioners-by-provider-and-service

M. Suesserman, S. Gorny, D. Lasaga, J. Helms, D. Olson, E. Bowen, and S. Bhattacharya, “Procedure code overutilization detection from healthcare claims using unsupervised deep learning methods,” BMC Medical Informatics and Decision Making, vol. 23, no. 1, Art. no. 196, 2023. https://link.springer.com/content/pdf/10.1186/s12911-023-02268-3.pdf

A. du Preez, S. Bhattacharya, P. Beling, and E. Bowen, “Fraud detection in healthcare claims using machine learning: A systematic review,” Artificial Intelligence in Medicine, vol. 160, Art. no. 103061, 2025. https://www.sciencedirect.com/science/article/pii/S0933365724003038

A. A. Harby and F. H. Zulkernine, “Data Lakehouse: A survey and experimental study,” Information Systems, vol. 127, Art. no. 102460, 2025, doi: 10.1016/j.is.2024.102460. https://doi.org/10.1016/j.is.2024.102460

Sardana, R. Hariharan, and R. S. Pudi, “AI-driven secure and resilient healthcare data platforms with intelligent search and personalized communication,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–6. https://ieeexplore.ieee.org/document/11374370

A. Fannouch, J. Gharib, and Y. Gahi, “A governed Lakehouse DataOps architecture: Design and evaluation in healthcare,” HighTech and Innovation Journal, vol. 7, no. 2, pp. 704–731, Jun. 2026, doi: 10.28991/HIJ-2026-07-02-021. https://doi.org/10.28991/HIJ-2026-07-02-021

N. M. K. Koneru, “AI-driven optimization of CI/CD pipelines for multi-cloud infrastructure deployment,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–6. https://ieeexplore.ieee.org/abstract/document/11374441/

Centers for Medicare & Medicaid Services, “Medicare Physician & Other Practitioner Look-up Tool,” CMS Data, 2024 data. https://data.cms.gov/tools/medicare-physician-other-practitioner-look-up-tool

Centers for Medicare & Medicaid Services, “Medicare monthly enrollment,” CMS Data, 2024. https://data.cms.gov/summary-statistics-on-beneficiary-enrollment/medicare-and-medicaid-reports/medicare-monthly-enrollment

C. Bonthu, G. Malik, and M. R. Dhanagari, “AI-driven secure and scalable data architecture with multi-domain MDM and cybersecurity for real-time FinTech decision-making,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374416

T. Knezevic Ivanovski, S. Honap, R. Matic, S. Markovic, and L. Peyrin-Biroulet, “Building a healthcare data warehouse: Considerations, opportunities, and challenges,” Frontiers in Digital Health, vol. 7, Art. no. 1691142, 2025. https://www.frontiersin.org/journals/digital-health/articles/10.3389/fdgth.2025.1691142/pdf

M. Schlegel and K.-U. Sattler, “Capturing end-to-end provenance for machine learning pipelines,” Information Systems, vol. 132, Art. no. 102495, 2025. https://www.sciencedirect.com/science/article/pii/S0306437924001534

Z. Hamid, F. Khalique, S. Mahmood, A. Daud, A. Bukhari, and B. Alshemaimri, “Healthcare insurance fraud detection using data mining,” BMC Medical Informatics and Decision Making, vol. 24, no. 1, Art. no. 112, 2024. https://link.springer.com/content/pdf/10.1186/s12911-024-02512-4.pdf

S. Sepuri, “Comparative analysis of artificial intelligence test generation methods for compliance validation in regulated enterprise workflows,” Journal of Intelligent Decision Making and Information Science, vol. 3, no. 2s, pp. 107–123, 2026. https://doi.org/10.59543/jidmis.v3.632

J. Debener, V. Heinke, and J. Kriebel, “Detecting insurance fraud using supervised and unsupervised machine learning,” Journal of Risk and Insurance, vol. 90, no. 3, pp. 743–768, 2023. https://onlinelibrary.wiley.com/doi/pdf/10.1111/jori.12427

A. Cejudo, Y. Tellechea, A. Calvo, A. Almeida, C. Martín, and A. Beristain, “Scalable big data platform with end-to-end traceability for health data monitoring in older adults: Development and performance evaluation,” JMIR Medical Informatics, vol. 13, Art. no. e81701, 2025. https://medinform.jmir.org/2025/1/e81701

C. Flynn, T. Murphy, J. Walsh, and D. Riordan, “An open industrial energy dataset with asset-level measurements and high-coverage 15-minute aggregates from a manufacturing facility,” Data, vol. 11, no. 5, Art. no. 101, 2026. https://www.mdpi.com/2306-5729/11/5/101

L. L. Motta, W. S. Alvarez, H. R. Carvajal, F. B. Neto, E. R. de Lima, and L. G. P. Meloni, “A cloud architecture for home energy management systems: A conceptual model,” Energy Informatics, vol. 8, no. 1, Art. no. 142, 2025. https://link.springer.com/content/pdf/10.1186/s42162-025-00599-1.pdf

S. Rangu, K. S. Chadha, and S. Samala, “AI-powered cloud architecture, agile analytics, and enterprise DevOps-enabled framework for healthcare and insurance ecosystem optimisation,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374414

S. Scheider and M. K. Mallick, “Exploring metadata catalogs in health care data ecosystems: Taxonomy development study,” JMIR Formative Research, vol. 9, Art. no. e63396, 2025. https://formative.jmir.org/2025/1/e63396

N. Cheimarios, “Scientific software development in the AI era: Reproducibility, MLOps, and applications in soft matter physics,” Frontiers in Physics, vol. 13, Art. no. 1711356, 2025. https://www.frontiersin.org/journals/physics/articles/10.3389/fphy.2025.1711356/pdf

M. Armbrust et al., “Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores,” Proceedings of the VLDB Endowment, vol. 13, no. 12, pp. 3411–3424, 2020, doi: 10.14778/3415478.3415560. https://doi.org/10.14778/3415478.3415560

H. De Meulemeester, F. De Smet, J. van Dorst, E. Derroitte, and B. De Moor, “Explainable unsupervised anomaly detection for healthcare insurance data,” BMC Medical Informatics and Decision Making, vol. 25, no. 1, Art. no. 14, 2025. https://link.springer.com/content/pdf/10.1186/s12911-024-02823-6.pdf

D. J. Hekman et al., “Dashboarding to monitor machine-learning-based clinical decision support interventions,” Applied Clinical Informatics, vol. 15, no. 1, pp. 164–169, 2024. https://www.thieme-connect.com/products/ejournals/pdf/10.1055/a-2219-5175.pdf

J. M. Johnson and T. M. Khoshgoftaar, “Data-centric AI for healthcare fraud detection,” SN Computer Science, vol. 4, no. 4, Art. no. 389, 2023. https://pmc.ncbi.nlm.nih.gov/articles/PMC10173919/pdf/42979_2023_Article_1809.pdf

Y. Shahsavari, Y. Baseri, A. Hafid, O. A. Dambri, and D. Makrakis, “Integration of federated learning and blockchain in health care: Tutorial on medical data, architectures, privacy, security, and regulatory compliance,” Journal of Medical Internet Research, vol. 28, Art. no. e80178, 2026. https://doi.org/10.2196/80178

K. Lekadir et al., “FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare,” BMJ, vol. 388, 2025. https://www.bmj.com/content/bmj/388/bmj-2024-081554.full.pdf

K. H. Hohman, M. Klompas, B. Zambarano, H. K. Wall, S. L. Jackson, and E. M. Kraus, “Validation of Multi-State EHR-Based Network for Disease Surveillance (MENDS) data and implications for improving data quality and representativeness,” Preventing Chronic Disease, vol. 21, Art. no. E43, 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11192496/pdf/PCD-21-E43.pdf

S. Nagababu, “Towards intelligent data testing: Integrating rule-based and adaptive validation techniques in healthcare systems,” International Journal of Computer Information Systems and Industrial Management Applications, vol. 18, no. 13s, pp. 758–766, 2026. https://cspub-ijcisim.org/index.php/ijcisim/article/download/4119/3301

S. K. R. Vanama, “Integrating Site Reliability Engineering SRE principles into enterprise architecture for predictive resilience,” International Journal of Emerging Trends in Computer Science and Information Technology, vol. 4, no. 3, pp. 164–170, 2023. https://www.ijetcsit.org/index.php/ijetcsit/article/download/514/462

X. Tang, L. Chen, and L. Zhang, “HF-DIF: Hierarchical feature-selected deep isolation forest for unsupervised anomaly detection on tabular data,” Journal of King Saud University Computer and Information Sciences, 2026. https://link.springer.com/content/pdf/10.1007/s44443-026-00706-6_reference.pdf

Z. Wang, X. Chen, Y. Wu, L. Jiang, S. Lin, and G. Qiu, “A robust and interpretable ensemble machine learning model for predicting healthcare insurance fraud,” Scientific Reports, vol. 15, no. 1, Art. no. 218, 2025. https://www.nature.com/articles/s41598-024-82062-x.pdf

J. T. Hancock, R. A. Bauder, H. Wang, and T. M. Khoshgoftaar, “Explainable machine learning models for Medicare fraud detection,” Journal of Big Data, vol. 10, Art. no. 154, 2023. https://link.springer.com/content/pdf/10.1186/s40537-023-00821-5.pdf

V. Nagaraj, S. R. Gundla, and P. R. Vennamaneni, “AI-driven secure and cloud-native microservice architectures for high-performance financial and hardware accelerator systems,” in Proc. 2025 Tenth International Conference on Science Technology Engineering and Mathematics (ICONSTEM), 2025, pp. 1–7. https://ieeexplore.ieee.org/document/11374740

Shekhar, J. Leder-Luis, and L. Akoglu, “Can machine learning target health care fraud? Evidence from Medicare hospitalizations,” Journal of Policy Analysis and Management, vol. 45, no. 1, Art. no. e70078, 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC13026505/

L. Grout, B. Anderson, A. Hannah, T. MacKenzie, S. Moonaz, and J. Whedon, “Geographic variation in utilization of chiropractic and primary care for new onset neck pain among Medicare fee-for-service beneficiaries in the United States in 2021,” BMC Health Services Research, 2026. https://link.springer.com/article/10.1186/s12913-026-15531-w

H. Shi, M. A. Tayebi, J. Pei, and J. Cao, “Cost-sensitive learning for medical insurance fraud detection with temporal information,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 10, pp. 10451–10463, 2023. https://www.sfu.ca/science/stat/cao/Papers/Fraud.pdf

M. A. A. El-Enen et al., “Fraud detection in medical insurance claims using majority voting of multiple unsupervised algorithms,” Procedia Computer Science, vol. 244, pp. 9–22, 2024. https://doi.org/10.1016/j.procs.2024.10.173

K. Razzaq and M. Shah, “Next-generation machine learning in healthcare fraud detection: Current trends, challenges, and future research directions,” Information, vol. 16, no. 9, Art. no. 730, 2025. https://www.mdpi.com/2078-2489/16/9/730

Most read articles by the same author(s)

<< < 1 2 3 4 5 6 7 > >>

International Journal of Intelligent Data and Machine Learning

An Analytical Study of a Governed Lakehouse Framework in Multi-State Medicare Claims and Provider Data for Fraud, Waste, and Abuse Detection

Abstract

Keywords

References

Most read articles by the same author(s)

Similar Articles