eISSN: 3087-4270 editor@aimjournals.com
All Journals
Submit your article

International Journal of Advanced Artificial Intelligence Research

Open Access Peer Review International
Open Access

Policy-As-Code Governance For Bronze-Layer Ingestion In Regulated Databricks Medallion Lakehouse Architectures

DOI https://doi.org/10.55640/ijaair-v03i10-03
PDF
Shikha Sharma 1 ORCID iD ,
4 Independent Researcher, Arlington, VA, USA

Abstract

Modern data platforms widely use the medallion architecture to organize pipelines on lakehouse systems such as Databricks. Governance controls, including schema validation, sensitive-data handling, retention, lineage tracking, quality checks, and access management, are typically applied in downstream layers or through manual processes. This leaves the bronze layer, where raw data first enters the platform, inconsistently governed. In regulated sectors, a lapse at this layer, such as an unhandled schema change or unmasked sensitive field, can spread across the pipeline and create compliance, financial, and operational risk, because downstream analytics and machine learning depend on the integrity of this raw foundation. This paper reviews and synthesizes existing governance, data quality, and policy-as-code approaches, then proposes a unifying framework that expresses governance requirements as declarative, version-controlled policies enforced automatically at bronze ingestion. The framework combines a policy repository, an enforcement engine with allow, warn, quarantine, and reject outcomes, and an append-only audit store, using Databricks-native tools and general policy engines. The synthesis shows that six governance concerns, namely schema, sensitive data, retention, lineage, quality, and access, can be coordinated by a single policy layer in which each concern is handled by the mechanism suited to it: an ingestion-time gate for schema, sensitive data, and quality; a lifecycle policy for retention; metadata capture for lineage; and access control applied at query and access time. The contribution is a unified policy layer that coordinates these six concerns with continuous auditing, rather than forcing all of them through one enforcement point. Novelty is claimed relative to the approaches examined in this review.

Keywords

Data governance, policy-as-code, Databricks medallion architecture, bronze-layer ingestion

References

I. Moncivais, "Databricks Unity Catalog: A Unified Governance Framework for Enterprise Data and AI Platforms," 2026. DOI: 10.2139/ssrn.6905498.
L. M. K. Brahmandam, "An Empirical Evaluation of the Medallion Architecture on Databricks and Apache Spark with Snowflake: Throughput, Latency, and Cost for Batch and Real-Time Ingestion Patterns," Int. J. AI, BigData, Computational and Management Studies, vol. 5, no. 3, pp. 197–206, 2024. DOI: 10.63282/3050-9416.ijaibdcms-v5i3p122.
A. K. Kolla, A. Kokate, R. Vellanki, R. Lolam, and P. Bandi, "A Metadata-Driven Framework for Automated Bronze-to-Silver Data Ingestion, Validation and Loading on Databricks," European J. Information Technologies and Computer Science, vol. 6, no. 3, pp. 1–7, 2026. DOI: 10.24018/compute.2026.6.3.70578.
K. Madan, "ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Code," arXiv:2509.07006, 2025.
Z. Qiu, "Data Engineering Patterns for Cross-System Reconciliation in Regulated Enterprises: Architecture, Anomaly Detection, and Governance," arXiv:2604.15108, 2026.
H. A. Mohna, T. Barua, M. Mohiuddin, and M. M. Rahman, "AI-Ready Data Engineering Pipelines: A Review of Medallion Architecture and Cloud-Based Integration Models," American J. Scholarly Research and Innovation, vol. 1, no. 1, pp. 319–350, 2022. DOI: 10.63125/51kxtf08.
A. Fannouch, J. Gharib, and Y. Gahi, "A Governed Lakehouse DataOps Architecture: Design and Evaluation in Healthcare," HighTech and Innovation Journal, 2026. DOI: 10.28991/hij-2026-07-02-021.
M. Armbrust, T. Das, S. Paranjpye, R. Xin, S. Zhu, A. Ghodsi, et al., "Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores," Proc. VLDB Endowment, vol. 13, no. 12, pp. 3411-3424, 2020. DOI: 10.14778/3415478.3415560.
M. Armbrust, A. Ghodsi, R. Xin, and M. Zaharia, "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics," in Proc. Conf. Innovative Data Systems Research (CIDR), 2021. [Online]. Available: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf.
A. A. Harby and F. Zulkernine, "From Data Warehouse to Lakehouse: A Comparative Review," in Proc. IEEE Int. Conf. Big Data (Big Data), 2022, pp. 389-395. DOI: 10.1109/BigData55660.2022.10020719.
E. Begoli, I. Goethert, and K. Knight, "A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks," in Proc. IEEE Int. Conf. Big Data (Big Data), 2021, pp. 4643-4651.
A. A. Harby and F. Zulkernine, "Data Lakehouse: A Survey and Experimental Study," Information Systems, vol. 127, art. 102460, 2025. DOI: 10.1016/j.is.2024.102460.
P. Jain, P. Kraft, C. Power, T. Das, I. Stoica, and M. Zaharia, "Analyzing and Comparing Lakehouse Storage Systems," in Proc. Conf. Innovative Data Systems Research (CIDR), 2023.
Open Policy Agent, "Open Policy Agent: Policy-based control for cloud native environments," Cloud Native Computing Foundation. [Online]. Available: https://www.openpolicyagent.org/. Accessed: 2026.
Amazon Web Services, "Governing Infrastructure as Code Using Pattern-Based Policy as Code," AWS Security Blog, 2026. [Online]. Available: https://aws.amazon.com/blogs/security/governing-infrastructure-as-code-using-pattern-based-policy-as-code/.
I. Moncivais, "Data Architecture and Governance in the Agentic Era: Paradigm Shifts from 2026 Databricks DAIS," 2026. DOI: 10.2139/ssrn.6979203.
Acceldata, "What Does Policy-as-Code Look Like for Enterprise Data Governance," 2026. [Online]. Available: https://www.acceldata.io/blog/what-does-policy-as-code-look-like-for-enterprise-data-governance.
R. Kang, "Governed AI-Assisted Engineering: Graduated Human Oversight for Agentic Code Generation in Regulated Domains," arXiv:2606.22484, 2026.
H. Morgan, "AI Governance Control Stack for Operational Stability: Achieving Hardened Governance in AI Systems," arXiv:2604.03262, 2026.

Most read articles by the same author(s)

<< < 2 3 4 5 6 7 8 9 10 11 > >> 

Similar Articles

1-10 of 78

You may also start an advanced similarity search for this article.