eISSN: 3087-4262 editor@aimjournals.com
All Journals
Submit your article

International Journal of Intelligent Data and Machine Learning

Open Access Peer Review International
Open Access

A Scalable Multi-Tenant Framework for AI-Driven Big Data Lake Management and Processing

DOI https://doi.org/10.55640/ijidml-v03i08-03
PDF
Dr. Nguyen Minh Anh 1 , Dr. Tran Hoang Nam 1 ,
4 Department of Artificial Intelligence Vietnam Institute of Advanced Computing Hanoi, Vietnam
4 Department of Machine Learning and Data Science Center for Intelligent Technology Ho Chi Minh City, Vietnam

Abstract

The rapid growth of artificial intelligence (AI), machine learning, and heterogeneous data sources has increased the need for scalable data-lake architectures capable of supporting concurrent workloads, diverse data types, and dynamically changing computational requirements. Conventional data-management approaches often struggle to provide adequate isolation, resource elasticity, governance, and intelligent workload coordination in multi-tenant environments. This research proposes a conceptual scalable multi-tenant framework for AI-driven big data lake management and processing in which tenant-aware orchestration, workload classification, resource allocation, data governance, and AI-assisted decision mechanisms operate as integrated architectural components. The framework is theoretically grounded in scalable data-lake orchestration, formal reasoning, explainable decision-making, and resource-aware computational management. Particular emphasis is placed on separating tenant-level policies from shared infrastructure while maintaining efficient utilization of storage and processing resources. The architectural rationale is informed by research on rule-based learning, formal verification, satisfiability solving, explainability, and computational reasoning. The proposed framework further incorporates principles associated with multitenant data-lake orchestration for AI workloads (Goyal, 2025). Analytical findings indicate that a policy-aware, AI-driven orchestration layer can improve workload prioritization, resource utilization, isolation, and operational transparency compared with static allocation models. The study also identifies limitations associated with governance complexity, model dependence, computational overhead, and fairness across tenants. The resulting framework provides a research foundation for scalable, intelligent, and explainable big data lake management.

Keywords

Big Data Lake, Multi-Tenancy, Artificial Intelligence, AI Workloads

References

Acharya, M. S., Armaan, A., & Antony, A. S. (2019). A comparison of regression models for prediction of graduate admissions.2019 International Conference on Computational Intelligence in Data Science (ICCIDS), 1–5.
Angelino, E., Larus-Stone, N., Alabi, D., Seltzer, M., & Rudin, C. (2018). Learning certifiably optimal rule lists for categorical data .Journal of Machine Learning Research,18(234),1–78.
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. Pro Publica. Retrieved February 26, 2024, from
Barenstein, M. (2019).Pro Publica’s COMPAS data revisited. arXiv: 1906.04711[cs, econ,q-fin, stat].
Bench-Capon, T. (1993). Neural networks and open texture. Proceedings of the Fourth International Conference on Artificial Intelligence and Law, 292–297.
Bjørner, N., & Nachmanson, L. (2020). Navigating the universe of Z3 theory solvers. Formal Methods: Foundations and Applications, 8–24.
Bradley, A. R., & Manna, Z. (2007).The calculus of computation. Springer.
ˇCyras, K., Satoh, K., & Toni, F. (2016). Explanation for case-based reasoning via abstractargu mentation. In P. Baroni, T. F. Gordon, T. Scheffler, & M. Stede (Eds.),Com-putational Models of Argument. Proceedings of COMMA 2016(pp. 243–254). IOSPress.
Davey, B. A., & Priestley, H. A. (2002).Introduction to lattices and order(2nd ed.).Cambridge University Press.
de Moura, L., & Bjørner, N. (2008). Z3: An efficient SMT solver. Tools and Algorithms forthe Construction and Analysis of Systems, 337–340.
de Moura, L., & Bjørner, N. (2009). Satisfiability modulo theories: An appetizer. Formal Methods: Foundations and Applications, 23–36.
Dieterich, W., Mendoza, C., & Brennan, T. (2016).COMPAS risk scales: Demonstratin gaccuracy equity and predictive parity(Research report). Northpointe Inc. Research Department.
Horty, J. (2011). Rules and reasons in the theory of precedent. Legal Theory,17(1), 1–33.
Horty, J. (2019). Reasoning with dimensions and magnitudes. Artificial Intelligence andLaw,27(3), 309–345.
Koh, P. W., & Liang, P. (2017). Understanding black-box predictions via influence functions. Proceedings of the 34th International Conference on Machine Learning, 1885–1894.
K. K. Goyal, "Scalable Data Lakes for AI Workloads: A Multitenant Architecture for Big Data Orchestration," 2025 IEEE International Conference on Computing (ICOCO), Kuching, Malaysia, 2025, pp. 266-271, doi: 10.1109/ICOCO67189.2025.11334100.
Vollem, S., Mulla, F. M., Shah, A. K., Kodela, S., Kaur, M., & Kumar, V. (2026, February). Multi-Model Time-Series Forecasting Framework for Stock Price Prediction Using Statistical and Deep Learning Techniques. In 2026 2nd International Conference on Big Data & Machine Learning (ICBDML) (pp. 1-6). IEEE.

Most read articles by the same author(s)

1 2 3 > >> 

Similar Articles

1-10 of 59

You may also start an advanced similarity search for this article.