Open Access

Debugging Billion-Core AI Clusters: Distributed Crash Diagnosis for Heterogeneous Accelerator Systems

https://doi.org/10.55640/ijmcsit-v03i09-04

PDF

Mohammad Shuab Siddique ¹ ,

⁴ Independent Researcher

Abstract

Correlated kernel panics are a common distributed system reliability issue because modern AI systems involve billions of logical processing cores on a variety of NVIDIA, TPU, AMD, and specialized accelerator platforms. This study examined Cluster Crash Diagnosis (CCD) as an architecture that uses Linux kdump technology and transforms it into a scalable, secure fleet telemetry system. The study involved two production validations, using an 8,000-node NVIDIA H100 cluster and a 512-node TPU v5e pod, respectively. CCD introduced 2 GB of crashkernel reservation, vendor sidecars, local persistence of vmcore, LZO compression, 50 Mbps transport control, exponential backoff, hashing stack with a hash function that uses SHA256, normalized Levenshtein clustering at a threshold of τ = 0.85, vendor-neutral classification, and topology-aware analytics. In the H100 case, the crash rate went up from 0.5 crashes/day to 15 crashes/day after a patch to the operating system; CCD reduced diagnosis time from 6 hours to 8 minutes, a 45× faster turnaround. In the TPU case, all crashes observed were associated with the same failing top-of-rack optical switch. Overall, CCD lowered the MTTD by 70-85% while improving multi-tenant security. These results demonstrate how coordinated capture, transport, deduplication, correlation, and access controls can help enhance resilience in exascale AI operations. Future work should extend predictive topology correlation and cross-vendor validation.

Keywords

Cluster Crash Diagnosis (CCD), Heterogeneous Accelerator Systems, Linux Kdump, Distributed Crash Diagnosis, Kernel Panic Deduplication, Exascale AI Clusters

References

P. Metzger, A. T. Markettos, E. T. Napierała, M. Naylor, R. N. Watson, and T. M. Jones, “Deprivileging low-level GPU drivers efficiently with user-space processes and CHERI compartments,” in Proc. 2025 ACM SIGSAC Conf. Computer and Communications Security, Nov. 2025, pp. 2699–2713. https://dl.acm.org/doi/pdf/10.1145/3719027.3765036

T. Breitenbach, S. M. Divakar, L. Rasbach, and P. Jahnke, “ML-driven risk estimation for memory failure in a data center environment with convolutional neural networks, self-supervised data labeling and distribution-based model drift determination,” J. Parallel Distrib. Comput., vol. 185, Art. no. 104800, 2024. https://www.sciencedirect.com/science/article/pii/S0743731523001703

A. Mathai, C. Huang, P. Maniatis, A. Nogikh, F. Ivančić, J. Yang, and B. Ray, “KGYM: A platform and dataset to benchmark large language models on Linux kernel crash resolution,” Advances in Neural Information Processing Systems, vol. 37, pp. 78053–78078, 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/8e9ed2a28af7d9085180e3817b2c9a57-Paper-Datasets_and_Benchmarks_Track.pdf

M. Baqar, S. Naqvi, and R. Khanda, “AI augmented CI/CD pipelines: From code commit to production with autonomous decisions,” in Proc. 2025 3rd Int. Conf. Foundation and Large Language Models (FLLM), Nov. 2025, pp. 1041–1048.https://arxiv.org/pdf/2508.11867

S. K. RaoVanama, “AI-augmented CI/CD pipeline optimization for scalable cloud-native deployment,” Int. J. Artificial Intelligence, Data Science, and Machine Learning, vol. 5, no. 4, pp. 175–187, 2024. https://ijaidsml.org/index.php/ijaidsml/article/download/368/338

Y. Dehfouli and A. Habibi Lashkari, “Memory analysis for malware detection: A comprehensive survey using the OSCAR methodology,” ACM Comput. Surv., vol. 58, no. 4, pp. 1–58, 2025. https://scholar.google.com/scholar?output=instlink&q=info:nyhjUuDr3GQJ:scholar.google.com/&hl=en&as_sdt=0,5&scillfp=9643302646581153144&oi=lle

IBM, “Dump collection failed due to out of memory (OOM) error,” IBM Support, 2026. https://www.ibm.com/support/pages/node/7060322

G. Malik, “Biometric authentication—Risks and advancements in biometric security systems,” J. Computer Science and Technology Studies, vol. 6, no. 3, pp. 159–180, 2024. https://alkindipublisher.com/index.php/jcsts/article/download/9131/7816

P. Ananthula, S. Sepuri, M. Sethi, and H. Goel, “The strategic value of AI-driven physical identity and access management: Establishing cyber physical zero trust in critical infrastructure,” in Proc. 2026 4th Int. Conf. Self Sustainable Artificial Intelligence Systems (ICSSAS), May 2026, pp. 292–300. https://ieeexplore.ieee.org/abstract/document/11559385/

W. Luo, R. Fan, Z. Li, D. Du, Q. Wang, and X. Chu, “Benchmarking and dissecting the NVIDIA Hopper GPU architecture,” in Proc. 2024 IEEE Int. Parallel and Distributed Processing Symp. (IPDPS), May 2024, pp. 656–667. https://arxiv.org/pdf/2402.13499

M. Ozkan et al., “Performance, efficiency, and cost analysis of wafer-scale AI accelerators vs. single-chip GPUs,” Device, 2025. https://www.cell.com/device/fulltext/S2666-9986(25)00147-4

A. Smith and V. K. Alla, “AMD Instinct MI300X: A generative AI accelerator and platform architecture,” IEEE Micro, vol. 45, no. 3, pp. 41–48, 2025. https://hc2024.hotchips.org/assets/program/conference/day1/23_HC2024.AMD.MI300X.ASmith%28MI300X%29.v1.Final.20240817.pdf

CloudOptimo, “Google TPU Ironwood: Revolutionizing AI inference at scale,” 2025. https://www.cloudoptimo.com/blog/google-tpu-ironwood-revolutionizing-ai-inference-at-scale/

Duan et al., “Efficient training of large language models on distributed infrastructures: A survey,” Vicinagearth, vol. 3, no.1, Art. no. 9, 2026.https://link.springer.com/content/pdf/10.1007/s44336-026-00038-z.pdf

S. Gondi, S. Sepuri, and P. Narayan, “AI ethics for public financial systems: A cross-sector perspective,” Int. J. Applied Mathematics, vol. 38, pp. 2212–2233, 2025.

M. Chow et al., “ServiceLab: Preventing tiny performance regressions at hyperscale through pre-production testing,” in Proc. 18th USENIX Symp. Operating Systems Design and Implementation (OSDI 24), 2024, pp. 545–562. https://www.usenix.org/system/files/osdi24-chow.pdf

N. Jouppi et al., “TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings,” in Proc. 50th Annu. Int. Symp. Computer Architecture, Jun. 2023, pp. 1–14. https://dl.acm.org/doi/pdf/10.1145/3579371.3589350

A. Kokolis, M. Kuchnik, J. Hoffman, A. Kumar, P. Malani, F. Ma, Z. DeVito, S. Sengupta, K. Saladi, and C.-J. Wu, “Revisiting reliability in large-scale machine learning research clusters,” arXiv preprint arXiv:2410.21680, 2024. https://arxiv.org/pdf/2410.21680

Oracle Linux Documentation, “Collecting crash dumps using Kdump utility,” 2025. https://docs.oracle.com/en-us/iaas/oracle-linux/oci/diagnostics-kdump.htm

S. Tao, B. Zhang, and Q. Zhang, “ECHO: Enhancing Linux kernel fuzzing via call stack-aware crash deduplication,” Electronics, vol. 14, no. 14, Art. no. 2914, 2025. https://www.mdpi.com/2079-9292/14/14/2914

S. Zhang et al., “Robust failure diagnosis of microservice system through multimodal data,” IEEE Trans. Services Comput., vol. 16, no. 6, pp. 3851–3864, 2023. https://arxiv.org/pdf/2302.10512

Z. Zhao, B. Wu, and L. Guo, “Integrated failure localization of optical switches and links in all-optical DCNs using DM-trails based on bidirectional voltage constraint,” J. Lightwave Technol., vol. 43, no. 8, pp. 3524–3535, 2025. https://ieeexplore.ieee.org/abstract/document/10840248/

Red Hat Enterprise Linux, “Chapter 14: Configuring Kdump on the command line,” in RHEL 9 Managing, Monitoring and Updating the Kernel, 2025. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/installing-kdump_assembly_managing-kernel-command-line-parameters-with-uki

S. Cheng et al., “Thorough characterization and analysis of large transformer model training at-scale,” Proc. ACM Meas. Anal. Comput. Syst., vol. 8, no. 1, pp. 1–25, 2024. https://dl.acm.org/doi/pdf/10.1145/3639034

Z. Jiang et al., “L4: Diagnosing large-scale LLM training failures via automated log analysis,” in Proc. 33rd ACM Int. Conf. Foundations of Software Engineering, Jun. 2025, pp. 51–63. https://dl.acm.org/doi/pdf/10.1145/3696630.3728531

S. Grover, “Self-healing web automation: An empirical comparison of traditional Selenium frameworks and Healenium stack,” Int. J. Applied Mathematics, vol. 38, no. 12s, pp. 428–438, 2025. https://ijamjournal.org/ijam/publication/index.php/ijam/article/download/1391/1265

M. J. Gossman, B. Nicolae, and J. C. Calhoun, “Scalable I/O aggregation for asynchronous multi-level checkpointing,” Future Gener. Comput. Syst., vol. 160, pp. 420–432, 2024. https://www.sciencedirect.com/science/article/am/pii/S0167739X24002929

X. Han et al., “vClos: Network contention aware scheduling for distributed machine learning tasks in multi-tenant GPU clusters,” Comput. Netw., vol. 268, Art. no. 111285, 2025. https://jhc.sjtu.edu.cn/~shizhenzhao/computer_network_2025.pdf

S. Pavlenko and P. Kuliabko, “Deduplication of error reports in software malfunction: Algorithms for comparing call stacks,” Bulletin of Cherkasy State Technological University: Technical Sciences, vol. 28, no. 4, pp. 59–69, 2023. https://er.chdtu.edu.ua/bitstream/ChSTU/4887/1/8.pdf

W. Beer, “Transform your operations with Davis AI root cause analysis,” Dynatrace, 2024. https://www.dynatrace.com/news/blog/transform-your-operations-with-davis-ai-root-cause-analysis/

S. Bekele et al., “THAPI: Tracing heterogeneous APIs,” in Proc. European Conf. Parallel Processing, Cham, Switzerland: Springer Nature, Aug. 2025, pp. 54–70. https://arxiv.org/pdf/2504.03683

J. K. Lastre, Y. Ko, H. Kwon, and I. You, “Evaluating Transport Layer Security 1.3 optimization strategies for 5G cross-border roaming: A comprehensive security and performance analysis,” Sensors, vol. 25, no. 19, Art. no. 6144, 2025. https://www.mdpi.com/1424-8220/25/19/6144

Ruijie, “Multi-tenant isolation technology in AIGC networks—Data security and performance stability,” 2024. https://www.ruijie.com/en-global/support/tech-gallery/multi-tenant-isolation-technology-in-aigc-networks-data-security-and-performance-stability

J. Dong et al., “Evolution of Aegis: Fault diagnosis for AI model training service in production,” in Proc. 22nd USENIX Symp. Networked Systems Design and Implementation (NSDI 25), 2025, pp. 865–881. https://www.usenix.org/system/files/nsdi25-dong.pdf

A. C. Jha, P. Gannavarapu, and S. Durgam, “Machine learning-driven security system for next-generation ultra-low-latency financial network governance,” 2025. https://ieeexplore.ieee.org/abstract/document/11374574

Y. Sun et al., “ClusterRCA: An end-to-end approach for network fault localization and classification for HPC system,” in Proc. 2025 IEEE 36th Int. Symp. Software Reliability Engineering (ISSRE), Oct. 2025, pp. 347–358. https://arxiv.org/pdf/2506.20673

K. S. Chadha, “Predictive risk modeling in P&C insurance using Guidewire DataHub and Power BI embedded analytics,” Int. J. Networks and Security, vol. 5, no. 2, pp. 1–29, 2025.

J. Krebs, D. Gavrilenko, D. Amir, S. Landau Feibish, and M. Silberstein, “FlowPulse: Catching network failures in ML clusters,” in Proc. 24th ACM Workshop Hot Topics in Networks, Nov. 2025, pp. 139–148. https://dl.acm.org/doi/pdf/10.1145/3772356.3772384

N. Kumar, “Edge computing patterns for real-time order flow optimization,” Int. J. Applied Mathematics, 2025. https://www.papjournals.com/index.php/edm/article/download/785/540

A. Wongpanich et al., “Machine learning fleet efficiency: Improving TPU systems at scale with ML productivity goodput,” Proc. Machine Learning and Systems, vol. 8, pp. 1291–1303, 2026. https://proceedings.mlsys.org/paper_files/paper/2026/file/fbe2b2f74a2ece8070d8fb073717bda6-Paper-Conference.pdf

L. Rzepka, J. Ottmann, R. Stoykova, F. Freiling, and H. Baier, “A scenario-based quality assessment of memory acquisition tools and its investigative implications,” Forensic Sci. Int.: Digital Investigation, vol. 52, Art. no. 301868, 2025. https://www.sciencedirect.com/science/article/pii/S2666281725000071

E. Shibaev, D. Sushentsev, Y. Golubev, and A. Khvorov, “Stack trace deduplication: Faster, more accurately, and in more realistic scenarios,” in Proc. 2025 IEEE Int. Conf. Software Analysis, Evolution and Reengineering (SANER), Mar. 2025, pp. 511–521. https://arxiv.org/pdf/2412.14802

S. Rangu, “Enterprise digital transformation in financial services: Emerging trends and technologies,” Computer Fraud & Security, 2025. https://computerfraudsecurity.com/index.php/journal/article/view/786

Most read articles by the same author(s)

<< < 4 5 6 7 8 9 10 11 12 > >>

International Journal of Modern Computer Science and IT Innovations

Debugging Billion-Core AI Clusters: Distributed Crash Diagnosis for Heterogeneous Accelerator Systems

Abstract

Keywords

References

Most read articles by the same author(s)

Similar Articles