ML-Enabled Systems Model Deployment and Monitoring: Status Quo and ProblemsShow others and affiliations
2024 (English)In: Software Quality as a Foundation for Security / [ed] Peter Bludau, Rudolf Ramler, Dietmar Winkler, Johannes Bergsmann, Springer Science+Business Media B.V., 2024, p. 112-131Conference paper, Published paper (Refereed)
Abstract [en]
Systems that incorporate Machine Learning (ML) models, often referred to as ML-enabled systems, have become commonplace. However, empirical evidence on how ML-enabled systems are engineered in practice is still limited; this is especially true for activities surrounding ML model dissemination. [Goal] We investigate contemporary industrial practices and problems related to ML model dissemination, focusing on the model deployment and the monitoring ML life cycle phases. [Method] We conducted an international survey to gather practitioner insights on how ML-enabled systems are engineered. We gathered a total of 188 complete responses from 25 countries. We analyze the status quo and problems reported for the model deployment and monitoring phases. We analyzed contemporary practices using bootstrapping with confidence intervals and conducted qualitative analyses on the reported problems applying open and axial coding procedures. [Results] Practitioners perceive the model deployment and monitoring phases as relevant and difficult. With respect to model deployment, models are typically deployed as separate services, with limited adoption of MLOps principles. Reported problems include difficulties in designing the architecture of the infrastructure for production deployment and legacy application integration. Concerning model monitoring, many models in production are not monitored. The main monitored aspects are inputs, outputs, and decisions. Reported problems involve the absence of monitoring practices, the need to create custom monitoring tools, and the selection of suitable metrics. [Conclusion] Our results help provide a better understanding of the adopted practices and problems in practice and support guiding ML deployment and monitoring research in a problem-driven manner. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Place, publisher, year, edition, pages
Springer Science+Business Media B.V., 2024. p. 112-131
Series
Lecture Notes in Business Information Processing, ISSN 18651348, E-ISSN 18651356 ; 505
Keywords [en]
Deployment, Machine Learning, Monitoring, Life cycle, Statistical methods, Complete response, Contemporary practices, Industrial practices, Industrial problem, International survey, Machine learning models, Machine-learning, Status quo, System models
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-26219DOI: 10.1007/978-3-031-56281-5_7ISI: 001267936400007Scopus ID: 2-s2.0-85192177513ISBN: 9783031562808 (print)OAI: oai:DiVA.org:bth-26219DiVA, id: diva2:1859534
Conference
16th International Conference on Software Quality, SWQD 2024, Vienna, 23 April through 25 April 2024
2024-05-222024-05-222024-09-16Bibliographically approved