Critical infrastructure information systems are complex, open, connected and heterogeneous computer network systems and - as the name implies - play an important role for some critical infrastructure. These systems occasionally fail and the need arises to explain what happened and assert that whatever it was it will not happen again. In this abstract we describe ongoing work towards an approach for analyzing hard-to-explain behavior in software-intensive systems. This approach is based on input both from the critical infrastructure community and from diagnosing very large deeply embedded systems - two kinds of systems that share several relevant properties in regard to diagnosing malfunction.