Collaborative artificial intelligence systems (CAISs) aim to work with humans in a shared space to achieve a common goal, but this can pose hazards that could harm human beings. We identify emerging problems in this context and report our vision of and progress toward a risk-driven assurance process for CAISs.